I looked in the system keychain and tried to install the new ones there, but kept getting an error saying the the identity already existed.
It turns out that in fact, the server copies certificates from the keychain into /etc/certificates at boot time. I hadn't known this. When I deleted the certificates from the keychain, everything "just worked" after I installed the new certs into /etc/certificates. The missing piece of the puzzle was the server scribbling in /etc/certificates.
Chapter three of this (in progress) is that now the startcom signing authority cert is in the server's default list. I verified this on a new install of the server software--on that system, the standard Server Admin approach works flawlessly, no direct access to /etc/certificates is needed at all. So, the next step on the older system is to turn off all ssl services (at least iCal, iChat, Mail, OD, RADIUS, VPN, and Web), clean out /etc/certificates, and install the up-to-date certs into Server Admin. Then, go through each service and ditch the custom configurations, replacing them with the standard wildcard cert installed normally.
OK, I think I've done this successfully: it seems to be working. So, the comment about the server scribbling in /etc/certificates no longer is relevant to my particular configuration, but it is very relevant to someone who has a custom configuration. My advice: go ahead and put the certs in /etc/certificates, but (1) don't name them either Default or the address certified (e.g., *.domain.net), and (2) make sure they are NOT entered in the keychain as well. One or the other, please.