We have a server to server communication scenario, where we would use a CA signed certificate for the TLS server authentication and would like to use a self-signed certificate for the Client authentication.
Is there any additional security that you would get by using a CA signed certificate as compared to using self-signed certificate in case of this scenario (client authentication) ?
UPDATED We are importing the relevant Client certificates into the Server trust store / whitelist. Given this context does the CA signed cert offer any additional security benefits ?
[This is a re-write of the question addressing the update. See the original in the revision history]
TLS client auth is something that almost always requires you to extend your TLS engine by writing your own cert custom validation code on the server-end. It seems like you've avoided this by piggy-backing on the trust-store with no additional validation code.
The key question is:
When your servers are verifying the client certificate, how do they know that it's a cert issued by you, and not a cert issued by a bad-guy?.
You say:
We are importing the relevant Client certificates into the Server trust store / whitelist.
This might be ok security-wise. The first question to ask is what else is in the Server trust store, and if it will accept connections from certs other than the ones you whitelisted. Are we talking about the OS trust store containing all the publicly trusted root CAs, or is this a custom java trust store that is empty prior to you inserting your whitelist? For example, if the public CA Let's Encrypt is in the trust store, then anybody with a Let's Encrypt cert might be able to get a connection to your server.
Basically, you want to convince yourself that the self-signed certs you created are the only certs that will be able to establish a connection to the server. I have seen products that work this way, so it's not entirely far-fetched, and proper security is possible, however there are a few other things to think about. Let's first talk about the usual way this is set up, then I'll compare.
Usually, the way client-auth works in a situation like this is one of two ways:
You have a private CA that you control. It only issues certificates for valid TLS clients. Therefore the TLS server can simply verify that the client presents a cert issued by this CA, and you know that it is authentic.
Your clients get certificates from a publicly-trusted CA like Let's Encrypt. Suddenly, just checking that the cert was issued by Let's Encrypt is not good enough because half the internet has a cert like that, so that doesn't prove anything. Here, you want your "TLS server" to keep a whitelist of domain names of the "TLS clients". When validating a client cert, in addition to all the usual validity checking, it should that the DN in the cert matches a domain name on its whitelist.
The security goal you want is that your TLS Clients (and only your TLS clients) can establish connections to your TLS Server(s). You can certainly accomplish this with the system you described. The initial setup will be simpler, but I predict that over the long-term the additional operational complexity will come back to bite you, either in terms of additional time and effort, or when it hasn't needed to be modified in so long that nobody remembers how it works. Make sure you prepare thorough documentation on how it works!
External links referenced by this document: