Cryptography
rsa padding hash-signature oaep rsa-pss
Updated Fri, 20 May 2022 19:51:03 GMT

RSA Signing: SHA512 incompatible with PSS?


I am wrapping the RSACng class from the .Net API. Whilst unit testing, I run checks against an array of possibilities:

  • Key Size = 1024, 2048, or 3072 bits
  • Padding Mode = PKCS v1, or PSS (=PKCS v2?)
  • Hash Algorithm = SHA-1, SHA256, SHA512

I test for all 3*2*3=18 possible configurations. All tests pass except one combination:

  • Key size = 1024 bits
  • Padding Mode = PSS
  • Hash Algorithm = SHA512

Should this be expected, and how could have I anticipated it? So far, Google has not been my friend.

I also ran a similar batch of unit tests for RSA encryption, and noted a 1024 bits key is incompatible with OAEP-SHA1 and OAEP-SHA256 padding modes. I suspect the key is not large enought, but I was again unable to read some info as to why this was happening.

Solution

Yes, this is expected, if the RSASSA-PSS signing code/test uses salt the width of the hash, which is customary. In that case, a $h$-bit hash (with $h$ multiple of 8) requires an RSA public modulus at least $2h+9$ bits.

That's semi-clearly stated in PKCS#1 v2.2, section 9.1.1, condition on enBits, with actual test in step 3. That's because the message representative needs to convey $h$ bits for the message hash, $h$ bits for the salt, $8$ bits for a one-byte identifier, and then some.

In PSS, the salt length can be made a parameter, rather than fixed to the hash width as usual. Also, it can optionally be deterministic or fixed, rather than random as usual. But far from all APIs allow that.

There are similar limitations for RSAES-OAEP (without flexibility: the tag's hash always is as wide as the hash, even when the tag is empty/absent, as it often is).

That identifier BCh plays a role when using PSS padding for Rabin signatures or when it is used $r\to\min(r^d\bmod N,N-(r^d\bmod N))$ instead of the textbook RSA private key operation, in order to allow immediate textbook-RSA encryption of the signature with any public key with public modulus of the same size as the one used for signature.

Comments (5)

  • +0 – I've got the OAEP calculations for size in handy bytes over here. Note the N/A for RSA-1024 with OAEP using SHA-512. — Feb 08, 2020 at 04:32  
  • +0 – Thanks @fgrieu for the spot on answer! — Feb 10, 2020 at 16:47  
  • +0 – @MaartenBodewes that's interesting because .Net allows me to combine RSA1024 & OAEP-SHA512, but not to combine RSA1024 & OAEP-SHA1 or SHA256 (allow = does not throw an error). I understand some combinations might make little sense, which is partly why I am testing all possible scenarios: I wanted to know if my wrapper requires to "validate" the configuration (key length & hash length) another programmer would set up whilst using the wrapper. (sort of a last resort layer of security) — Feb 10, 2020 at 16:54  
  • +0 – I'll be short: If you want to make sense of anything, the .NET crypto API is not a good starting point. It does some things correctly, many other things are unspecified or just badly designed. — Feb 10, 2020 at 16:56  
  • +0 – About the salt length: "rather than fixed to the hash width as usual". Is the hash width recommended or optimal in some way? If not, why has it become the "usual" choice? OpenSSL by default seems to choose the maximum salt length when signing. — Jul 27, 2021 at 17:30  

External Links

External links referenced by this document:

References