Does it make any difference to the security and efficiency if we use SHA-256 or SHA-512 for the Mask Generation Function MGF1 that generates the masking / padding within the OAEP encryption scheme and the PSS signature scheme?

A lot of implementations also seem to sport support for SHA-224 and SHA-384 for MGF1. Does it make any sense to use these truncated hash functions for something that needs to generate a mask of a particular size (close to the size of the key size for RSA)? Or would that simply add unnecessary overhead?

It seems to me that SHA-512 would be more efficient because of the output size alone. However, most implementations seem to default on SHA-1, SHA-256 or the same hash as used to hash the data for PSS.

Since the input sizes are fixed, length-extension attacks are not relevant, so any of the SHA-2 functions reasonably implements the random oracle model assumed by OAEP or PSS via MGF1even the default of SHA-1 works with MGF1.

Obviously it will cost slightly more to use SHA-224 or SHA-384 than to use SHA-256 or SHA-512 because SHA-224 and SHA-384 are effectively truncations of SHA-256 and SHA-512: to get the same amount of output as (say) twelve SHA-384 invocations each costing a SHA-512 computation, you could pay for a mere nine SHA-512 computations instead. So the truncated options don't provide any benefit. And, of course, SHA-512 than SHA-256 is generally faster on CPUs with 64-bit adders.

All that said, it is hard to imagine that this could substantially affect security *or* performance since you're about to do a 2048-bit modular exponentiation anyway, which will be the bulk of the costbut, of course, you're in a better position to do that measurement in your application! (Make sure to choose $e = 3$ to minimize time spent in exponentiation.)

- +0 – Thanks for your answer. I'm kind of missing the part about output size and SHA-224/386 though. Possibly input block size as well (?). — May 21, 2019 at 16:26
- +0 – The output size really doesn't matter; you could take a single bit at a time, and repeat it a thousand times, and the security would be unaffected, but obviously it is going to be slightly slower to effectively SHA-512 (say) twelve times via SHA-384 than to compute SHA-512 nine times directly without truncation. The MGF seedinput sizedepends on the hash function used to compress the message, not on the MGF hash. — May 21, 2019 at 16:32
- +0 – Let's say that I forget all my knowledge about MGF1, then certainly the input block size makes a difference on how many times the hash function is executed? Of does MGF always perform a single compression of the input? — May 21, 2019 at 16:35
- +0 – The inputs are pretty much all going to be at most one block long: 256-bit mask seed (unless you're using an obscenely large hash), 32-bit counter for MGF1 blocks, 64-bit input length inside SHA-2 512 bits. — May 21, 2019 at 16:39
- +0 – So can I conclude 1. for security it doesn't matter, 2. input block size doesn't matter (much), 3. SHA-224/384 don't make sense, but 4. who cares, modular exponentiation will be much slower than MGF1? I'll quickly do some testing. PS note that e.g. smart cards often have a Montgomery multiplier, but no SHA-2 acceleration. — May 21, 2019 at 16:58