Does it make any difference to the security and efficiency if we use SHA-256 or SHA-512 for the Mask Generation Function MGF1 that generates the masking / padding within the OAEP encryption scheme and the PSS signature scheme?
A lot of implementations also seem to sport support for SHA-224 and SHA-384 for MGF1. Does it make any sense to use these truncated hash functions for something that needs to generate a mask of a particular size (close to the size of the key size for RSA)? Or would that simply add unnecessary overhead?
It seems to me that SHA-512 would be more efficient because of the output size alone. However, most implementations seem to default on SHA-1, SHA-256 or the same hash as used to hash the data for PSS.
Since the input sizes are fixed, length-extension attacks are not relevant, so any of the SHA-2 functions reasonably implements the random oracle model assumed by OAEP or PSS via MGF1even the default of SHA-1 works with MGF1.
Obviously it will cost slightly more to use SHA-224 or SHA-384 than to use SHA-256 or SHA-512 because SHA-224 and SHA-384 are effectively truncations of SHA-256 and SHA-512: to get the same amount of output as (say) twelve SHA-384 invocations each costing a SHA-512 computation, you could pay for a mere nine SHA-512 computations instead. So the truncated options don't provide any benefit. And, of course, SHA-512 than SHA-256 is generally faster on CPUs with 64-bit adders.
All that said, it is hard to imagine that this could substantially affect security or performance since you're about to do a 2048-bit modular exponentiation anyway, which will be the bulk of the costbut, of course, you're in a better position to do that measurement in your application! (Make sure to choose $e = 3$ to minimize time spent in exponentiation.)