java ssl encryption bouncycastle aes-gcm
Updated Mon, 15 Aug 2022 19:59:37 GMT

decrypt TLS 1.2 AES-GCM packet

I'm working on a Java program to decrypt a TLS 1.2 Session which is using the TLS_RSA_WITH_AES_128_GCM_SHA256 cipher. I recorded a test session using wireshark. The Master Secret is known.

No.     Time           Protocol Length Info
      4 0.000124000    TLSv1.2  166    Client Hello
      6 0.000202000    TLSv1.2  1074   Server Hello, Certificate, Server Hello Done
      8 0.001071000    TLSv1.2  393    Client Key Exchange, Change Cipher Spec, Finished
      9 0.003714000    TLSv1.2  301    New Session Ticket, Change Cipher Spec, Finished
     11 6.443056000    TLSv1.2  116    Application Data
     12 6.443245000    TLSv1.2  765    Application Data
     15 6.443390000    TLSv1.2  103    Alert (Level: Warning, Description: Close Notify)

Packet 11 Contains a HTTP GET Request that I'm trying to decrypt.

Handshake Data:

Cipher:        TLS_RSA_WITH_AES_128_GCM_SHA256
Client Random: 375f5632ba9075b88dd83eeeed4adb427d4011298efb79fb2bf78f4a4b7d9d95
Server Random: 5a1b3957e3bd1644e7083e25c64f137ed2803b680e43395a82e5b302b64ba763    
Master Secret: 2FB179AB70CD4CA2C1285B4B1E294F8F44B7E8DA26B62D00EE35181575EAB04C

Packet 11 Data:

Direction is Client -> Server.
Secure Sockets Layer
    TLSv1.2 Record Layer: Application Data Protocol: Application Data
    Content Type: Application Data (23)
    Version: TLS 1.2 (0x0303)
    Length: 45
    Encrypted Application Data: c91de005e2ae50a8a57abee55c183667b136343feef4a387cb7cf83030a47e230af268378c4f33c8b5bab3d26d

What I have done so far:

Key Derivation:

I only need Client keys here, as I want to decrypt a Client->Server package. I expanded server and client keys and IVs as per RFC. Client Write Key: 4B119DFBFC930ABE130030BD53C3BF78 Client Write IV: 2029CAE2


I create AES-GCM nonce from salt (=Client Write IV) and explicit nonce (=first 8 Byte of encrypted data). Salt: 2029CAE2 explicitNonce: C91DE005E2AE50A8 Nonce: 2029CAE2C91DE005E2AE50A8

Additional Authentication Data (AAD):

This is where I apparently got stuck. The RFC5246 says:

additional_data = seq_num + TLSCompressed.type + TLSCompressed.version + TLSCompressed.length; where "+" denotes concatenation.

So I made this:

byte[] aad = {0, 0, 0, 0, 0, 0, 0, 1,   // seq_no uint64
    0x17,               // type 0x17 = Application Data
    0x03, 0x03,             //  TLS Version 1.2
    0, 45};             // 45 Bytes of encrypted data

I think seq_no is 1. It gets reset to zero, when Change Cipher Spec record is sent. (Packet #8) Then the encrypted Finished record has seq_no = 0. And the next client packet is our Packet #11 with seq_no = 1.


Now I'm feeding everything into BouncyCastle:

AEADParameters parameters = new AEADParameters(new KeyParameter(clientWriteKey), 128, nonce, aad);
GCMBlockCipher gcmBlockCipher = new GCMBlockCipher(new AESFastEngine());
gcmBlockCipher.init(false, parameters);
byte[] plainText = new byte[gcmBlockCipher.getOutputSize(cipherText.length)];
try {
    int decLen = gcmBlockCipher.processBytes(cipherText, 0, cipherText.length, plainText, 0);
    decLen += gcmBlockCipher.doFinal(plainText, decLen);
} catch (InvalidCipherTextException e) {
    System.out.println("MAC failed: " + e.getMessage());

This always throws MAC failed: mac check in GCM failed. BUT the decrypted output is correct:

byte[] decomp = decompress(plainText);
System.out.println(new String(decomp, "UTF-8"));

This prints GET / HTTP/1.0\n.

Decompress helper:

public static byte[] decompress(byte[] data) throws IOException, DataFormatException {
    Inflater inflater = new Inflater(true);
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length);
    byte[] buffer = new byte[1024];
    while (inflater.getRemaining() > 0) {
        int count = inflater.inflate(buffer);
        outputStream.write(buffer, 0, count);
    byte[] output = outputStream.toByteArray();
    return output;

Conclusion: As the decrypted output is correct, i can safely assume that key derivation and decryption are working fine. Only authentication fails. So I think maybe I'm doing something wrong with the Additional Authentication Data (AAD). So this question boils down to:

How are the Additional Authentication Data (AAD) correctly assembled?

Thank you!


GCM mode computes MAC from message, associated data and public nonce, you covered it very well.

I think you are using wrong length, it should be plaintext length before encrypting and appending MAC. Try 45 - 8 (explicit nonce) - 16 (MAC) = 21.