Programming
java encryption bouncycastle aes-gcm
Updated Sat, 03 Sep 2022 22:37:01 GMT

Reading AES/GCM encoded data in chunks with BouncyCastle in Java


I am trying to figure out how to read data that was encoded with AES/GCM/NoPadding. The data that I am working with will be arbitrarily large, and I'm hoping to read it in chunks, but I'm having difficulties figuring out how that would be accomplished. Here is an example of where I'm at right now:

@Test
public void chunkDecrypt() throws Exception {
    key = MessageDigest.getInstance("MD5").digest("som3C0o7p@s5".getBytes());
    iv = Hex.decode("EECE34808EF2A9ACE8DF72C9C475D751");
    byte[] ciphertext = Hex
            .decode("EF26839493BDA6DA6ABADD575262713171F825F2F477FDBB53029BEADB41928EA5FB46737D7A94D5BE74B6049008443664F0E0D883943D0EFBEA09DB");
    Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
    cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(key, "AES"), new IvParameterSpec(iv));
    byte[] fullDecryptedPlainText = cipher.doFinal(ciphertext);
    assertThat(new String(fullDecryptedPlainText),
            is("The quick brown fox jumps over the lazy dogs"));
    byte[] first32 = Arrays.copyOfRange(ciphertext, 0, 32);
    byte[] final28 = Arrays.copyOfRange(ciphertext, 32, 60);
    byte[] decryptedChunk = new byte[32];
    int num = cipher.update(first32, 0, 32, decryptedChunk);
    assertThat(num, is(16));
    assertThat(new String(decryptedChunk, 0, 16), is("The quick brown "));
    num = cipher.update(first32, 0, 32, decryptedChunk);
    assertThat(num, is(32));
    assertThat(new String(decryptedChunk, 0, 16), is("fox jumps over t"));
    num = cipher.update(final28, 0, 24, decryptedChunk);
    assertThat(num, is(44));
    assertThat(new String(decryptedChunk, 0, 12), is("he lazy dogs"));
}

Note that I get past the first assert no problem, so the data can be decoded in a single go. Also, the next two sets of asserts (decoding the first 32 bytes in 16-byte chunks) work "correctly" but I arrived at this formula through trial-and-error. There are a few things about them that I don't understand:

  • Even though I am reading in 16-byte chunks, all of my numbers seem to need to be multiples of 32. If I change to the following code, then the first call to cipher.update() fails with a return value of 0.

    byte[] first16 = Arrays.copyOfRange(ciphertext, 0, 16);
    byte[] decryptedChunk = new byte[16];
    int num = cipher.update(first16, 0, 16, decryptedChunk);
    
  • If I change back to 32 on the input side, but I work with a 16-byte output buffer, then the first call succeeds and returns the expected data, but the second call to cipher.update() throws ArrayIndexOutOfBoundsException.

    byte[] first32 = Arrays.copyOfRange(ciphertext, 0, 32);
    byte[] decryptedChunk = new byte[16];
    int num = cipher.update(first32, 0, 32, decryptedChunk);
    num = cipher.update(first32, 0, 32, decryptedChunk);
    
  • So, if I change the code back to my original example (decryptedChunk is sized at 32 bytes) then the third call to cipher.update() returns a value of 16 (meaning what???) and decryptedChunk contains garbage data.

  • I also tried replacing the last call to cipher.update() with a call to cipher.doFinal() instead:

    decryptedChunk = cipher.doFinal(final28);
    assertThat(new String(decryptedChunk, 0, 12), is("he lazy dogs"));
    

But this fails with a BadPaddingException (mac check in GCM failed).

Any suggestions?


Update with solution

After playing around some with the suggested code from Ebbe M. Pedersen, I have been able to put together the following solution:

@Test
public void chunkDecrypt() throws Exception {
    byte[] key = MessageDigest.getInstance("MD5").digest("som3C0o7p@s5".getBytes());
    byte[] iv = Hex.decode("EECE34808EF2A9ACE8DF72C9C475D751");
    byte[] ciphertext = Hex
            .decode("EF26839493BDA6DA6ABADD575262713171F825F2F477FDBB53029BEADB41928EA5FB46737D7A94D5BE74B6049008443664F0E0D883943D0EFBEA09DB");
    Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding", "BC");
    cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(key, "AES"), new IvParameterSpec(iv));
    int chunkSize = 16;
    byte[] inBuffer = new byte[chunkSize];
    int outBufferSize = ((chunkSize + 15) / 16) * 16;
    byte[] outBuffer = new byte[outBufferSize];
    for (int i = 0; i < ciphertext.length; i += chunkSize) {
        int thisChunkSize = Math.min(chunkSize, ciphertext.length - i);
        System.arraycopy(ciphertext, i, inBuffer, 0, thisChunkSize);
        int num = cipher.update(inBuffer, 0, thisChunkSize, outBuffer);
        if (num > 0) {
            logger.debug("update #" + ((i / chunkSize) + 1) + " - data <"
                    + new String(outBuffer, 0, num) + ">");
        }
    }
    int num = cipher.doFinal(inBuffer, chunkSize, 0, outBuffer);
    logger.debug("doFinal - data <" + new String(outBuffer, 0, num) + ">");
}

This works properly for any value of chunkSize that I have selected. I have marked that answer as accepted. Thank you all for the help.




Solution

Block ciphers [ed: in Bouncy Castle] have an internal buffer that they keep updating, and only when they have enough data for a full block, will the decrypt occur, and a chunk of the decrypted data be returned.

You can see this if you try and decrypt it 1 byte at a time like this:

    byte[] buffer = new byte[32];
    for (int i = 0; i < ciphertext.length; i++) {
        int num = cipher.update(ciphertext, i, 1, buffer);
        if (num > 0) {
            System.out.println("update #" + (i + 1) + " - data <" + new String(buffer, 0, num) + ">");
        }
    }
    int num = cipher.doFinal(ciphertext, ciphertext.length, 0, buffer);
    System.out.println("doFinal - data <" + new String(buffer, 0, num) + ">");

This gives the following output with your encrypted data:

update #32 - data <The quick brown >
update #48 - data <fox jumps over t>
doFinal - data <he lazy dogs>

Notice that I need to do a doFinal(), to get the last piece of data out.


Note that this is particular to the Bouncy Castle implementation, at least up to version 1.50. CTR mode allows to pre-compute blocks of the key stream used to encrypt/decrypt the data (by XOR'ing, analog of a OTP encryption). So in principle each byte or even bit can be encrypted/decrypted on its own.





Comments (5)

  • +1 – For a bock cipher in streaming mode, each byte can be decrypted on its own. Normally a block of key stream is created when a decrypt is required, each and every byte can then be encrypted or decrypted on its own. Of course, if this is performed in this way depends on the implementation. In principle, large parts or even the entire key stream can be pre-generated. — Jun 20, 2014 at 21:59  
  • +1 – @owlstead Quite correct, but just to be explicit about things - the BC implementation (currently) only offers block-wise encryption/decryption for GCM. — Jun 21, 2014 at 10:36  
  • +0 – @PeterDettman Edited answer to show that this is true for this particular implementation. Thanks for dropping in (as a BC maintainer)! — Jun 21, 2014 at 14:37  
  • +0 – This gets me a bit closer to a solution, but it makes the assumption that I will have the entire ciphertext in a single monolithic buffer. That is not what I was aiming for. If I change the above solution to pull smaller buffers from ciphertext in variable chunk sizes, then the decryption always breaks with an IllegalArgumentException at the time that my buffer rolls, regardless of what chunk size I select. Perhaps it's not possible to do what I want to do with GCM? For now I have switched to use CTR with a SHA-256 digest in the end of the encrypted stream. — Jun 23, 2014 at 12:20  
  • +0 – Update: After a bit more research, I am pretty sure that a CTR encrypted digest is not the correct way to go, and instead I will end my CTR stream with an HmacSHA256 MAC. — Jun 23, 2014 at 14:31