JDK-7031830 : bad_record_mac failure on TLSv1.2 enabled connection with SSLEngine
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.net.ssl
  • Affected Version: 6,7
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: linux,windows_xp
  • CPU: x86
  • Submitted: 2011-03-28
  • Updated: 2015-12-16
  • Resolved: 2011-10-29
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 6 JDK 7 JDK 8
6u111Fixed 7u2Fixed 8 b12Fixed
Related Reports
Duplicate :  
Description
FULL PRODUCT VERSION :
java version "1.7.0-ea"
Java(TM) SE Runtime Environment (build 1.7.0-ea-b134)
Java HotSpot(TM) Client VM (build 21.0-b04, mixed mode, sharing)

A DESCRIPTION OF THE PROBLEM :
About sun.security.ssl.CipherBox class'  "decrypt(ByteBuffer)" function : if a ByteBuffer(created by wraping a byte array) is passed to this function, it does array copy on this internal byte array without considering offset of the wrapper ByteBuffer.

As a result, copying on internal byte array is improper so data and mac values is corrupted. It fails in mac verification with bad_record_mac error.

ps: This problem also applies to encrypt(ByteBuffer) function of the same class.

 int decrypt(ByteBuffer bytebuffer) throws BadPaddingException
    {
        int i = bytebuffer.remaining();
        if(cipher == null)
        {
            bytebuffer.position(bytebuffer.limit());
            return i;
        }
        try
        {
            int pos = bytebuffer.position();
            ByteBuffer bytebuffer1 = bytebuffer.duplicate();
            int k = cipher.update(bytebuffer1, bytebuffer);
            if(k != i)
                throw new RuntimeException(/*...*/);
			
	    //some debug logging operation
            if(blockSize != 0)
            {
                bytebuffer.position(pos);
                k = removePadding(bytebuffer, blockSize, protocolVersion);
                if(protocolVersion.v >= ProtocolVersion.TLS11.v)
                {
                    if(k < blockSize)
                        throw new BadPaddingException("invalid explicit IV");
                    Object obj = null;
                    int lim = bytebuffer.limit();
                    if(bytebuffer.hasArray())
                    {
                        byte inarray[] = bytebuffer.array();
		        /*HERE IS THE FAULTY PART*/
                        System.arraycopy(inarray, pos + blockSize, inarray, pos, lim - pos - blockSize);
                        bytebuffer.limit(lim - blockSize);
                    } else
                    {
                        byte inarray[] = new byte[lim - pos - blockSize];
                        bytebuffer.position(pos + blockSize);
                        bytebuffer.get(inarray);
                        bytebuffer.position(pos);
                        bytebuffer.put(inarray);
                        bytebuffer.limit(lim - blockSize);
                    }
                    lim = bytebuffer.limit();
                    bytebuffer.position(lim);
                }
            }
            return k;
        }
        catch(ShortBufferException shortbufferexception)
        { /*error handling*/ }
    }

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Any TLSv1.2 protocol enabled client-server application with NIO(SSLEngine)!


REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
System.arraycopy(inarray, pos + blockSize, inarray, pos, lim - pos - blockSize);

part can be replaced with:

System.arraycopy(inarray, pos + offset + blockSize, inarray, pos + offset, lim - pos - blockSize);

Comments
SQE OK to take the fix in jdk6u111 in order to complete TLSv1.1 support.
10-12-2015

verified on jsn-vw1-00-vm1 by test sun/security/ssl/com/sun/net/ssl/internal/ssl/SSLEngineImpl/SSLEngineBadBufferArrayAccess.java B11 and before test failed. B12 and after test success
19-06-2013

EVALUATION Fix is ready to go for 8 and 7u2. Just waiting on JPRT jobs finishing for 8, and then approvals for 7u2.
18-10-2011

EVALUATION Workaround are to not use TLS v1.1/1.2, or use non-direct byte buffers, but neither of these are very good options.
15-10-2011

EVALUATION With fix, customer reports that no errors remain. nodemanager tests passes.
12-10-2011

EVALUATION When using a ByteBuffer that is backed by an array, once you have the array, you need to offset into that array to get to the real first byte of the ByteBuffer. In just quick testing, ByteBuffer.allocate(20/20000) will return byte backed arrays on solaris/linux/windows, and allocatedDirect(20/20000) will not. So if you're using Direct byte buffers, you probably will not run into this bug.
12-10-2011

PUBLIC COMMENTS The position of a byte buffer is the offset. Considering the bad_record_mac failure, we need more information to tell what the underlying problem.
29-03-2011