United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-7031830 bad_record_mac failure on TLSv1.2 enabled connection with SSLEngine
JDK-7031830 : bad_record_mac failure on TLSv1.2 enabled connection with SSLEngine

Details
Type:
Bug
Submit Date:
2011-03-28
Status:
Closed
Updated Date:
2013-06-19
Project Name:
JDK
Resolved Date:
2011-10-29
Component:
security-libs
OS:
linux,windows_xp
Sub-Component:
javax.net.ssl
CPU:
x86
Priority:
P2
Resolution:
Fixed
Affected Versions:
6,7
Fixed Versions:

Related Reports
Backport:
Duplicate:

Sub Tasks

Description
FULL PRODUCT VERSION :
java version "1.7.0-ea"
Java(TM) SE Runtime Environment (build 1.7.0-ea-b134)
Java HotSpot(TM) Client VM (build 21.0-b04, mixed mode, sharing)

A DESCRIPTION OF THE PROBLEM :
About sun.security.ssl.CipherBox class'  "decrypt(ByteBuffer)" function : if a ByteBuffer(created by wraping a byte array) is passed to this function, it does array copy on this internal byte array without considering offset of the wrapper ByteBuffer.

As a result, copying on internal byte array is improper so data and mac values is corrupted. It fails in mac verification with bad_record_mac error.

ps: This problem also applies to encrypt(ByteBuffer) function of the same class.

 int decrypt(ByteBuffer bytebuffer) throws BadPaddingException
    {
        int i = bytebuffer.remaining();
        if(cipher == null)
        {
            bytebuffer.position(bytebuffer.limit());
            return i;
        }
        try
        {
            int pos = bytebuffer.position();
            ByteBuffer bytebuffer1 = bytebuffer.duplicate();
            int k = cipher.update(bytebuffer1, bytebuffer);
            if(k != i)
                throw new RuntimeException(/*...*/);
			
	    //some debug logging operation
            if(blockSize != 0)
            {
                bytebuffer.position(pos);
                k = removePadding(bytebuffer, blockSize, protocolVersion);
                if(protocolVersion.v >= ProtocolVersion.TLS11.v)
                {
                    if(k < blockSize)
                        throw new BadPaddingException("invalid explicit IV");
                    Object obj = null;
                    int lim = bytebuffer.limit();
                    if(bytebuffer.hasArray())
                    {
                        byte inarray[] = bytebuffer.array();
		        /*HERE IS THE FAULTY PART*/
                        System.arraycopy(inarray, pos + blockSize, inarray, pos, lim - pos - blockSize);
                        bytebuffer.limit(lim - blockSize);
                    } else
                    {
                        byte inarray[] = new byte[lim - pos - blockSize];
                        bytebuffer.position(pos + blockSize);
                        bytebuffer.get(inarray);
                        bytebuffer.position(pos);
                        bytebuffer.put(inarray);
                        bytebuffer.limit(lim - blockSize);
                    }
                    lim = bytebuffer.limit();
                    bytebuffer.position(lim);
                }
            }
            return k;
        }
        catch(ShortBufferException shortbufferexception)
        { /*error handling*/ }
    }

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Any TLSv1.2 protocol enabled client-server application with NIO(SSLEngine)!


REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
System.arraycopy(inarray, pos + blockSize, inarray, pos, lim - pos - blockSize);

part can be replaced with:

System.arraycopy(inarray, pos + offset + blockSize, inarray, pos + offset, lim - pos - blockSize);

                                    

Comments
PUBLIC COMMENTS

The position of a byte buffer is the offset.

Considering the bad_record_mac failure, we need more information to tell what the underlying problem.
                                     
2011-03-29
EVALUATION

When using a ByteBuffer that is backed by an array, once you have the array, you need to offset into that array to get to the real first byte of the ByteBuffer.

In just quick testing, ByteBuffer.allocate(20/20000) will return byte backed arrays on solaris/linux/windows, and allocatedDirect(20/20000) will not.  So if you're using Direct byte buffers, you probably will not run into this bug.
                                     
2011-10-12
EVALUATION

With fix, customer reports that no errors remain.  nodemanager tests passes.
                                     
2011-10-12
EVALUATION

Workaround are to not use TLS v1.1/1.2, or use non-direct byte buffers, but neither of these are very good options.
                                     
2011-10-15
EVALUATION

Fix is ready to go for 8 and 7u2.  Just waiting on JPRT jobs finishing for 8, and then approvals for 7u2.
                                     
2011-10-18
verified on jsn-vw1-00-vm1 by test 
sun/security/ssl/com/sun/net/ssl/internal/ssl/SSLEngineImpl/SSLEngineBadBufferArrayAccess.java

B11 and before test failed.
B12 and after test success
                                     
2013-06-19



Hardware and Software, Engineered to Work Together