FULL PRODUCT VERSION :
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) Server VM (build 24.45-b08, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
SunOS enok 5.10 Generic_142910-17 i86pc i386 i86pc


A DESCRIPTION OF THE PROBLEM :
Type of Service (TOS) can generally not be set in a IPv6 header from Java when using TCP. Also for IPv4 some tweaks have to be performed in order to get all cases working. Extensive testing has been done for different cases. Please consider all these cases. Testing was done by collecting snoops and analyzing the resulting ip headers of the TCP and UDP communication between Java clients and servers.

These are the results for TCP and UDP:

TCP:
IPv6 (java.net API blocking IO)
1.  IPv6 Client sockets
    The TOS field can be set for IPv6 client sockets.
2.  IPv6 Server sockets
    The TOS field cannot be set for IPv6 server sockets.

IPv6 (java.nio API non-blocking IO)
3.  The TOS field cannot be set

IPv4 (java.net API blocking IO)
4.  IPv4 Client sockets
    The TOS field can be set when using IPv4 only stack or when using IPv4/IPv6 dual stack with -Djava.net.preferIPv4Stack set to true.
    
5.  IPv4 Server sockets
    The TOS field will only be set after initial TCP handshake is completed.
    Reflection has to be used to set the TOS field for the full TCP stream. This can be done on the server socket before binding.

IPv4 (java.nio API non-blocking IO)
6.  The TOS field cannot be set.

UDP:
No issue.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Start the snoop
2. Start a TCP server socket or channel and set a non 0 traffic class using the java.net.Socket API (setTrafficClass).
3. Start a TCP client socket or channel and set a non 0 traffic class using the java.net.Socket API (setTrafficClass).
4. Send a packet from client to server.
5. Stop the snoop
6. Examine the IP header field of all packages to the port of the server node with Wireshark or an automated tool.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Critical priority
Expected result is that the TOS is correctly set for IPv4 and IPv6 for both blocking (Sockets) and non-blocking IO (Channels) for the packets of the entire TCP stream (see cases listed above).
High priority
This functionality should be readily available without using system properties like java.net.preferIPv4Stack and without using reflection.
ACTUAL -
For TCP the TOS field can only be correctly set when using IPv4 only stack or when using IPv4/IPv6 dual stack with -Djava.net.preferIPv4Stack set to true. When acting as server reflection has to be used to correctly set the TOS field for the packets of the entire TCP stream.
For TCP and IPv6 the TOS field can only be set for client sockets (blocking IO)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
IPv6 (java.net API blocking IO)
1. The TOS field can be set for unconnected (client) IPv6 sockets

Socket socket = mySocketFactory.createSocket();
socket.setTrafficClass(<tos>);
socket.bind(<local IPv6 socket address>);
socket.connect(<remote IPv6 socket address>, <timeout>);
socket.setTrafficClass(<tos>);

2. The TOS field cannot be set for connected (server) IPv6 sockets

ServerSocketFactory socketFactory = ServerSocketFactory.getDefault();
ServerSocket serverSocket = socketFactory.createServerSocket();
serverSocket.bind(<local IPv6 socket address>);
...
Socket socket = serverSocket.accept();
socket.setTrafficClass(<tos>);

It has also been tried to set the TOS directly on the server socket using reflection but with the same result:

ServerSocketFactory socketFactory = ServerSocketFactory.getDefault();
ServerSocket serverSocket = socketFactory.createServerSocket();

try {
    Class<?> c = serverSocket.getClass();
    while (!ServerSocket.class.equals(c)) {
        c = c.getSuperclass();
    }
    // Use reflection to set the TOS value directly on the server socket
    Method m = c.getDeclaredMethod("getImpl", new Class[] {});
    if (m != null) {
        m.setAccessible(true);
        Object o = m.invoke(socket, new Object[] {});
        if (o instanceof SocketImpl) {
            SocketImpl impl = (SocketImpl) o;
            impl.setOption(SocketOptions.IP_TOS, new Integer(trafficClass));
        }
        m.setAccessible(false);
    }
} catch (Exception e) {
    //Could not set traffic class
    ...
}
...
Socket socket = serverSocket.accept();

IPv6 (java.nio API non-blocking IO)
3.  The TOS field cannot be set.

Client socket:

SocketChannel channel = SocketChannel.open();
channel.configureBlocking(false);
Socket socket = channel.socket();
socket.setTrafficClass(<tos>);

Server socket

Selector selector = Selector.open();
...
Set<SelectionKey> eventSet = selector.selectedKeys();
if (eventSet.size() > 0) {
    Iterator<SelectionKey> keys = eventSet.iterator();
    while (keys.hasNext()) {
        SelectionKey key = keys.next();
        try {
            if (key.attachment() == null && key.isAcceptable()) {
                SocketChannel channel = ((ServerSocketChannel)key.channel()).accept();
                if (channel != null) {
                    ...
                    channel.configureBlocking(false);
                    Socket socket = channel.socket();
                    if (socket != null) {
                        socket.setTrafficClass(<tos>);
                        ...
                    }
                    ...
                }
            }
            ...
        }
        ...
    }
    ...
}

IPv4 (java.net API blocking IO)

4.  IPv4 Client sockets
    The TOS field can be set when using unconnected IPv4 only stacks or when using IPv4/IPv6 dual stacks if -Djava.net.preferIPv4Stack is set to true.

Socket socket = mySocketFactory.createSocket();
socket.setTrafficClass(<tos>);
socket.bind(<local IPv6 socket address>);
socket.connect(<remote IPv6 socket address>, <timeout>);
socket.setTrafficClass(<tos>);
    
5.  IPv4 Server sockets
    The TOS field will only be set after initial TCP handshake is completed.
    

ServerSocketFactory socketFactory = ServerSocketFactory.getDefault();
ServerSocket serverSocket = socketFactory.createServerSocket();
serverSocket.bind(<local IPv6 socket address>);
...
Socket socket = serverSocket.accept();
socket.setTrafficClass(<tos>);

Reflection has to be used to set the TOS field for the full TCP stream. This can be done on the server socket before binding:

ServerSocketFactory socketFactory = ServerSocketFactory.getDefault();
ServerSocket serverSocket = socketFactory.createServerSocket();

try {
    Class<?> c = serverSocket.getClass();
    while (!ServerSocket.class.equals(c)) {
        c = c.getSuperclass();
    }
    // Use reflection to set the TOS value directly on the server socket
    Method m = c.getDeclaredMethod("getImpl", new Class[] {});
    if (m != null) {
        m.setAccessible(true);
        Object o = m.invoke(socket, new Object[] {});
        if (o instanceof SocketImpl) {
            SocketImpl impl = (SocketImpl) o;
            impl.setOption(SocketOptions.IP_TOS, new Integer(trafficClass));
        }
        m.setAccessible(false);
    }
} catch (Exception e) {
    //Could not set traffic class
    ...
}

...
Socket socket = serverSocket.accept();

IPv4 (java.nio API non-blocking IO)
6.  The TOS field cannot be set.

Client socket:

SocketChannel channel = SocketChannel.open();
channel.configureBlocking(false);
Socket socket = channel.socket();
socket.setTrafficClass(<tos>);

Server socket

Selector selector = Selector.open();
...
Set<SelectionKey> eventSet = selector.selectedKeys();
if (eventSet.size() > 0) {
    Iterator<SelectionKey> keys = eventSet.iterator();
    while (keys.hasNext()) {
        SelectionKey key = keys.next();
        try {
            if (key.attachment() == null && key.isAcceptable()) {
                SocketChannel channel = ((ServerSocketChannel)key.channel()).accept();
                if (channel != null) {
                    ...
                    channel.configureBlocking(false);
                    Socket socket = channel.socket();
                    if (socket != null) {
                        socket.setTrafficClass(<tos>);
                        ...
                    }
                    ...
                }
            }
            ...
        }
        ...
    }
    ...
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
No known workaround.