United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-6860824 serialized object causes StackOverflowError independently of -Xss setting on Windows
JDK-6860824 : serialized object causes StackOverflowError independently of -Xss setting on Windows

Details
Type:
Bug
Submit Date:
2009-07-15
Status:
Resolved
Updated Date:
2011-08-17
Project Name:
JDK
Resolved Date:
2011-05-17
Component:
core-libs
OS:
windows_xp
Sub-Component:
java.io:serialization
CPU:
x86
Priority:
P2
Resolution:
Fixed
Affected Versions:
1.4.2_21
Fixed Versions:
1.4.2_33 (b01)

Related Reports

Sub Tasks

Description
J2RE 1.4.2 Update 13, 19, and 21 gives java.lang.StackOverflowError, when 
trying to deserialize a serialized object, independently of -Xss setting.

The issue is strictly reproducible on Windows platform:
 
> java -Xss20m -classpath ./classes;./lib/jaxrpc-api.jar;./lib/s
ap.com~xapps~osp~infra~ejbmodule_assembly.jar;./lib/sap.com~xapps~osp~infra~serv
ices~ejbmodule.jar;./lib/sap.com~xapps~osp~itemhandler~ejbmodule.jar;./lib/webse
rvices_api.jar;./lib/sap.com~xapps~osp~formatter~wsproxy.jar;./lib/webservices_l
ib.jar;./lib/logging.jar;./lib/exception.jar;./lib/sapxmltoolkit.jar test.jms.De
serializeTest 11470100_bad.ser
Exception in thread "main" java.lang.StackOverflowError
       at java.io.ObjectInputStream$PeekInputStream.read(Unknown Source)
       at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
       at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source
)
       at java.io.ObjectInputStream.readHandle(Unknown Source)
       at java.io.ObjectInputStream.readClassDesc(Unknown Source)
       at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
       at java.io.ObjectInputStream.readObject0(Unknown Source)
       at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
       at java.io.ObjectInputStream.readSerialData(Unknown Source)
       at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
       at java.io.ObjectInputStream.readObject0(Unknown Source)
       at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
       at java.io.ObjectInputStream.readSerialData(Unknown Source)
       at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
       at java.io.ObjectInputStream.readObject0(Unknown Source)
       at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
       at java.io.ObjectInputStream.readSerialData(Unknown Source) 
[ ... ]
>

                                    

Comments
SUGGESTED FIX

webrev for jdk4:
http://jpsesvr.sfbay.sun.com:8080/ctetools/html/ViewDetail.jsp?index=4050

integration request:
https://jetsvr.sfbay.sun.com:8443/BugApproval/ViewDetail.jsp?index=10615&mode=view
                                     
2010-05-03
EVALUATION

Attached test case contains 11470100_bad.ser file which is serialized
com.sap.xapps.osp.itemhandler.common.RequestMessage object.
This is a complex object so in order to be deserialized it requires
java.io.ObjectInputStream.readOrdinaryObject() method
to be recursively called.
The recursion depth is about 4000 so it can't be handled by a default JVM stack size which is less then 1MB.
Required stack size to complete the deserialization is 4MB.

The problem is that -Xss parameter doesn't work on Windows for a main Java thread in Java4 and Java5
see bugs:

4854196 -Xss does not work on Win32 causing Java application to crash(Stack Overflow)
4689767 main thread stack size not the same as other threads on all platforms (-Xss<N>)
6316197 Java launcher should create JVM from non-primordial thread
6316878 JVM should use STACK_SIZE_PARAM_IS_A_RESERVATION when creating threads on Windows

This problem has been fixed in Java6 but the fix has never been backported to Java5 and Java4.

The workaround for now is to launch the test case from non-primordial thread. See workaround section.
                                     
2009-09-11
WORK AROUND

The workaround is to launch the test case from non-primordial thread.
Attached is workarond.zip which contains test.jms.DeserializeTest.java
which launches the test case from non-primordial thread.
Please replace this file in original test case. 
Serialized in com.sap.xapps.osp.itemhandler.common.RequestMessage object (in 11470100_bad.ser file) will get properly deserialized if stack size is set to 4MB.
                                     
2009-09-14



Hardware and Software, Engineered to Work Together