JDK-8130875 : Ucrypto library leaks memory when null output buffer is specified
  • Type: Bug
  • Component: security-libs
  • Sub-Component: java.security
  • Affected Version: 9
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: solaris
  • Submitted: 2015-07-09
  • Updated: 2016-06-13
  • Resolved: 2015-09-04
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
8u72Fixed 9 b82Fixed
Related Reports
Relates :  
With 16g java heap size, after sometime into the runtime of EEJS benchmark, noticed the error that ���There is insufficient memory for the Java Runtime Environment to continue. 
Native memory allocation (malloc) failed to allocate 1640 bytes for AllocateHeap��� with the cipher suite TLS_RSA_WITH_AES_128_GCM_SHA256
Always use non-null output buffer when calling ucrypto for decryption/encryption

Underlying cause is a Solaris bug, however given that the Solaris fix may only be integrated into S12. It'd be nice if java can add a workaround to avoid this for all Solaris releases. Re-open this for implementing the java workaround.

1) OS version is S11U3 build 22 2) It seems the problem only happens in Ucrypto, I tried to disable GCM in Ucrypto and the problem does not exist. We root caused the problem and this native memory allocation failure happens because the process heap keeps growing and it overlaps with the java heap. I tried to increase the process heap by setting HeapBaseMinAddress to 256G, and it does not show the error. The default process heap size is 32G.

To better evaluate this, I'd need the following info: 1) OS version 2) java.security provider config. Is this all sunpkcs11/ucrypto, partially sunpkcs11/ucrypto/sunjce, or all sunjce? 3) A JFR would help see where the memory is been accumulated and GCed. Did this behave better or worse if you did not put a heap limit? Did it fail sooner if the heap limit was smaller?