In awt_new.cpp, AWT has a handler for memory allocation. When an allocation
fails (either malloc or new), this handler is invoked. It tries to compact
the heap by invoking the garbage collector, and if successful, tries the
memory allocation again. This may have made sense in the days when we were
using the Symantec JIT, but it doesn't anymore. In fact, it has caused
problems for the VM team. They have asked that the handler be removed.
Throwing an OutOfMemory Exception is fine, but invoking the garbage collector
is not.
The handler function is:
// Called when malloc or operator new runs out of memory. We try to
// compact the heap by initiating a Java GC. If the amount of free
// memory available after this operation increases, then we return
// (1) to indicate that malloc or operator new should retry the
// allocation. Returning (0) indicates that the allocation should fail.
int
NewHandler::handler(size_t) {