Code review showed that on Linux, if heap is allocated via reserve_memory_special_huge_tlbfs_mixed(), mmap(addr, MAP_FIXED) gets called to establish heap memory which will trash existing mappings which happen to be located in that area.
I could also reproduce this experimentally.
The error is that reserve_memory_special_huge_tlbfs_mixed() calls os::reserve_memory() with a requested_addr != null to establish an initial memory mapping where none existed before.
It might be smart to get rid of the requested_addr parameter to os::reserve_memory altogether, see http://comments.gmane.org/gmane.comp.java.openjdk.hotspot.devel/18049. I don't think there is a way this parameter can be used correctly in this API.