We should add a compilation timeout flag to the VM to catch long running compilations. We could piggy-back on -XX:+CITime

--- a/src/hotspot/share/runtime/timerTrace.cpp
+++ b/src/hotspot/share/runtime/timerTrace.cpp
@@ -75,6 +75,9 @@ TraceTime::~TraceTime() {
   if (_accum != NULL) {
     _accum->add(_t);
   }
+  if (_t.seconds() > 3) {
+    assert(false, "Compilation timeout in phase %s", _title);
+  }
   if (!_verbose) {
     return;
   }

As part of this, or with a separate change, we might also want to improve CITime/CITimeVerbose/CITimeEach to narrow down the cause of the timout.

The disadvantage of above solution is that it would not catch endless looping compilations. For that, we would need a watcher thread (or maybe we can use an existing VM thread for that?).