Bug ID: JDK-8094590 Mac: Intermittent build failures in task graphics:ccMacFontT2K

Type: Bug
Component: javafx
Sub-Component: build
Affected Version: 8

Priority: P3
Status: Resolved
Resolution: Fixed

Submitted: 2014-02-20
Updated: 2024-09-20
Resolved: 2014-03-27

JDK 8
8u11Fixed

There is another bug on a concurrent modification bug in this same module (RT-34437), but this problem has a different final cause listed in the error output as seen below:

-----------------------------
:graphics:ccMacFontT2K FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':graphics:ccMacFontT2K'.
> Could not call NativeCompileTask.compile() on task ':graphics:ccMacFontT2K'

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
-----------------------------

Those error lines are accompanied by 4000 lines of error messages which are attached in a text file.

http://hg.openjdk.java.net/openjfx/8u-dev/rt/rev/86830871c071
27-03-2014
This bug is enough of a bother that I am going to push the fix now so our nightly Hudson builds will stop failing randomly. Developers can always set NUM_COMPILE_THREADS=8 in their gradle.properties file or pass "-PNUM_COMPILE_THREADS=8" on the command line. I filed RT-36433 to track solving the root cause.
27-03-2014
Yeah, it's definitely odd. I showed it to Phil (since it is T2K code getting hit), and he thinks it might be a resource issue of some sort. The error message suggests that the compiler fails to start up properly. Btw, I checked and our Hudson slaves report 16 available processors so that is the number of compile jobs that the native cc tasks will execute concurrently. I left it running on my machine for about 90 minutes in a loop (118 passes of nativeT2K completed) with NUM_COMPILE_THREADS=1 with no problems. SInce we don't know the root cause, though, this may be more of a workaround than a fix. It is odd that we've never seen this on any other platform, all of which compile the same T2K code, nor have we yet seen it with any other native module. There is more native code in the T2K directory than any other single module, for those that use parallel compilation, for whatever that's worth.
27-03-2014
It's really odd that compiling multiple files could cause a parallelism problem, unless the code there is used to generate other code, but that wouldn't be a gcc task.
27-03-2014
I'm going to let the build run in a loop for another hour (it's up to 50 iterations with no failures so far). Here is the proposed patch: diff --git a/build.gradle b/build.gradle --- a/build.gradle +++ b/build.gradle @@ -382,7 +382,12 @@ ext.DO_JCOV = Boolean.parseBoolean(JCOV) // Define the number of threads to use when compiling (specifically for native compilation) -defineProperty("NUM_COMPILE_THREADS", "${Runtime.runtime.availableProcessors()}") +// On Mac we limit it to 1 by default due to problems running gcc in parallel +if (IS_MAC) { + defineProperty("NUM_COMPILE_THREADS", "1") +} else { + defineProperty("NUM_COMPILE_THREADS", "${Runtime.runtime.availableProcessors()}") +} // // The next three sections of properties are used to generate the @@ -740,6 +745,7 @@ logger.quiet("jdk version: ${jdkVersion}") logger.quiet("jdk build number: ${jdkBuildNumber}") logger.quiet("minimum java build number: ${jfxBuildJdkBuildnumMin}") +logger.quiet("NUM_COMPILE_THREADS: $NUM_COMPILE_THREADS") logger.quiet("COMPILE_TARGETS: $COMPILE_TARGETS") logger.quiet("COMPILE_FLAGS_FILES: $COMPILE_FLAGS_FILES") logger.quiet("HUDSON_JOB_NAME: $HUDSON_JOB_NAME")
27-03-2014
Confirmed: it is failing in the :graphics:ccMacFontT2K task and never making it to the :graphics:linkMacFontT2K task. I'd like to push the simple fix to set the default number of compile threads to 1. A possibly better fix would be to override the number of thread (and force it to 1) for just the nativeFontT2K task on Mac. That way compiling PrismES2 and Glass would still benefit from parallel compilation (and I've never seen it fail anywhere but FontT2K). I can file a follow-on issue for that.
27-03-2014
The parallelism is only within each native stage. The linkTask "dependsOn" the ccTask so unless something is going wrong with the dependencies, it seems that just having the ccTask run compilations in parallel is sufficient to cause the bug. I will double-check this as I saved a log file from a failing build run with "gradle --debug".
27-03-2014
Isn't there a way to detect or specify which gcc invocations can be done in parallel? My impression was that the files needed to be built in stages and that occasionally a file from the next stage was starting to be processed before a file from a prior stage (more specifically, a link task was started before the compile tasks were done?).
27-03-2014
Note that on my machine setting -PNUM_COMPILE_THREADS=1 adds about 35 seconds to the build time, as measured by "gradle cleanNative native".
27-03-2014
The problem is somehow related to parallel execution of gcc. The NUM_COMPILE_THREADS property in our build.gradle file is used by NativeCompileTask (defined in "buildSrc/src/main/groovy/...") to determine the number of worker threads to run in parallel. The individual gcc commands are exec'd by a worker thread. By default NUM_COMPILE_THREADS is set to the number of processors (cores) on the build machine. This number is 8 on my system, and likely the same on our Hudson build machines (I will check). I plan to fix this by setting the default number of compile threads to 1 on Mac OS X. I did several test runs (in a loop) of just building the nativeFontT2K task and it fails on my machine eventually. Setting it 1 by doing "-PNUM_COMPILE_THREADS=1" appears to resolve the problem. I want to let it run for longer to be sure I'm not just getting lucky.
27-03-2014
We've started seeing this more often, so I am bumping the priority.
07-03-2014
Some notes on my build environment: MacOS 10.8.5 Java 1.8.0-ea-b121 XCode Version 4.6.3 (4H1503)
03-03-2014

Relates :	JDK-8089417 - Mac: intermittent T2K gcc failure prevents parallel compilation
Relates :	JDK-8339505 - Enable parallel compilation of native code on macOS
Relates :	JDK-8097343 - Mac: Intermittent build failures when compiling T2K native code on Mac OS X