JDK-8088038 : Unexpected color variance in Glass Robot tests on some Mac systems
  • Type: Bug
  • Component: javafx
  • Sub-Component: window-toolkit
  • Affected Version: 8
  • Priority: P4
  • Status: Open
  • Resolution: Unresolved
  • Submitted: 2013-09-14
  • Updated: 2018-09-05
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Relates :  
JDK-8095883 - Change 'toolkit' to be a property of the App, but not Platform
Relates :  
JDK-8118536 - Visual unit tests fail intermittently
Relates :  
JDK-8088910 - [ColorPicker] chosen and shown colors don't match on mac
Relates :  
JDK-8090003 - Color change after drag and drop.
Description
We are getting some unexpected color variance on the Mac Hudson slave system used for nightly testing. When we run these tests on our desktop system we don't see this problem.

Here is the failure we got before we increased the tolerance to 0.03 (we were using 0.01 prior to that).

helloworld.RectangleTest > testSceneFillColor FAILED
    java.lang.AssertionError: expected:<0.5843137502670288> but was:<0.5686274766921997>

helloworld.RectangleTest > testFillRect FAILED
    java.lang.AssertionError: expected:<0.5960784554481506> but was:<0.6117647290229797>

helloworld.RectangleTest > testAddFillRect FAILED
    java.lang.AssertionError: expected:<0.5960784554481506> but was:<0.6117647290229797>
Comments
Moved to Van Ness as we do not know what causes the issue, so it might take a lot of time to handle it.
23-09-2013
I have increased the tolerance to 7% so I think this should not be a problem for us in the short term. If you want to move this to Van Ness that would be fine.
23-09-2013
Thanks for the verification. It matches what I saw on my machine when I selected a different color profile. So the question is: what is different about Jennifer's machine and the Hudson slave that is making them fail?
18-09-2013
The new assertColorEquals() looks good to me, but I was unable to get a test failure to occur when I ran the test so I couldn't test its behavior. Note that you can infer the color component which is failing from the line in the old testColorEquals() method. The tests failing at line 171 are failing on the blue component; and the tests failing on line 169 are failing on the red component - but that is a weak dependency that depends on the current source code being used for the failing run. I just tried mangling my color profile by pulling up the displays Color tab and choosing the calibrate option. I chose the "other whitepoint" and then a "color temperature" that was way different than what it was originally set for. My screen was painfully dully blue while the tests ran, but they still all passed. I also tried the "Generic RGB Profile" and had no failures either.
18-09-2013
Here is the updated "assertColorEquals" method I would like to push. Any comments? private static final String colorToString(Color c) { int r = (int)(c.getRed() * 255.0); int g = (int)(c.getGreen() * 255.0); int b = (int)(c.getBlue() * 255.0); int a = (int)(c.getOpacity() * 255.0); return "rgba(" + r + "," + g + "," + b + "," + a + ")"; } protected void assertColorEquals(Color expected, Color actual, double delta) { double deltaRed = Math.abs(expected.getRed() - actual.getRed()); double deltaGreen = Math.abs(expected.getGreen() - actual.getGreen()); double deltaBlue = Math.abs(expected.getBlue() - actual.getBlue()); double deltaOpacity = Math.abs(expected.getOpacity() - actual.getOpacity()); if (deltaRed > delta || deltaGreen > delta || deltaBlue > delta || deltaOpacity > delta) { throw new AssertionFailedError("expected:" + colorToString(expected) + " but was:" + colorToString(actual)); } }
17-09-2013
Jim: Yes, that is pretty much what I was thinking.
17-09-2013
I see the same results on my Mac as Petr does on his. No difference in behavior based on profile selection.
17-09-2013
Ah, so this could be a case of one color profile causing a difference in one component that is ".6-ish" and the other profile causing a difference in a different component that is "0-ish"...?
17-09-2013
Hmmm. Yes this interesting. No, the tests are not using a random color. It occurred to me last week, and I am reminded today, that printing out which color component failed might be helpful (maybe printing all of them out when any one has a difference of > delta).
17-09-2013
Here is something else really odd. The "was" and "expected" values on the two different runs are radically different on the tests that failed on both. You'd expect one of the values to be different, but not both, and you would expect the values on the different runs to be near each other. They aren't even the same color by a long shot - one goes from 0.0 to 0.03 and the other goes from 0.596 to 0.541??? (It just occurred to me - are the tests using a random color and so run-to-run variations are to be expected in the absolute values?)
17-09-2013
I am wondering why this is not reproducible on my local machines. I've tried different settings and different color profiles - the result of Robot.getPixelColor is always the same.
17-09-2013
So there is definitely something going on with the pixel readback that is affected by color profiles (which seems odd in and of itself). I would note that in the case of Jennifer's system, changing the color profile to "Generic RGB Profile" actually makes it worse. With her former setting, increasing the tolerance from 0.03 to 0.05 will suppress the failure. With the "Generic RGB Profile" setting, the error is greater than 0.05.
17-09-2013
Jennifer, could you please post the detail of your local system and which color profile you are using in Preferences->Displays->Color? The issue is not reproducible on any of the available local machines, so you configuration might be a clue.
17-09-2013
In my system with default display: helloworld.RectangleTest > testFillRect FAILED java.lang.AssertionError: expected:<0.0> but was:<0.03921568766236305> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:441) at org.junit.Assert.assertEquals(Assert.java:510) at testharness.VisualTestBase.assertColorEquals(VisualTestBase.java:171) at helloworld.RectangleTest.access$1400(RectangleTest.java:39) helloworld.RectangleTest > testAddFillRect FAILED java.lang.AssertionError: expected:<0.0> but was:<0.03921568766236305> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:441) at org.junit.Assert.assertEquals(Assert.java:510) at testharness.VisualTestBase.assertColorEquals(VisualTestBase.java:171) at helloworld.RectangleTest.access$2700(RectangleTest.java:39) 194 tests completed, 2 failed, 19 skipped +++++++++++++++++ After changing to Generic RGB Profile: helloworld.RectangleTest > testSceneFillColor FAILED java.lang.AssertionError: expected:<0.3921568691730499> but was:<0.32156863808631897> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:441) at org.junit.Assert.assertEquals(Assert.java:510) at testharness.VisualTestBase.assertColorEquals(VisualTestBase.java:169) at helloworld.RectangleTest.access$700(RectangleTest.java:39) helloworld.RectangleTest > testFillRect FAILED java.lang.AssertionError: expected:<0.5960784554481506> but was:<0.5411764979362488> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:441) at org.junit.Assert.assertEquals(Assert.java:510) at testharness.VisualTestBase.assertColorEquals(VisualTestBase.java:169) at helloworld.RectangleTest.access$1000(RectangleTest.java:39) helloworld.RectangleTest > testAddFillRect FAILED java.lang.AssertionError: expected:<0.5960784554481506> but was:<0.5411764979362488> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:441) at org.junit.Assert.assertEquals(Assert.java:510) at testharness.VisualTestBase.assertColorEquals(VisualTestBase.java:169) at helloworld.RectangleTest.access$1700(RectangleTest.java:39) 194 tests completed, 3 failed, 19 skipped :systemTests:test FAILED
17-09-2013
Kevin, could you please try to change the system color profile on a Hudson machine to "Generic RGB Profile"? To do this you should go to Preferences->Displays->Color, unset the "Show profiles for this display only" checkbox and change the default profile. Would this affect the tests?
16-09-2013
Kevin. You are saying that hudson is running tests without being logged into the system and without a window manager. How is that possible? When I try to log out of my Mac and run the test using ssh I'm getting errors like this: 2013-09-16 17:55:12.011 java[10948:6603] invalid CoreGraphics connection 2013-09-16 17:55:12.012 java[10948:6603] invalid pixel format 2013-09-16 17:55:12.012 java[10948:6603] invalid context Fail in createContext _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL. RenderJob.run: internal exception java.lang.ArrayIndexOutOfBoundsException: 0 at java.util.Arrays$ArrayList.get(Arrays.java:3827) at java.util.Collections$UnmodifiableList.get(Collections.java:1369) at com.sun.glass.ui.Screen.getMainScreen(Screen.java:79) at com.sun.prism.sw.SWPipeline.getDefaultResourceFactory(SWPipeline.java:80) at com.sun.prism.GraphicsPipeline.getDefaultResourceFactory(GraphicsPipeline.java:118) at com.sun.javafx.tk.quantum.QuantumRenderer$3.run(QuantumRenderer.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:129) at java.lang.Thread.run(Thread.java:724) So, how does Hudson manage to get up to comparing colors without failing like this?
16-09-2013
Interestingly, the Glass robot screen capture method is almost copy-pasted from the AWT implementation. At least all the part of the code about actual capturing is the same. Do we have the same problems in JDK?
16-09-2013
The failing tests are located here: rt/tests/system/src/test/java/helloworld/RectangleTest.java If you change the 0.03 tolerance back to 0.01 and run them on our nightly test machine, it will fail with the above error. I took a quick look at what Robot is using to read the pixels: CGDataProviderCopyData with a provider created from CGWindowListCreateImage. What I find interesting is that the screenCapture method, which returns a rectangle of pixels, specifies a parameter of kCGColorSpaceGenericRGB to CGBitmapContextCreate(). Not sure if that's relevant. Also, it's odd that when running on our desktop / laptop systems, the test runs fine as is. One possible difference that may or may not be relevant: we are running our nightly tests without being logged into the system or running the window manager.
14-09-2013