I've noticed that CustomColorDialog appears shifted from the right edge of ColorPalette PopupWindow. In fact, my investigation shows that PopupWindow x, y, width, height rectangle is shifted to the right and bottom from the actual position. 

To illustrate that I've changed ColorPalette code so when you click on Custom Color link it opens four stages each is expected to be adjacent to Palette window. In fact, only top and left are adjucent while bottom and right are at a distance. See attached screenshot.

The problem is caused by PopupWindow.syncWithRootBounds() method which does some "black" magic behind the scenes:

        // update popup dimensions
        setWidth(layoutBounds.getMaxX() - layoutBounds.getMinX());
        setHeight(layoutBounds.getMaxY() - layoutBounds.getMinY());
        // update transform
        rootNode.setTranslateX(-layoutBounds.getMinX());
        rootNode.setTranslateY(-layoutBounds.getMinY());
        if (alignWithContentOrigin) {
            // update window position
            setWindowTranslate(layoutBounds.getMinX(), layoutBounds.getMinY());
            ...
        }

The root cause is that root node has shadow and the shadow contributes to width and height of the PopupWindow but doesn't affect its x and y.