Bug ID: JDK-8358957 [ubsan]: The assert in layout_helper_boolean

Type: Enhancement
Component: hotspot
Sub-Component: runtime

Priority: P4
Status: In Progress
Resolution: Unresolved

Submitted: 2025-06-09
Updated: 2025-11-04

Other
tbdUnresolved

In layout_helper_boolean_diffbit(), inside the loop that is calculating the result, there is an assert that diffbit (the in-progress calculated result) is not zero. But it can never be zero in the absence of UB. It is initialized to 1, and then left shifted 1 for each iteration. That bit could be shifted all the way to the sign bit, so that the value is INT_MIN. But another left shift, which naively would result in zero, is actually UB.

Reopening; the rationale for closing is incorrect. Where I said > It probably doesn't fail currently because of JDK-8314258. "It" is the checked_casts in the suggested change, NOT the problem that is the topic of this issue.
21-10-2025
The related issue 8314258 was the root cause of this issue.
20-10-2025
I've been studying layout_helper_boolean_diffbit, and it has other problems besides the assert being potentially optimized away. I think it is also algorithmically suspect, and only works by accident. If the array_layout were different, or if the values of T_BOOLEAN and T_BYTE were different it could produce the wrong result, or assert, or maybe even loop forever if the assert were optimized away. Based on usage, what we want from it is a mask containing a single bit which is set in array_layout_helper(T_BOOLEAN) and clear in array_layout_helper(T_BYTE). What it's actually computing is the lowest bit that is either different between alh(T_BOOLEAN) and alh(T_BYTE), or set in alh(T_BOOLEAN). That just happens to be the same bit as what we actually want, given the array_layout format and the values of T_BYTE and T_BOOLEAN. But there is a much simpler and more direct method for obtaining the value we want, without any looping at all. {@code} uint zlh = checked_cast<uint>(array_layout_helper(T_BOOLEAN)); uint blh = checked_cast<uint>(array_layout_helper(T_BYTE)); // get all the bits that are set in zlh and clear in blh uint candidates = (zlh & ~blh); assert(candidates != 0, "must be"); // must be some if there is a solution. // Use well known bit hack to isolate the low bit of candidates. // The usual form is (x & -x), but VS warns (C4146) about unary minus of unsigned. // So use alternate form of negation to avoid warning. uint result = candiates & (~candidates + 1); return static_cast<int>(result); {@code} [later] Use of check_cast is probably wrong. I think an alh is negative. Oops, my mistake. It probably doesn't fail currently because of JDK-8314258.
15-09-2025
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk/pull/27288 Date: 2025-09-15 09:01:56 +0000
15-09-2025
I rewrote the description of this issue to more accurately describe the problem.
11-09-2025
[~kbarrett], thank you for your feedback. The link is corrected and description is filled now. I wanted to get your comments on my understanding and then fill in the description field.
09-09-2025
During discussion on https://github.com/openjdk/jdk/pull/24184 , [~kbarrett] pointed out this issue. IIUC and IIRC, the left-shift of signed int is UB when sign-bit is set (==1). The assertion works only if the sign bit is discarded during shift op.
09-09-2025
[~azafari] That information belongs in the bug description, not in a comment. And really, the description should be self-contained, rather than linking to a PR comment. And I get a 404 when clicking on the link, because for some reason the following comma is being included in the link.
08-09-2025