Quick summary, more details on GitHub PR description:
- We have an issue with the mix of token (deferred) and immediate-return-queries. For example "sample" (returns immediately in lambda execution) can float above "addDataName" (token, deferred to token evaluation). That is confusing and should be fixed.
- We need to turn (almost) all immediate-return-queries into token based queries. That way, we can make sure that lambda and token order are essentially synchronized, and we get rid of the "floating" issue.
- We should make the use of scopes explicit. We need scopes anyway now, so that we can limit the availability of hashtags to the scope of a token based query.

---------------- OLD REPORT BELOW ---------------------------

Currently, we have an issue that this pattern fails:

addDataName("x", myType),
dataNames(MUTABLE_OR_IMMUTABLE).exactOf(myType).sample()

The issue is that addDataName creates a token that is not immediately evaluated, the adding of "x" is deferred to token evaluation.
But "sample" is done in the lambda execution, i.e. immediately. And so it does not find "x" at that time.

We should make addDataName evaluate immediately as well, and not deffer to token evaluation.
However: this would mean that addDataName could float outside of a "hook.anchor":

hook.anchor(
    addDataName("x", myType), // is evaluated before we call hook.anchor, so it floats to the outer scope, not good!
    dataNames(MUTABLE_OR_IMMUTABLE).exactOf(myType).sample() // now finds "x", good
)
dataNames(MUTABLE_OR_IMMUTABLE).exactOf(myType).sample() // also finds "x", not good!

Possible solution: require hook.anchor to consume a lambda. That way, we execute hook.anchor before it then calls the lambda. This allows us to set up the scope before executing anything inside the hook.anchor body, and it prevents the executions inside hook.anchor from "floating" out of the scope.

I'll need to experiment a little, but I hope this works. I'll have to adjust a lot of tests probably, but it will be worth it.

Thanks to [~galder] who tried out the Template Framework, and hit this issue. Basically, he was just confused about the order of execution, and it showed that we probably could not keep things as they are now. Understanding lambda vs token vs string orders is too much. I hope this change will allow us to mostly have lambda vs string order.

The 3 orders:
- Lambda: execution inside the lambda
- Token: executed after lambda
- String: may be different from token, because of hook.insert

Thanks to [~chagedorn] for taking some time to brain-strom with me in the Office :)

WIP DRAFT:
https://github.com/openjdk/jdk/pull/27255

----------------------------------

Some more thoughts. The current issue is the interleaving between lambda and token eval.
The issue is also with the TemplateToken: an addDataName would float above it.
This means that we have to probably eliminate the TemplateToken (and also HookAnchorToken and HookInsertToken).
Instead, we will have to do:
- template.call()
- Template.TwoArgs -> template.apply(x, y) -> Template.ZeroArgs -> template.call()
- Template.TwoArgs -> template.call(x, y)

This gives us a much more "immediate" lambda nesting structure.
Probably, we will have Template.ZeroArgs be an interface, and the multi-args versions will have an "applied" counterpart that implements it. Or maybe we can wrap the lambda? Probably! That would allow us to do essentially arbitrary partial application ... but that's probably not too useful for now. Most useful is to have List<Template.ZeroArgs>.

This is really going to be a major refactoring. But it would massively reduce the complexity.
It means we have no tokens that are "alive" and can call lambdas later.
All tokens would be "dead", at most they will do hashtag and dollar replacements.

------------------------

I now realized that the "all-in on lambda" approach does not really play so nicely with the token list. We would probably have to redesign everything in a StringBuilder approach, which also is not very nice. So I decided to go "all-in on tokens".