A DESCRIPTION OF THE PROBLEM :
Traditionally Signature.setParameter() has been called before initSign()/initVerify(). In SignatureScheme.java it is now specified the other way round which results in older providers set up to only do things using the previous convention failing. In this case resulting in a SSLHandshakeException.

There's two things about this:
1. legacy providers that do not support the new approach cannot be used to handle RSA-PSS signatures with the JSSE, current release of the BC and BCFIPS providers are two examples of this.
2. the comments in the code imply something bad will happen if Signature.setParameter() is now called before Signature.initSign(), yet the JavaDoc for the class has not been updated to reflect this, nor does the Signature class prevent developers from doing things the old way. If it really has become the case that setParameter() should only be called after initSign()/initVerify() it needs to be flagged.

That said, at least with the SunRsaSign provider it appears from the results in:

http://mail.openjdk.java.net/pipermail/security-dev/2018-September/018265.html

setParameter() can be called before or after. In the case of  the BC we will try to adapt, but we are aware that there is a lot of legacy code out there which may not be able to, or may need, for other reasons, to stay with old versions of our providers - our FIPS users particularly.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
There is an extremely good analysis of the issue in:

http://mail.openjdk.java.net/pipermail/security-dev/2018-September/018265.html

which includes sample code with instructions for showing the problem.


CUSTOMER SUBMITTED WORKAROUND :
No work around is possible.

FREQUENCY : always