The message digest is incorrectly truncated for ECDSA signatures when the bit length of the base point order, n, is not equal to the field size.
In sun/security/ec/impl/ec.c ECDSA_SignDigestWithSeed() lines 755-760, the digest is truncated using the following code:
/* In the definition of EC signing, digests are truncated
* to the length of n in bits.
* (see SEC 1 "Elliptic Curve Digit Signature Algorithm" section 4.1.*/
if (digest->len*8 > (unsigned int)ecParams->fieldID.size) {
mpl_rsh(&s,&s,digest->len*8 - ecParams->fieldID.size);
}
However, the fieldID.size is not necessarily equal to the number of bits in the base point order, n. For example, with the standard F2M curves sect163r1 and sect283r1, the number of bits in n is one less than the field size: 162 and 282 bits, respectively. This leads to incorrect signatures when using hash algorithms that produce message digests longer than the field size.
Some cases to reproduce the issue are:
sect163r1 for ECDSA signatures using SHA 256, 384, 512
sect283r1 for ECDSA signatures using SHA 384, 512
I observed this issue when using jdk1.7.0_45, and still see the above code in the source for openjdk-8-src-b132-03_mar_2014.zip.