In 6u, the bug fix JDK-8072452 increased DH/DSA keysize limits. 

From regression test (sun.security.pkcs11.KeyPairGenerator.TestDH2048) with SunPKCS11-NSS
        KeyPairGenerator kpg = KeyPairGenerator.getInstance("DH", p);
        kpg.initialize(3072);//not supported with nss version 3.13
        KeyPair kp1 = kpg.generateKeyPair();


In this case  (when tested with unsupported keysize) the sun/security/pkcs11/P11KeyPairGenerator.java throws java.security.ProviderException instead if IPE:

java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_TEMPLATE_INCOMPLETE
        at sun.security.pkcs11.P11KeyPairGenerator.generateKeyPair(P11KeyPairGenerator.java:338)
        at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:650)
        at TestDH2048.main(TestDH2048.java:45)
        at PKCS11Test.premain(PKCS11Test.java:30)
        at PKCS11Test.testNSS(PKCS11Test.java:162)
        at PKCS11Test.main(PKCS11Test.java:38)
        at TestDH2048.main(TestDH2048.java:66)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:110)
        at java.lang.Thread.run(Thread.java:682)
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_TEMPLATE_INCOMPLETE
        at sun.security.pkcs11.wrapper.PKCS11.C_GenerateKeyPair(Native Method)
        at sun.security.pkcs11.P11KeyPairGenerator.generateKeyPair(P11KeyPairGenerator.java:329)
        ... 12 more

The 7u-cpu build jdk1.7.0_171b06 gives InvalidParameterException as expected when checked with higher unsupported keysize.