This problem is a regression caused by a security fix in 6u3 build 2 (CR 6569621) to prevent malicious exploitation of the fact that browser and plugin allowed communication via liveconnect even when DNS rebinding IP to a different hostname.
From the testcase: http://oueb.org/AppletFireFoxLiveconnectBug.html provided by Cyrille,
oueb.org IP is 184.108.40.206, but the reversed hostname for that IP from Sun DNS is 240plan.ovh.net.
The call from JS -> J is made with codebase oueb.org, but the Java open connection call is made with 240plan.ovh.net, which is different from codebase oueb.org and therefore the connection is prevented.
Cyrille also posted the same testcase on digimedia.fr. since digimedia.fr's IP is 220.127.116.11 and its reversed DNS hostname is also digimedia.fr, and therefore no issue there.
While 6569621 is a security fix, it may be too abrasive, and the fact that testcase from oueb.org works fine with IE shows inconsistency in behavior.
However, since 6569621 is a security fix, I will need to talk to Jeff to make sure whatever solution we come up with won't create another security hole.
From Jeff Nisewanger:
The defaultDeny flag is set only for SocketPermission instances
created by the pre-FireFox 3 plugin for the purpose of
is directly invoking bootstrap loader classes (rather than invoking a
method from an associated applet). In that situation, we are relying on
the browser's DNS resolution of the script codebase to match the
plugin's independent DNS resolution of the same codebase host. Because,
the DNS resolutions and pinning databases are seperate the plugin cannot
safely rely that the browser downloaded the script from the same IP
address that the plugin would grant a SocketPermission to for the
ProtectionDomain representing the script codebase during the
flag in the SocketPermission to require enhanced assurance that the
plugin's DNS resolution of the script codebase host looks legitimate
via a reverse DNS resolution check. If, instead, the script invokes
on a non-bootstrap class that is part of an associated applet then we
don't set the defaultDeny flag and just rely on the normal
SecurityManager check which will require both the script and applet
on the call stack to have SocketPermission grants that will approve
the network attempt and since the plugin/jvm downloads the applet code
it can use a consistent DNS pinning database for the applet codebase.
Before this fix, I was ham-handedly additionally setting the script
codebase into the "proxied host list" which persistently enforced the
enhanced DNS resolution checks in unnecessary situations.