JDK-4715206 : new ArrayList().addAll(myWeakHashMap.keySet()) can throw NoSuchElementException
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util:collections
  • Affected Version: 1.4.0
  • Priority: P4
  • Status: Closed
  • Resolution: Fixed
  • OS: linux
  • CPU: x86
  • Submitted: 2002-07-16
  • Updated: 2021-03-03
  • Resolved: 2002-11-07
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
1.4.2 mantisFixed
Related Reports
Duplicate :  
Description

Name: gm110360			Date: 07/15/2002


FULL PRODUCT VERSION :
java version "1.4.0_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0_01-b03)
Java HotSpot(TM) Client VM (build 1.4.0_01-b03, mixed mode)


FULL OPERATING SYSTEM VERSION :
glibc-2.2.4-24
Linux dhcp-168-239 2.4.7-10 #1 Thu Sep 6 17:27:27 EDT 2001
i686 unknown
Red Hat Linux release 7.2 (Enigma)

A DESCRIPTION OF THE PROBLEM :
There is a problem arising between the behavior of
WeakHashMap and the implementation of
ArrayList.addAll(Collection).

Sometimes the addAll call can cause a NoSuchElementException
to be thrown. This is at the mercy of timing in the garbage
collector. The reason is simple:

....
int numNew = c.size();
....
Iterator e = c.iterator();
for (int i = 0; i < numNew; i++) {
    elementData[size++] = e.next();
}
....

If Collection c is completely well-behaved, this is fine. If
it is e.g. the key set from a WeakHashMap, however, c.size()
may be greater than the number of times e.next() can safely
be called. The safe impl is this:

....
int numNew = c.size();
....
Iterator e = c.iterator();
while (e.hasNext()) {
    elementData[size++] = e.next();
}
....

I would suggest one of the following be done:

1. Patch ArrayList.addAll to use code like that suggested
above, i.e. be sure to always call Iterator.hasNext() before
calling Iterator.next(), and do not trust Collection.size()
to always be accurate. But there may be a performance
disadvantage to doing this.

2. Patch ArrayList.addAll to use the safer form only if the
collection is the key set (or value set or entry set) from a
WeakHashMap, otherwise use the faster form. A nice
compromise but will not solve the problem for third-party
(non-java.util.*) collection classes that use weak or soft
references, such as:

http://www.netbeans.org/source/browse/~checkout~/openide/src/org/openide/util/WeakSet.java

3. Document in ArrayList.addAll, and perhaps in other method
Javadoc, that it is unsafe to call the method with a
Collection parameter derived from a WeakHashMap. Also warn
in the Javadoc for WeakHashMap that its derived Set's are
unsuitable for being passed to certain methods.

  Bug originally observed unreproducibly in NetBeans IDE:

http://www.netbeans.org/issues/show_bug.cgi?id=25285

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Try running the demo class shown. On my machine, it shows
the exception about 50% of the time. Depends on timing factors.

EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected result:

Always prints "OK".

Actual result:

Sometimes prints "OK". Sometimes dumps a stack trace.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Exception in thread "main" java.util.NoSuchElementException
	at java.util.WeakHashMap$HashIterator.nextEntry(WeakHashMap.java:736)
	at java.util.WeakHashMap$KeyIterator.next(WeakHashMap.java:767)
	at java.util.ArrayList.addAll(ArrayList.java:438)
	at WeakHashMapSynchTest.main(WeakHashMapSynchTest.java:8)


REPRODUCIBILITY :
This bug can be reproduced often.

---------- BEGIN SOURCE ----------
import java.util.*;
public class WeakHashMapSynchTest {
    public static void main(String[] args) {
        Map m = new WeakHashMap(100000);
        for (int i = 0; i < 100000; i++) {
            m.put(new Object(), Boolean.TRUE);
        }
        new ArrayList().addAll(m.keySet());
        System.out.println("OK.");
    }
}

---------- END SOURCE ----------

CUSTOMER WORKAROUND :
If you know about the bug, it is easy to work around, i.e.
it is mostly a matter of unmet expectations about the safety
of using the API. Rather than

WeakHashMap m;
ArrayList a;
a.addAll(m.keySet());

you can just do:

WeakHashMap m;
ArrayList a;
Iterator i = m.keySet().iterator();
while (i.hasNext()) a.add(i.next());
(Review ID: 158660) 
======================================================================

Comments
CONVERTED DATA BugTraq+ Release Management Values COMMIT TO FIX: mantis FIXED IN: mantis INTEGRATED IN: mantis mantis-b07 VERIFIED IN: mantis-beta
14-06-2004

SUGGESTED FIX There are three strategies to fix the problem: 1) Precopy the to-be-added Colleciton into an array. 2) Use the normal iteration algorithm instead of the optimized one. 3) Catch the exception in the rare case that the size method returns an overestimate. Which approach to use is largely a performance decision. Approach 1 is the obvious approach if we assume that the performance of the addAll method isn't critical. Approach 3 makes me a bit nervous. The addAll(int index, Collection c) method is a bit trickier, as the size is used to shift forward the "tail" prior to inserting the new elements. Here's a cute approach that copes with the rare case of an overestimate by the size() method without precopying the to-be-added method into a temporary array. I have not tested (or even compiled) it, but I'm quite sure the idea is sound: public synchronized boolean addAll(int index, Collection c) { modCount++; if (index < 0 || index > elementCount) throw new ArrayIndexOutOfBoundsException(index); // Upper bound on actual value - exact except for weak collections int estNumNew = c.size(); ensureCapacityHelper(elementCount + numNew); int numMoved = elementCount - index; if (numMoved > 0) System.arraycopy(elementData, index, elementData, index + estNew, numMoved); int numNew = 0; for (Iterator i = c.iterator(); i.hasNext(); ) { elementData[index++] = e.next(); numNew++; } // If estimate was high (rare!!!), move "tail" back if (numNew != estNew) { if (numNew > estNew) throw new ConcurrentModificationException(); System.arraycopy(elementData, index + estNew, elementData, index + numNew, numMoved); } elementCount += numNew; return numNew != 0; } The alternative is to copy the to-be-added collection into an array before adding it. Arguably we should benchmark these two approaches before selecting one. ###@###.### 2002-07-28 I benchmarked all three strategies: public boolean addAll(Collection c) { // Old modCount++; int numNew = c.size(); ensureCapacity(size + numNew); Iterator e = c.iterator(); for (int i=0; i<numNew; i++) elementData[size++] = e.next(); return numNew != 0; } public boolean addAll1(Collection c) { // New1 modCount++; Object[] a = c.toArray(); int cSize = a.length; ensureCapacity(size + cSize); System.arraycopy(a, 0, elementData, size, cSize); size += cSize; return cSize != 0; } public boolean addAll2(Collection c) { // New2 modCount++; int oldSize = size; ensureCapacity(size + c.size()); for(Iterator i = c.iterator(); i.hasNext(); ) elementData[size++] = i.next(); return size != oldSize; } public boolean addAll3(Collection c) { // New3 modCount++; int numNew = c.size(); ensureCapacity(size + numNew); int oldSize = size; try { Iterator it = c.iterator(); for (int i = 0; i < numNew; i++) elementData[size++] = it.next(); } catch(NoSuchElementException e) { // Caused by underestimate in numNew (possible in weak collections) } return size != oldSize; } Here are the results: N is size of collection to be added. All tests are on ArrayList. Times for 100,000 ops (+other junk, take them as approximate). Solaris Client VM Old New1 New2 New3 Size: 1 9 6 12 13 Size: 10 32 11 91 33 Size: 100 232 48 840 232 Size: 1000 2156 479 8286 2163 Size: 10000 21954 4577 83688 22098 Solaris Server VM Size: 10 18 11 13 13 Size: 100 101 46 118 106 Size: 1000 958 326 1146 1003 Size: 10000 9865 3843 11665 10330 As you can see, the toArray/arraycopy version (New1) crushes all competitors. Of course it's a space-time tradeoff, but I think it's reasonably compelling. ###@###.### 2002-11-05
05-11-2002

EVALUATION We attempted to fix the general problem in 1.4 (see 4486049, 4416923), but we missed a few instances. In particular, we missed ArrayList.addAll(Collection), ArrayList.addAll(int, Collection), and the analogous methods in LinkedList and Vector. The "copy constructors" of various collection implementations use the .size method to pre-size the underlying data structure, but they fail only if the returned value is too *low*. Any collection for which size produces an underestimate is completely broken, so we needn't worry about this. (For all normal collections, the size method gives an exact value; for weak collections, it rarely gives an overestimate.) The general consensus is that "clever" iteration idioms (which assume size() yields exact results) are OK for iterating over List instances, but not arbitrary Collection instances. (See Bloch's "Effective Java Programming Language Guide, P. 143) ###@###.### 2002-07-28
28-07-2002