JDK-4418160 : StringTokenizer support for empty tokens
  • Type: Enhancement
  • Component: core-libs
  • Sub-Component: java.util
  • Affected Version: 1.3.0
  • Priority: P4
  • Status: Closed
  • Resolution: Won't Fix
  • OS: generic
  • CPU: generic
  • Submitted: 2001-02-22
  • Updated: 2002-08-31
  • Resolved: 2002-04-24
Description
Name: rmT116609			Date: 09/18/2001


java version "1.3.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.0)
Java HotSpot(TM) Client VM (build 1.3.0, mixed mode)


Whilst looking through the newsgroup articles at groups.google.com, I have seen
several people stating that they would like to be able to get empty tokens back
from StringTokenizer. There have also been people writing code to do this. Would
it not be possible to do this in the core.


SOURCE CODE BELOW:

import java.io.*;
import java.util.*;

public class reporter {

  public reporter() {
  }

  public static void main(String args[]) {
    String line = "22||2|2|2";

    StringTokenizer lineToken = new StringTokenizer(line, "|");
    System.out.println(line);
    System.out.println("Tokens:"+lineToken.countTokens());
    while (lineToken.hasMoreElements()) {
      System.out.println("T:"+lineToken.nextElement());
    }
  }

}
(Review ID: 132119)
======================================================================


Classic VM (build JDK-1.2.2-004, green threads, sunwjit)


I am trying to write a program that will reformat a pipe ("|") delimited file.
The feature I would like to see added to the StringTokenizer is the ability for
it to either return a null value or an empty string when two pipes appear
together (i.e. ||).

Currently it goes for the first string it finds regardless of what delimiters
are before it. This is not what I want.
(Review ID: 117513) 
======================================================================

Comments
EVALUATION With the addition of the java.util.regex package in 1.4.0, we have basically obsoleted the need for StringTokenizer. We won't remove the class for compatibility reasons. But regex gives you simply what you need. Here are the examples: Example 1: /* import java.util.*; public class Test { public static void main(String args[]) { String[] tokens = "22||2|2|2".split("\\|"); for(int token=0; token<tokens.length; token++) { System.out.println("T:"+tokens[token]); } } } output: T:22 T: T:2 T:2 T:2 */ Example 2: /* import java.util.*; public class Test1 { public static void main(String args[]) { String[] tokens = "Ted,Nancy,Bob,,George".split(","); for(int token=0; token<tokens.length; token++) { System.out.println("T:"+tokens[token]); } } } output: T:Ted T:Nancy T:Bob T: T:George */ The above code will give you an array, as you desire for your issue. StringTokenizer is a legacy class, which can not be improved much without breaking backwards compatibility. java.util.regex package provides a powerfull way to handle regular expressions. Please use java.util.regex if StringTokenizer can not handle the problem. ###@###.### 2002-08-30
30-08-2002