String tokenizser


DevX Home    Today's Headlines   Articles Archive   Tip Bank   Forums   

Results 1 to 2 of 2

Thread: String tokenizser

  1. #1
    Join Date
    Jan 2007
    Posts
    60

    String tokenizser

    Can anyone tell me how I would put this into a while loop so it checks each line of a string and counts the tokens?

    Code:
    StringTokenizer TokenizeMe;
    int NumberOfTokens = 0;
    int NumberOfWords = 0;
    
    TokenizeMe = new StringTokenizer(MyFile.readLine());
    NumberOfTokens = TokenizeMe.countTokens();
    
    while (NumberOfTokens != 0)
    {
        for (int WordsInLine=1;
        WordsInLine<=NumberOfTokens; WordsInLine++)
        {
             System.out.print(TokenizeMe.nextToken()+" ");
        }
        System.out.println();
        NumberOfWords += NumberOfTokens;
        TokenizeMe = new StringTokenizer(MyFile.readLine());
        NumberOfTokens = TokenizeMe.countTokens();
    }
    System.out.println("\nThere are " + NumberOfWords + "words in the text");
    MyFile.close();

  2. #2
    Join Date
    Nov 2004
    Location
    Norway
    Posts
    1,560

    Many ways to skin the cat...

    Code:
    public class TokenCounter {
      String fileName=null;
      public TokenCounter(String fileName) {
        this.fileName=fileName;
      }
      /**
       * Not a good solution...
       */
      public int scanFile() throws IOException {
        int nTokens = 0;
        BufferedReader bR = new BufferedReader(new FileReader(fileName));
        String line = bR.readLine();
        while (line != null) {
          StringTokenizer strT = new StringTokenizer(line);
          nTokens += strT.countTokens();
          line = bR.readLine();
        }
        bR.close();
        return nTokens;
      }
      /**
       * I prefer this one
       */
      public int scanFile2() throws IOException {
        int nTokens = 0;
        BufferedReader bR = new BufferedReader(new FileReader(fileName));
        String line = bR.readLine();
        while (line != null) {
          nTokens += line.split(" ").length;
          line = bR.readLine();
        }
        bR.close();
        return nTokens;
      }
    
      public static void main(String[] args) {
        TokenCounter tc = new TokenCounter("c:\\tmp\\testdata.txt");
        try {
          System.out.println("File contains "+tc.scanFile()+" tokens");
          System.out.println("File contains "+tc.scanFile2()+" tokens (method 2)");
        }
        catch (IOException ex) {
          ex.printStackTrace();
        }
      }
    }
    ]

    Note: Your solution has a "dormant" bug, - the loop will break at the first blank line, and I don't think that is the intention.
    eschew obfuscation

Similar Threads

  1. Input string was not in a correct format
    By mdengler in forum ASP.NET
    Replies: 0
    Last Post: 11-26-2002, 02:32 PM
  2. Replies: 1
    Last Post: 06-05-2001, 06:12 AM
  3. How do I detect an FTP timeout?
    By Julian Milano in forum VB Classic
    Replies: 0
    Last Post: 08-10-2000, 09:16 PM
  4. Replies: 0
    Last Post: 06-22-2000, 07:30 AM
  5. Database problems
    By Robert Rieth in forum VB Classic
    Replies: 1
    Last Post: 04-11-2000, 03:21 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center
 
 
FAQ
Latest Articles
Java
.NET
XML
Database
Enterprise
Questions? Contact us.
C++
Web Development
Wireless
Latest Tips
Open Source


   Development Centers

   -- Android Development Center
   -- Cloud Development Project Center
   -- HTML5 Development Center
   -- Windows Mobile Development Center