Reading Large Files


DevX Home    Today's Headlines   Articles Archive   Tip Bank   Forums   

Results 1 to 9 of 9

Thread: Reading Large Files

  1. #1
    Join Date
    Oct 2004
    Posts
    151

    Question Reading Large Files

    Hi everyone,

    I am trying to to read and write large files about 600M.
    The thing is when i use the normal FileOutputStream methods an exception gets thrown saying

    Code:
    java.lang.OutofMemory
    This is how i am trying to read the file

    Code:
    FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
    InputStream in = //an input stream from a socket, etc 
    
    int len = 0;
    byte[] buffer1 = new byte[1024]; 
    
    while ((len = in.read(buffer1)) > 0)
    { 
    out.write(buffer1, 0, len);
    }
     
    in.close(); 
    out.close();
    My computer does have sufficient memory

    I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again

    Code:
    System.gc();
    My question is basically how do i increase the memory programatically
    or is there away in which i can read the file without having the above exception thrown by the JVM?

    Any help i greatly appreciated

    Thank You

    Yours Sincerely

    Richard West
    What type of softwares do you like and what would you pay for it?

    http://www.wizardsolutionsusa.com/fo...hread.php?t=12

  2. #2
    Join Date
    Jul 2005
    Location
    SW MO, USA
    Posts
    299
    look at the -X options for the java command. One of them sets the JVM size. Its default memory is like 64M. Set it to a larger value.

  3. #3
    Join Date
    Oct 2004
    Posts
    151
    Hi everyone,

    Norm is this what you meant
    Code:
    java -Xmx650m MyProgram

    I unfortunately can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

    Hoping to hear from you

    Richard West
    What type of softwares do you like and what would you pay for it?

    http://www.wizardsolutionsusa.com/fo...hread.php?t=12

  4. #4
    Join Date
    Jul 2005
    Location
    SW MO, USA
    Posts
    299
    I imagine the -X options are used by the JVM when it starts. After that, when your program starts executing is too late. Perhaps there's some way you could rewrite your program to NOT require too much memory. Do you have a memory leak? Are you in a recursive loop that keeps getting more and more memory?
    Try writing the simplest version of your program that does the I/O and gets the Out of Memory problem.

    Another approach:

    On Windows, clicking on a file causes the OS to go to a table of commands for that file type and execute the command. For example for jar files the command is:
    java -jar %1 - where %1 is replaced by the filename of the file that was clicked on.
    This command is stored in the Registry. You can manually change it in Windows Explorer by using the Tools|Folder Options...|FileTypes menu. Or you can create a .reg file and use regedit to merge it into your registry.
    So you could create a new entry for your user's jar open command that has the required -X option in and have them merge it into their Registry.

  5. #5
    Join Date
    Oct 2004
    Posts
    151

    Question

    Hi everyone,

    Quote Originally Posted by Norm
    Try writing the simplest version of your program that does the I/O and gets the Out of Memory problem.
    I tried searching somewhere else in the program and you were right norm as there seems to be a memory leak but it seemed to came from this part

    Code:
    FileOutputStream fStream = new FileOutputStream(SF);
    ObjectOutput stream = new ObjectOutputStream(fStream);
    
    //The memory exception occurs at the below command line
    
    stream.writeObject(TextPane1.getDocument());
    
    stream.flush();
    stream.close();
    fStream.close();
    I know for a fact that the document in the JTextPane spans couple of of thousand pages but it seems that the exception is coming from that area.

    Could it be that because the stream is writing the document at one go. If it is then is there a better way to do it?

    On an unrelated issue is there a way to increase the heap area programmatically maybe by use of properties?

    Richard West

    ps. please feel free to answer here or at JavaRanch whichever is comfortable to you
    What type of softwares do you like and what would you pay for it?

    http://www.wizardsolutionsusa.com/fo...hread.php?t=12

  6. #6
    Join Date
    Jul 2005
    Location
    the Netherlands
    Posts
    128

  7. #7
    Join Date
    Oct 2004
    Posts
    151
    Hi everyone,

    Sorry i forgot to mention something. I don't know if this means anything but if the document in the JTexPane class is too large can it cause the above exception i mentioned.

    If it can is there a way in which i can work around it so that even if the document itself is large it wont cause a outofmemory exception

    Richard West
    What type of softwares do you like and what would you pay for it?

    http://www.wizardsolutionsusa.com/fo...hread.php?t=12

  8. #8
    Join Date
    Jul 2005
    Location
    SW MO, USA
    Posts
    299
    If you can't change your design and rewrite the program to use less memory ...
    Another solution:
    Does your program work with the java -X option?
    Then you can make a registry entry for Windows applications say for a new extension: jarX with the command for it: java.exe -Xmx600m -jar "%1"

    Installation would then require running the regedit program to merge this entry into the registry on the user's system. Your jar files would be renamed to have the extension: jarX.

  9. #9
    Join Date
    Oct 2004
    Posts
    151
    Hi everyone,

    I think you are right, i may need to redesign the code.

    Richard West
    What type of softwares do you like and what would you pay for it?

    http://www.wizardsolutionsusa.com/fo...hread.php?t=12

Similar Threads

  1. Replies: 2
    Last Post: 02-28-2002, 11:53 AM
  2. Reading Large Binary Files for String Operations
    By Ryan Shelley in forum Java
    Replies: 0
    Last Post: 01-16-2002, 06:57 PM
  3. Win 2000 File Server - Large numbers of files
    By Andy Cortwright in forum Enterprise
    Replies: 1
    Last Post: 07-31-2001, 12:21 AM
  4. Large XML files into SQL
    By msuk in forum XML
    Replies: 1
    Last Post: 01-30-2001, 09:26 AM
  5. Reading operating system files
    By Venky in forum XML
    Replies: 4
    Last Post: 09-27-2000, 06:09 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center
 
 
FAQ
Latest Articles
Java
.NET
XML
Database
Enterprise
Questions? Contact us.
C++
Web Development
Wireless
Latest Tips
Open Source


   Development Centers

   -- Android Development Center
   -- Cloud Development Project Center
   -- HTML5 Development Center
   -- Windows Mobile Development Center