Reading Large Files

[freesoft_2000]

Hi everyone,

I am trying to to read and write large files about 600M.
The thing is when i use the normal FileOutputStream methods an exception gets thrown saying

java.lang.OutofMemory

This is how i am trying to read the file

FileOutputStream out = new FileOutputStream("C:/my_life_story.zip");
InputStream in = //an input stream from a socket, etc 

int len = 0;
byte[] buffer1 = new byte[1024]; 

while ((len = in.read(buffer1)) > 0)
{ 
out.write(buffer1, 0, len);
}
 
in.close(); 
out.close();

My computer does have sufficient memory

I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again

System.gc();

My question is basically how do i increase the memory programatically
or is there away in which i can read the file without having the above exception thrown by the JVM?

Any help i greatly appreciated

Thank You

Yours Sincerely

Richard West

[Norm]

look at the -X options for the java command. One of them sets the JVM size. Its default memory is like 64M. Set it to a larger value.

[freesoft_2000]

Hi everyone,

Norm is this what you meant

java -Xmx650m MyProgram

I unfortunately can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

Hoping to hear from you

Richard West

[Norm]

I imagine the -X options are used by the JVM when it starts. After that, when your program starts executing is too late. Perhaps there's some way you could rewrite your program to NOT require too much memory. Do you have a memory leak? Are you in a recursive loop that keeps getting more and more memory?
Try writing the simplest version of your program that does the I/O and gets the Out of Memory problem.

Another approach:

On Windows, clicking on a file causes the OS to go to a table of commands for that file type and execute the command. For example for jar files the command is:
java -jar %1 - where %1 is replaced by the filename of the file that was clicked on.
This command is stored in the Registry. You can manually change it in Windows Explorer by using the Tools|Folder Options...|FileTypes menu. Or you can create a .reg file and use regedit to merge it into your registry.
So you could create a new entry for your user's jar open command that has the required -X option in and have them merge it into their Registry.

[freesoft_2000]

Hi everyone,

Try writing the simplest version of your program that does the I/O and gets the Out of Memory problem.

I tried searching somewhere else in the program and you were right norm as there seems to be a memory leak but it seemed to came from this part

FileOutputStream fStream = new FileOutputStream(SF);
ObjectOutput stream = new ObjectOutputStream(fStream);

//The memory exception occurs at the below command line

stream.writeObject(TextPane1.getDocument());

stream.flush();
stream.close();
fStream.close();

I know for a fact that the document in the JTextPane spans couple of of thousand pages but it seems that the exception is coming from that area.

Could it be that because the stream is writing the document at one go. If it is then is there a better way to do it?

On an unrelated issue is there a way to increase the heap area programmatically maybe by use of properties?

Richard West

ps. please feel free to answer here or at JavaRanch whichever is comfortable to you

[prometheuzz]

The same question, the same person, multiple threads: nice work!

http://www.daniweb.com/techtalkforums/thread30336.html
http://www.codeguru.com/forum/showth...05#post1211505
http://www.sitepoint.com/forums/showthread.php?t=289575
http://forum.java.sun.com/thread.jspa?threadID=652675

[freesoft_2000]

Hi everyone,

Sorry i forgot to mention something. I don't know if this means anything but if the document in the JTexPane class is too large can it cause the above exception i mentioned.

If it can is there a way in which i can work around it so that even if the document itself is large it wont cause a outofmemory exception

Richard West

[Norm]

If you can't change your design and rewrite the program to use less memory ...
Another solution:
Does your program work with the java -X option?
Then you can make a registry entry for Windows applications say for a new extension: jarX with the command for it: java.exe -Xmx600m -jar "%1"

Installation would then require running the regedit program to merge this entry into the registry on the user's system. Your jar files would be renamed to have the extension: jarX.

[freesoft_2000]

Hi everyone,

I think you are right, i may need to redesign the code.

Richard West