How to avoid OutOfMemoryError

[WXY595]

In my information retrieval project, I have to build a vector space model. There are three arrays, i.e. a string array IndexTerm[i], a string array FileName[j], and a double array weight[i][j]. The total number of the files is about ten thousand, while the number of index terms might be twenty thousand. This causes a serious problem, i.e. java.lang.OutOfMemoryError

Could anybody tell me how to avoid this without reducing the number of files and index terms?

[Laszlo]

There's the "MicroSoft" answer: Throw hardware at it. More RAM.

Then there's the java.lang.management package. I have not used these classes yet, but I see things there like java.lang.management.MemoryUsage that sound tantilizing. Perhaps that will be of use.

[nspils]

Look at the heap size switches available for the java runtime engine reserving memory - or use a data structure for managing your files which doesn't need to have them all in memory at the same time.

[freesoft_2000]

Hi everyone,

Then there's the java.lang.management package. I have not used these classes yet, but I see things there like java.lang.management.MemoryUsage that sound tantilizing. Perhaps that will be of use.

Quite interesting but after looking at those classes from what i get itss only used to monitor memory and not allocate more memory. If i have understood wrongly maybe you could ellaborate a bit in detail on the using this class to increase memory during runtime

I had a similar problem as the OP a few months ago but what i did was to write my own application launcher. From there i would use the jar command to create another jvm but this time using the -Xmx flag thus setting the maximum amount of memory i want use but in my case it was around 800MB - 1GB. I used this way because i could not find anyway to increase the memory during runtime

Richard West

[WXY595]

Look at the heap size switches available for the java runtime engine reserving memory - or use a data structure for managing your files which doesn't need to have them all in memory at the same time.

Thank you very much. But could you further explain how to increase the heap size if I'm using JBuilder, and how to manage the files which doesn't need to have them all in memory at the same time. Thanks.

[Laszlo]

You can specify how large your heap will start at with the JVM command line parameter -Xms. But that will not affect the limitations on how much memory is available to the JVM. You may be able to get a little extra memory by reducing your JVM stack size with the -Xsssize command line parameter.

As for how to manage your data, you haven't given enough information to know what in the world you're doing or how to optimize it. What is the purpose for loading ten thousand pages into RAM? Whatever you're doing, can it be done in pieces in stead of all at once?