running large number of java threads

hi all,
i have a p2p simulation which basically runs thousands of java threads as peers to simulate peer interaction in a p2p network. currently, the maximum number of peers i can run is around 7000 on one machine. i tried to fiddling with the jvm parameters such as -xmx and -xms but still couldn't break the 7000 barrier. i also tried to run the program on different machines and os (i.e., xp and linux) and again without any success.

i have been looking around at distributed java implementation such as ProActive but after changing my code, it gives me hard-to-detect deadlocks due to the different implementation of threads in ProActive.

so is there any way for me to increase the number of threads for the simulation on one single machine? or if it has to be distributed, are there any packages that require minimum modifications to my code and is approved to be stable for such program?

thanks heaps,

lee

two ways to solve your problem:

1. Brute force way.
Each trhead consumes address space of the process. 7000 threads consume the whole 2GB address space. They do not each so much physical memory - the problem is that they consume all address space!

What you can do to run more threads:
1. Reduce stack size of each thread (play with -Xss)
2. Run your application on Windows Advanced Server (in 3GB mode) or on Linux (in 4/4 mode). This way, the application gets more address space.
3. Switch to another JVM that can run more threads (I work for JVM vendor company, so I have to offer this way :) )

However, 7000 threads is close to physical OS limit. By brute force methods, you can get 8000 threads, maybe even 15000, I don't know, but not 70 000 for sure.

2. The best way ;-)

Rewrite your program and schedule peers manually.
Instead of running each peer in separate thread, use a thread pool of, say, 1000 threads. Each tread spends some time executing one peer, then switches to the next one (randomly selecting which peer to choose).

You will have to rewrite peers code to break it to small pieces, each piece can be executed and then control goes to another peer.

Look at java.util.concurrent.ThreadPoolExecutor in Java 5.0

Does this sound sensible?

Denis.
----------------
www.excelsior-usa.com/jet.html
High-Performance JVM for J2SE

hi,
sorry it's been a while since i checked this post. i tried to thread pool method as you suggested, and with little changes to my code, now i am able to create 100,000 Runnable peer objects, and use vacant thread from the thread pool to simulate downloads for the peer objects.
thanks a lot for the help, it was a life-saver for me.

lee

btw, how many threads do you think i can create with JET on a window xp machine?

I do not know :(
The max thread count barrier is set by nature of OS, not by JVM. 7000, maybe 8000 or 15000, but surely not more.

Denis