running large number of java threads


DevX Home    Today's Headlines   Articles Archive   Tip Bank   Forums   

Results 1 to 4 of 4

Thread: running large number of java threads

  1. #1
    Join Date
    Nov 2004
    Posts
    6

    running large number of java threads

    hi all,
    i have a p2p simulation which basically runs thousands of java threads as peers to simulate peer interaction in a p2p network. currently, the maximum number of peers i can run is around 7000 on one machine. i tried to fiddling with the jvm parameters such as -xmx and -xms but still couldn't break the 7000 barrier. i also tried to run the program on different machines and os (i.e., xp and linux) and again without any success.

    i have been looking around at distributed java implementation such as ProActive but after changing my code, it gives me hard-to-detect deadlocks due to the different implementation of threads in ProActive.

    so is there any way for me to increase the number of threads for the simulation on one single machine? or if it has to be distributed, are there any packages that require minimum modifications to my code and is approved to be stable for such program?

    thanks heaps,

    lee

  2. #2
    Join Date
    Jul 2005
    Posts
    8
    two ways to solve your problem:

    1. Brute force way.
    Each trhead consumes address space of the process. 7000 threads consume the whole 2GB address space. They do not each so much physical memory - the problem is that they consume all address space!

    What you can do to run more threads:
    1. Reduce stack size of each thread (play with -Xss)
    2. Run your application on Windows Advanced Server (in 3GB mode) or on Linux (in 4/4 mode). This way, the application gets more address space.
    3. Switch to another JVM that can run more threads (I work for JVM vendor company, so I have to offer this way )

    However, 7000 threads is close to physical OS limit. By brute force methods, you can get 8000 threads, maybe even 15000, I don't know, but not 70 000 for sure.

    2. The best way ;-)

    Rewrite your program and schedule peers manually.
    Instead of running each peer in separate thread, use a thread pool of, say, 1000 threads. Each tread spends some time executing one peer, then switches to the next one (randomly selecting which peer to choose).

    You will have to rewrite peers code to break it to small pieces, each piece can be executed and then control goes to another peer.

    Look at java.util.concurrent.ThreadPoolExecutor in Java 5.0

    Does this sound sensible?

    Denis.
    ----------------
    www.excelsior-usa.com/jet.html
    High-Performance JVM for J2SE

  3. #3
    Join Date
    Nov 2004
    Posts
    6

    thanks

    hi,
    sorry it's been a while since i checked this post. i tried to thread pool method as you suggested, and with little changes to my code, now i am able to create 100,000 Runnable peer objects, and use vacant thread from the thread pool to simulate downloads for the peer objects.
    thanks a lot for the help, it was a life-saver for me.

    lee

    btw, how many threads do you think i can create with JET on a window xp machine?

  4. #4
    Join Date
    Jul 2005
    Posts
    8

    how many threads on JET/WinXP

    I do not know
    The max thread count barrier is set by nature of OS, not by JVM. 7000, maybe 8000 or 15000, but surely not more.

    Denis

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center
 
 
FAQ
Latest Articles
Java
.NET
XML
Database
Enterprise
Questions? Contact us.
C++
Web Development
Wireless
Latest Tips
Open Source


   Development Centers

   -- Android Development Center
   -- Cloud Development Project Center
   -- HTML5 Development Center
   -- Windows Mobile Development Center