I have a problem with an embedded Jetty application. The application suddenly requires à large amount of memory and threads and I get an out of memory error.
The application is running Jetty 9.1.2.v20140210 on java 1.7.51
The launch parameters are (now)
-Xmx4096m -Xss4096k -XX:ParallelGCThreads=2 -Djava.awt.headless=true -server
I have been trying a few launch options without much success so far.
The stack size is large because I need to serialize/unserialize a huge Java object.
The Jetty setup is:
QueuedThreadPool threadPool = new QueuedThreadPool(56, 10); server = new Server(threadPool); ServerConnector connector = new ServerConnector(server); connector.setIdleTimeout(1000); connector.setPort(port); connector.setHost(hote); server.addConnector(connector);
The application serves 6.67 requests per second on average (peak 16.23 request per second). The average request time is 42ms. There is no database I/O. There is a little file I/O by Lucene and logging.
I can't be 100% sure that there are no memory leeks but the various tools I have used (Java Visual VM and various Jetty LeekPreventers) seem to indicate that there are none (or that it is insignificant).
The application runs for 24 hours and is restarted. Sometime, randomly (after 3 hours or 20), the application suddenly requires à large amount of memory and threads and I get an out of memory error. I then have to kill the application.
I did a thread dump and found that almost all threads were waiting.
"qtp146419630-538" prio=10 tid=0x00007f07b0006800 nid=0x45f4 waiting on condition [0x00007f07fd7f3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000719d430f8> (a java.util.concurrent.Semaphore$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) at org.eclipse.jetty.util.BlockingCallback.block(BlockingCallback.java:94) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:125) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:351) at java.io.OutputStream.write(OutputStream.java:75) at org.eclipse.jetty.server.HttpOutput.print(HttpOutput.java:485) at javax.servlet.ServletOutputStream.println(ServletOutputStream.java:283) at com.example.MyHandler.doPost(PhpHandler.java:190) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
Does anyone have a clue as what would be the cause of this problem and a solution?