Troubleshooting

To troubleshoot Jetty when used as a standalone server, there are two main tools: the Jetty Server Dump and enabling DEBUG level logging.

Jetty is based on components organized as a tree, with the Server instance at the root of the tree.

As explained in the JMX section, these components can be exported as JMX MBeans and therefore be accessible from JMX Consoles such as Java Missions Control (JMC).

Being able to take a snapshot of the state of Jetty while it is running is the most useful information that can be attached when reporting an issue. Such state includes:

  • The thread pool configuration and its current state, including how many threads are in use, and their stack trace.

  • The TLS configuration.

  • The I/O configuration and its current state, including the ports Jetty listens to, how many connections are currently open, and he state of each connection, and the state of the request/response handling for each connection.

  • The Handler structure and its configuration.

  • The web applications deployed and their configurations, including the class loader information.

The prerequisite for troubleshooting is to enable JMX, so that Jetty — possibly a production server — can be accessed from a remote location to obtain the information exported via JMX, and possibly be able to reconfigure Jetty to solve the issue.

Make sure you read about how to secure the access to Jetty when using remote JMX.

Server Dump

The Jetty Server Dump is obtained by invoking, via JMX, the Server.dump() operation, as shown below using Java Mission Control (JMC):

jmc server dump

Find the Server MBean in the MBean Tree, under org.eclipse.jetty.server:type=server,id=0. Then click on the "Operations" tab, select the dump() operation, and then click the Execute button. In the bottom panel you will see the result of the invocation, that you can copy into a text editor and save to your file system.

Taking a Jetty Server Dump is a relatively expensive operation, as it dumps the state of all connections (which can be thousands), and the state of all threads.

The result of the invocation may produce a large string, possibly few MiB, that may impact the server memory usage.

Furthermore, dumping the state of the I/O Jetty components takes a little CPU time off the handling of the actual I/O, possibly slowing it down temporarily.

While the slow-down caused by taking the Jetty Server Dump may be noticeable on highly loaded systems, it is typically a very small price to pay to obtain the information about the Jetty state that may be critical to the resolution of an issue.

The format of the Jetty Server Dump output is subject to change at any time, as Jetty developers modify the Jetty code and decide to include more state, or remove state that is no longer relevant.

The Jetty Server Dump is organized in a tree whose structure is similar to the runtime Jetty component tree.

At the end of the dump output there is a legend that explains the type of tree node: whether it is a node that represent a managed component, or an array node (or a map node) that represent some component state, etc.

Dump at Server Start/Stop

The Server.dump() operation may also be invoked just after the Server starts (to log the state of the freshly started server), and just before the Server stops (which may be useful to log the state of server that is not working properly).

You can temporarily enable the Jetty Server Dump at start time by overriding the jetty.server.dumpAfterStart property on the command line:

$ java -jar $JETTY_HOME/start.jar jetty.server.dumpAfterStart=true

To make this change persistent across server restarts, see the server module configuration for more information about how to configure the server to dump at start/stop time.

Detailed ThreadPool Information

By default, the dump of the thread pool will only dump the topmost stack frame of each thread. It is possible to configure the thread pool to dump the whole stack trace for each thread; while this may be a little more expensive, it provides complete information about the state of each thread, which may be important to diagnose the issue.

See the threadpool module configuration for more information about how to configure the thread pool to dump detailed thread information.

Detailed thread pool information can also be turned on/off on-the-fly via JMX, by finding the ThreadPool MBean under org.eclipse.jetty.util.thread:type=queuedthreadpool,id=0, then selecting the detailedDump attribute and setting it to true. You can now perform the Server.dump() operation as explained above, and then set detailedDump back to false.

Dump Example

Below you can find a simple example of a Jetty Server Dump, with annotations for the principal components:

oejs.Server@477b4cdf{STARTING}[12.0.15-SNAPSHOT,sto=5000] - STARTING (1)
+= QueuedThreadPool[qtp394721749]@1786f9d5{STARTED,4<=4<=200,i=2,r=-1,t=59928ms,q=0}[ReservedThreadExecutor@1bb266b3{capacity=1,threads=ThreadIdPool@306cf3ea{capacity=1}}] - STARTED (2)
|  +- org.eclipse.jetty.util.thread.ThreadPoolBudget@2beee7ff
|  += ReservedThreadExecutor@1bb266b3{capacity=1,threads=ThreadIdPool@306cf3ea{capacity=1}} - STARTED
|  |  +~ QueuedThreadPool[qtp394721749]@1786f9d5{STARTED,4<=4<=200,i=2,r=-1,t=59926ms,q=0}[ReservedThreadExecutor@1bb266b3{capacity=1,threads=ThreadIdPool@306cf3ea{capacity=1}}] - STARTED
|  |  +- ThreadIdPool@306cf3ea{capacity=1}
|  +> threads size=4
|     +> qtp394721749-17 TIMED_WAITING tid=17 prio=5 IDLE
|     +> qtp394721749-16 TIMED_WAITING tid=16 prio=5 IDLE
|     +> qtp394721749-15-acceptor-0@43a7e535-ServerConnector@68567e20{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} RUNNABLE tid=15 prio=3 ACCEPTING
|     +> qtp394721749-14 RUNNABLE tid=14 prio=5 SELECTING
+= oejut.ScheduledExecutorScheduler@45f45fa1{STARTED} - STARTED
+- org.eclipse.jetty.io.ArrayByteBufferPool@3315d2d7{min=0,max=65536,buckets=16,heap=0/118087680,direct=0/118087680}
|  +> direct size=16
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@482bce4f{capacity=4096,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@6a6afff2{capacity=8192,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@1649b0e6{capacity=12288,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@865dd6{capacity=16384,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@4da4253{capacity=20480,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@3972a855{capacity=24576,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@62e7f11d{capacity=28672,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@503d687a{capacity=32768,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@6a370f4{capacity=36864,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@2abf4075{capacity=40960,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@770d3326{capacity=45056,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@4cc8eb05{capacity=49152,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@51f116b8{capacity=53248,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@19d481b{capacity=57344,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@1f97cf0d{capacity=61440,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  |  +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@140c9f39{capacity=65536,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|  +> indirect size=16
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@4d910fd6{capacity=4096,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@26275bef{capacity=8192,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@7690781{capacity=12288,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@77eca502{capacity=16384,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@3246fb96{capacity=20480,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@2e222612{capacity=24576,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@61386958{capacity=28672,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@73ee04c8{capacity=32768,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@7671cb68{capacity=36864,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@49dc7102{capacity=40960,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@6b8ca3c8{capacity=45056,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@68c72235{capacity=49152,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@10959ece{capacity=53248,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@3a6bb9bf{capacity=57344,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@34f7cfd9{capacity=61440,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
|     +> org.eclipse.jetty.io.ArrayByteBufferPool$RetainedBucket@65f095f8{capacity=65536,in-use=0/0,pooled/acquires=0/0(NaN%),non-pooled/evicts/removes/releases=0/0/0/0}
+~ org.eclipse.jetty.util.resource.FileSystemPool@59af0466
+= oejsh.DefaultHandler@3e6ef8ad{showContext=true,favIcon=true,STARTED} - STARTED
+= oejsh.ContextHandlerCollection@346d61be{STARTED} - STARTED (3)
+= ServerConnector@68567e20{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} - STARTED (4)
|  +~ QueuedThreadPool[qtp394721749]@1786f9d5{STARTED,4<=4<=200,i=2,r=-1,t=59861ms,q=0}[ReservedThreadExecutor@1bb266b3{capacity=1,threads=ThreadIdPool@306cf3ea{capacity=1}}] - STARTED
|  +~ oejut.ScheduledExecutorScheduler@45f45fa1{STARTED} - STARTED
|  +~ org.eclipse.jetty.io.ArrayByteBufferPool@3315d2d7{min=0,max=65536,buckets=16,heap=0/118087680,direct=0/118087680}
|  += HttpConnectionFactory@1aafa419[HTTP/1.1] - STARTED
|  |  +- HttpConfiguration@74455848{32768/8192,8192/8192,https://:0,[]}
|  |     +> customizers size=0
|  |     +> formEncodedMethods size=2
|  |     |  +> POST
|  |     |  +> PUT
|  |     +> outputBufferSize=32768
|  |     +> outputAggregationSize=8192
|  |     +> requestHeaderSize=8192
|  |     +> responseHeaderSize=8192
|  |     +> headerCacheSize=1024
|  |     +> headerCacheCaseSensitive=false
|  |     +> secureScheme=https
|  |     +> securePort=0
|  |     +> idleTimeout=-1
|  |     +> sendDateHeader=false
|  |     +> sendServerVersion=true
|  |     +> sendXPoweredBy=false
|  |     +> delayDispatchUntilContent=true
|  |     +> persistentConnectionsEnabled=true
|  |     +> maxErrorDispatches=10
|  |     +> useInputDirectByteBuffers=true
|  |     +> useOutputDirectByteBuffers=true
|  |     +> minRequestDataRate=0
|  |     +> minResponseDataRate=0
|  |     +> httpCompliance=RFC7230[]
|  |     +> uriCompliance=DEFAULT[]
|  |     +> redirectUriCompliance=null
|  |     +> requestCookieCompliance=RFC6265@2e55dd0c[INVALID_COOKIES, OPTIONAL_WHITE_SPACE, SPACE_IN_VALUES]
|  |     +> responseCookieCompliance=RFC6265@2e55dd0c[INVALID_COOKIES, OPTIONAL_WHITE_SPACE, SPACE_IN_VALUES]
|  |     +> multiPartCompliance=RFC7578
|  |     +> notifyRemoteAsyncErrors=true
|  |     +> relativeRedirectAllowed=false
|  |     +> serverAuthority=null
|  |     +> localAddress=null
|  |     +> maxUnconsumedRequestContentReads=16
|  += ServerConnectorManager@e7edb54[keys=0] - STARTED
|  |  += oeji.ManagedSelector@11bd0f3b{STARTED}[id=0 keys=0 selected=0 updates=0 selection:tot=0/avg=0.00/max=0] - STARTED (5)
|  |     += AdaptiveExecutionStrategy@79da8dc5/SelectorProducer@1eb5174b/PRODUCING/p=0/QueuedThreadPool[qtp394721749]@1786f9d5{STARTED,4<=4<=200,i=2,r=-1,t=59820ms,q=0}[ReservedThreadExecutor@1bb266b3{capacity=1,threads=ThreadIdPool@306cf3ea{capacity=1}}][pc=0,pic=0,pec=0,epc=0]@2024-11-07T17:49:27.658413303Z - STARTED
|  |     |  +- SelectorProducer@1eb5174b
|  |     |  +~ QueuedThreadPool[qtp394721749]@1786f9d5{STARTED,4<=4<=200,i=2,r=-1,t=59820ms,q=0}[ReservedThreadExecutor@1bb266b3{capacity=1,threads=ThreadIdPool@306cf3ea{capacity=1}}] - STARTED
|  |     +> updates @ 2024-11-07T17:49:27.650274467Z size=0
|  |     +> keys @ 2024-11-07T17:49:27.656064937Z size=0 (6)
|  +- sun.nio.ch.ServerSocketChannelImpl[/[0:0:0:0:0:0:0:0]:8080]
|  +- qtp394721749-15-acceptor-0@43a7e535-ServerConnector@68567e20{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
+- org.eclipse.jetty.util.component.HaltLifeCycleListener@67080771
+- org.eclipse.jetty.server.Server$DynamicErrorHandler@72cde7cc
+> startJarLoader@470e2030 (7)
|  +> URLs size=8
|  |  +> file:/path/to/jetty.home-base/resources/
|  |  +> file:/path/to/jetty.home/lib/logging/slf4j-api-2.0.16.jar
|  |  +> file:/path/to/jetty.home/lib/logging/jetty-slf4j-impl-12.0.15-SNAPSHOT.jar
|  |  +> file:/path/to/jetty.home/lib/jetty-http-12.0.15-SNAPSHOT.jar
|  |  +> file:/path/to/jetty.home/lib/jetty-server-12.0.15-SNAPSHOT.jar
|  |  +> file:/path/to/jetty.home/lib/jetty-xml-12.0.15-SNAPSHOT.jar
|  |  +> file:/path/to/jetty.home/lib/jetty-util-12.0.15-SNAPSHOT.jar
|  |  +> file:/path/to/jetty.home/lib/jetty-io-12.0.15-SNAPSHOT.jar
|  +> parent: jdk.internal.loader.ClassLoaders$AppClassLoader@694355ec
|     +> packages size=4
|     |  +> package org.eclipse.jetty.start.config
|     |  +> package org.eclipse.jetty.start.builders
|     |  +> package org.eclipse.jetty.start.shaded.util
|     |  +> package org.eclipse.jetty.start
|     +> parent: jdk.internal.loader.ClassLoaders$PlatformClassLoader@44821a96
|        +> packages size=2
|           +> package sun.util.resources.provider
|           +> package sun.util.resources.cldr.provider
+> environments size=1
|  +> oejuc.Environment$Named@0{core}
|     +> startJarLoader@470e2030
|     |  +> URLs size=8
|     |  |  +> file:/path/to/jetty.home-base/resources/
|     |  |  +> file:/path/to/jetty.home/lib/logging/slf4j-api-2.0.16.jar
|     |  |  +> file:/path/to/jetty.home/lib/logging/jetty-slf4j-impl-12.0.15-SNAPSHOT.jar
|     |  |  +> file:/path/to/jetty.home/lib/jetty-http-12.0.15-SNAPSHOT.jar
|     |  |  +> file:/path/to/jetty.home/lib/jetty-server-12.0.15-SNAPSHOT.jar
|     |  |  +> file:/path/to/jetty.home/lib/jetty-xml-12.0.15-SNAPSHOT.jar
|     |  |  +> file:/path/to/jetty.home/lib/jetty-util-12.0.15-SNAPSHOT.jar
|     |  |  +> file:/path/to/jetty.home/lib/jetty-io-12.0.15-SNAPSHOT.jar
|     |  +> parent: jdk.internal.loader.ClassLoaders$AppClassLoader@694355ec
|     |     +> packages size=4
|     |     |  +> package org.eclipse.jetty.start.config
|     |     |  +> package org.eclipse.jetty.start.builders
|     |     |  +> package org.eclipse.jetty.start.shaded.util
|     |     |  +> package org.eclipse.jetty.start
|     |     +> parent: jdk.internal.loader.ClassLoaders$PlatformClassLoader@44821a96
|     |        +> packages size=2
|     |           +> package sun.util.resources.provider
|     |           +> package sun.util.resources.cldr.provider
|     +> Attributes core size=0
+> attributes size=0
+> org.eclipse.jetty.util.resource.FileSystemPool@59af0466
   +> buckets size=0
key: +- bean, += managed, +~ unmanaged, +? auto, +: iterable, +] array, +@ map, +> undefined (8)
JVM: Eclipse Adoptium OpenJDK 64-Bit Server VM 23.0.1+11; OS: Linux amd64 6.8.0-48-generic; Jetty: 12.0.15-SNAPSHOT; CPUs: 1; mem(free/total/max): 50/58/900 MiB
UTC: 2024-11-07T17:49:27.698124784Z; Etc/UTC: 2024-11-07T17:49:27.698124784Z
1 The Server instance at the root of the tree
2 The thread pool component
3 The root of the Handler structure
4 The connector listening on port 8080 for the HTTP/1.1 protocol
5 A selector component that manages connections
6 The connections currently managed by the selector component
7 The server ClassLoader and its classpath
8 The legend for the dump nodes

Enabling DEBUG Logging

Enabling DEBUG level logging for the org.eclipse.jetty logger name provides the maximum amount of information to troubleshoot Jetty issues.

Refer to the logging section for more information about how to configure logging in Jetty.

Enabling DEBUG level logging for org.eclipse.jetty is very, very expensive.

Your server could be slowed down to almost a halt, especially if it is under heavy load. Furthermore, the log file could quickly fill up the entire filesystem (unless configured to roll over), so you want to be really careful using DEBUG logging.

For production servers, consider using the Jetty Server Dump first, and enable DEBUG logging only as a last resort.

However, sometimes issues are such that only DEBUG logging can really tell what’s going on in the system, and enabling DEBUG logging is your best chance to figure the issue out. Below you can find few suggestions that can help you reduce the impact when you have to enable DEBUG logging.

Jetty Behind a Load Balancer

If Jetty instances are behind a load balancer, you may configure the load balancer to send less load to a particular Jetty instance, and enable DEBUG logging in that instance only.

Enabling DEBUG Logging for a Short Time

In certain cases the issue can be reproduced reliably, but only in the production environment.

You can use JMX to temporarily enable DEBUG logging, reproduce the issue, and then disable DEBUG logging.

Alternatively, if you cannot reliably reproduce the issue, but you know it is happening, you can temporarily enable DEBUG logging for a small period of time, let’s say 10-60 seconds, and then disable DEBUG logging.

Changing the log level at runtime is a feature of the logging implementation that you are using.

The Jetty SLF4J implementation, used by default, exposes via JMX method boolean JettyLoggerFactoryMBean.setLoggerLevel(String loggerName, String levelName) that you can invoke via a JMX console to change the level for the specified logger name. The method returns true if the logger level was successfully changed.

For example, you can pass the string org.eclipse.jetty as the first parameter, and the string DEBUG (upper case) as the second parameter. You can then use the string INFO or WARN (upper case) to restore the logging level to its previous value.

Enabling DEBUG Logging for SubPackages

Enabling DEBUG logging for the org.eclipse.jetty logger name implies that all children logger names, recursively, inherit the DEBUG level.

Processing a single HTTP request involves many Jetty components: the I/O subsystem (under org.eclipse.jetty.io), the thread pool (under org.eclipse.jetty.util), the HTTP/1.1 parsing (under org.eclipse.jetty.http), etc.

If you can cut the amount of DEBUG logging to just what you need to troubleshoot the issue, the impact of enabling DEBUG logging will be much less than enabling it for all Jetty components.

For example, if you need to troubleshoot a client that sends bad HTTP/1.1 requests, it may be enough to enable only the org.eclipse.jetty.http logger name, therefore saving the large amount of DEBUG logging produced by the I/O subsystem and by the thread pool.

In another case, you may need to troubleshoot only HTTP/2 requests, and therefore enabling only the org.eclipse.jetty.http2 logger name could be enough.

Remote Debugging

The Java Virtual Machines allows remote processes on different hosts to connect for debugging purposes, by using specific command line options.

While it is possible to enable remote debugging on a Jetty server, it is typically not recommended for security and performance reasons. Only enable remote debugging on a Jetty server as a last resort to troubleshoot issues that could not be troubleshot otherwise.

You can easily create a custom Jetty module (see this section) with the following content:

remote-debug.mod
[description]
Enables remote debugging

[exec]
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

The [exec] directive (documented here) is necessary to pass the -agentlib:jdwp JVM option to the forked JVM that runs Jetty, so that you can attach with a debugger.

The address parameter of the -agentlib:jdwp command line option specifies the network address and port the Jetty JVM listens on for remote debugging.

Please refer to the Java Debug Wire Protocol documentation for additional information about the -agentlib:jdwp command line option and its parameters.

You can now enable the remote-debug Jetty module with the following command issued from the $JETTY_BASE directory:

$ java -jar $JETTY_HOME/start.jar --add-modules=server,remote-debug

The command above minimally adds a Jetty server without connectors (via the server Jetty module) and the remote-debug Jetty module, and produces the following $JETTY_BASE directory structure:

$JETTY_BASE
├── modules
│   └── remote-debug.mod
├── resources
│   └── jetty-logging.properties
└── start.d
    ├── remote-debug.ini
    └── server.ini

You can easily disable the remote-debug Jetty module as explained in this section.

Alternatively, you can enable the remote-debug module on the command line, as explained in this section.

Starting the Jetty server with the remote-debug module enabled yields:

WARN  : Forking second JVM due to forking module(s): [remote-debug]. Use --dry-run to generate the command line to avoid forking.
Listening for transport dt_socket at address: 5005
2024-11-07 17:49:31.483:INFO :oejs.Server:main: jetty-12.0.15-SNAPSHOT; built: 2024-11-07T17:42:40.103Z; git: 4abbb310f9f838a7693a830ecf20c1017184a304; jvm 23.0.1+11
2024-11-07 17:49:31.544:INFO :oejs.Server:main: Started oejs.Server@2ea41516{STARTING}[12.0.15-SNAPSHOT,sto=5000] @1253ms

Note how the JVM is listening on port 5005 to allow remote debuggers to connect.

If you want to avoid to fork a second JVM to pass the -agentlib:jdwp JVM option, please read this section.

Troubleshooting Handlers

StateTrackingHandler

Jetty’s StateTrackingHandler (described in this module) can be used to troubleshoot problems in web applications.

StateTrackingHandler tracks the usages of Handler/Request/Response asynchronous APIs by web applications, emitting events (logged at warning level) when an invalid usage of the APIs is detected.

In conjunction with dumping the Jetty component tree, it dumps the state of current requests, detailing whether they have reads or writes that are pending, whether callbacks have been completed, along with thread stack traces (including virtual threads) of operations that have been started but not completed, or are stuck in blocking code.

You need to enable the state-tracking Jetty module, and configure it to track what you are interested in tracking (for more details, see the javadocs).