//
// ========================================================================
// Copyright (c) 1995 Mort Bay Consulting Pty Ltd and others.
//
// This program and the accompanying materials are made available under the
// terms of the Eclipse Public License v. 2.0 which is available at
// https://www.eclipse.org/legal/epl-2.0, or the Apache License, Version 2.0
// which is available at https://www.apache.org/licenses/LICENSE-2.0.
//
// SPDX-License-Identifier: EPL-2.0 OR Apache-2.0
// ========================================================================
//

= Troubleshooting

To troubleshoot Jetty when used as a standalone server, there are two main tools: the Jetty Server Dump and enabling DEBUG level logging.

It is typically very helpful to enable JMX, as detailed in xref:jmx/index.adoc[this section].
Enabling JMX allows to access the Jetty server (possibly a production server) from a remote location, and obtain the information exported by Jetty via JMX.

Access to the Jetty server via JMX allows to either diagnose the problem, possibly reconfigure and restart the Jetty server.

IMPORTANT: Make sure you read about how to secure the access to Jetty when using xref:jmx/index.adoc#remote[remote JMX].

[[dump]]
== Jetty Server Dump

The most useful too to troubleshoot a Jetty server is the xref:programming-guide:troubleshooting/component-dump.adoc[Jetty Component Dump], that reports the _internal_ state of the Jetty server and all its components (the thread pool, the `ByteBuffer` pool, the currently established connections, and so on).

The Jetty server dump can easily be obtained via JMX using a JMX console such as Java Mission Control (JMC), as explained in xref:programming-guide:troubleshooting/component-dump.adoc#dump-invoke[this section].

[[dump-start-stop]]
=== Dump at Server Start/Stop

The Jetty server dump may be reported just after the `Server` starts.
This is useful to verify that all the expected components have been correctly configured and started for the optimal functioning of the server.

Similarly, the Jetty server dump may be reported just before the `Server` stops.
This is useful to log the state of a server that is not working properly.

The xref:modules/standard.adoc#server[server] Jetty module defines, in xref:modules/standard.adoc#server-config[this section], the properties `jetty.server.dumpAfterStart` and `jetty.server.dumpBeforeStop` that you can configure to report the Jetty server dump.

These properties can be specified temporarily as command line arguments, as explained in xref:start/index.adoc#configure-enable-command-line[this section]:

[source,subs=+quotes]
----
$ java -jar $JETTY_HOME/start.jar *jetty.server.dumpAfterStart=true*
----

Alternatively, you can specify these properties in the `$JETTY_BASE/start.d/server.ini` file, as explained in xref:start/index.adoc#configure-edit-ini[this section], to make the values persistent across server restarts.

[[logging]]
== Enabling DEBUG Logging

Enabling DEBUG level logging for the `org.eclipse.jetty` logger name provides the maximum amount of information to troubleshoot Jetty issues.

Refer to the xref:server/index.adoc#logging[logging section] for more information about how to configure logging in Jetty.

[CAUTION]
====
Enabling DEBUG level logging for `org.eclipse.jetty` is very, *very* expensive.

Your server could be slowed down to almost a halt, especially if it is under heavy load.
Furthermore, the log file could quickly fill up the entire filesystem (unless configured to roll over), so you want to be really careful using DEBUG logging.

For production servers, consider using the xref:dump[Jetty Server Dump] first, and enable DEBUG logging only as a last resort.
====

However, sometimes issues are such that only DEBUG logging can really tell what's going on in the system, and enabling DEBUG logging is your best chance to figure the issue out.
Below you can find few suggestions that can help you reduce the impact when you have to enable DEBUG logging.

[[logging-backend]]
=== Jetty Behind a Load Balancer

If Jetty instances are behind a load balancer, you may configure the load balancer to send less load to a particular Jetty instance, and enable DEBUG logging in that instance only.

[[logging-jmx]]
=== Enabling DEBUG Logging for a Short Time

In certain cases the issue can be reproduced reliably, but only in the production environment.

You can use JMX to temporarily enable DEBUG logging, reproduce the issue, and then disable DEBUG logging.

Alternatively, if you cannot reliably reproduce the issue, but you _know_ it is happening, you can temporarily enable DEBUG logging for a small period of time, let's say 10-60 seconds, and then disable DEBUG logging.

Changing the log level at runtime is a feature of the logging implementation that you are using.

The Jetty SLF4J implementation, used by default, exposes via JMX method `boolean JettyLoggerFactoryMBean.setLoggerLevel(String loggerName, String levelName)` that you can invoke via a JMX console to change the level for the specified logger name.
The method returns `true` if the logger level was successfully changed.

For example, you can pass the string `org.eclipse.jetty` as the first parameter, and the string `DEBUG` (upper case) as the second parameter.
You can then use the string `INFO` or `WARN` (upper case) to restore the logging level to its previous value.

[[logging-subpackages]]
=== Enabling DEBUG Logging for SubPackages

Enabling DEBUG logging for the `org.eclipse.jetty` logger name implies that all children logger names, recursively, inherit the DEBUG level.

Processing a single HTTP request involves many Jetty components: the I/O subsystem (under `org.eclipse.jetty.io`), the thread pool (under `org.eclipse.jetty.util`), the HTTP/1.1 parsing (under `org.eclipse.jetty.http`), etc.

If you can cut the amount of DEBUG logging to just what you need to troubleshoot the issue, the impact of enabling DEBUG logging will be much less than enabling it for all Jetty components.

For example, if you need to troubleshoot a client that sends bad HTTP/1.1 requests, it may be enough to enable only the `org.eclipse.jetty.http` logger name, therefore saving the large amount of DEBUG logging produced by the I/O subsystem and by the thread pool.

In another case, you may need to troubleshoot only HTTP/2 requests, and therefore enabling only the `org.eclipse.jetty.http2` logger name could be enough.

[[debugging]]
== Remote Debugging

The Java Virtual Machines allows remote processes on different hosts to connect for debugging purposes, by using specific command line options.

[CAUTION]
====
While it is possible to enable remote debugging on a Jetty server, it is typically not recommended for security and performance reasons.
Only enable remote debugging on a Jetty server as a last resort to troubleshoot issues that could not be troubleshot otherwise.
====

You can easily create a custom Jetty module (see xref:modules/custom.adoc[this section]) with the following content:

.remote-debug.mod
----
include::code:example$jetty-modules/remote-debug.mod[]
----

The `[exec]` directive (documented xref:modules/index.adoc#directive-exec[here]) is necessary to pass the `-agentlib:jdwp` JVM option to the forked JVM that runs Jetty, so that you can attach with a debugger.

[NOTE]
====
The `address` parameter of the `-agentlib:jdwp` command line option specifies the network address and port the Jetty JVM listens on for remote debugging.

Please refer to the https://docs.oracle.com/en/java/javase/17/docs/specs/jpda/conninv.html[Java Debug Wire Protocol documentation] for additional information about the `-agentlib:jdwp` command line option and its parameters.
====

You can now enable the `remote-debug` Jetty module with the following command issued from the `$JETTY_BASE` directory:

----
$ java -jar $JETTY_HOME/start.jar --add-modules=server,remote-debug
----

The command above minimally adds a Jetty server without connectors (via the `server` Jetty module) and the `remote-debug` Jetty module, and produces the following `$JETTY_BASE` directory structure:

[source,subs=+quotes]
----
$JETTY_BASE
├── modules
│   └── remote-debug.mod
├── resources
│   └── jetty-logging.properties
└── start.d
    ├── ##remote-debug.ini##
    └── server.ini
----

You can easily disable the `remote-debug` Jetty module as explained in xref:start/index.adoc#configure-disable[this section].

Alternatively, you can enable the `remote-debug` module on the command line, as explained in xref:start/index.adoc#configure-enable-command-line[this section].

Starting the Jetty server with the `remote-debug` module enabled yields:

[jetty%nowrap]
....
[jetty]
setupModules=code:example$jetty-modules/remote-debug.mod
setupArgs=--add-modules=server,remote-debug
highlight=5005
....

Note how the JVM is listening on port `5005` to allow remote debuggers to connect.

If you want to avoid to fork a second JVM to pass the `-agentlib:jdwp` JVM option, please read xref:start/index.adoc#configure-dry-run[this section].

[[handlers]]
== Troubleshooting Handlers

[[handlers-state-tracking]]
=== `StateTrackingHandler`

Jetty's `StateTrackingHandler` (described in xref:modules/standard.adoc#state-tracking[this module]) can be used to troubleshoot problems in web applications.

`StateTrackingHandler` tracks the usages of `Handler`/`Request`/`Response` asynchronous APIs by web applications, emitting events (logged at warning level) when an invalid usage of the APIs is detected.

In conjunction with <<dump,dumping the Jetty component tree>>, it dumps the state of current requests, detailing whether they have reads or writes that are pending, whether callbacks have been completed, along with thread stack traces (including virtual threads) of operations that have been started but not completed, or are stuck in blocking code.

You need to enable the `state-tracking` Jetty module, and configure it to track what you are interested in tracking (for more details, see the link:{javadoc-url}/org/eclipse/jetty/server/handler/StateTrackingHandler.html[javadocs]).
