Learn Jenkins - Troubleshooting & Debugging Jenkins Pipelines

#Introduction

In episode 18 we kept Jenkins healthy with backups and monitoring. But however good the system is, pipelines will still fail — and what sets great engineers apart is how fast they find the root cause. In this episode we go into detective mode: reading logs correctly, testing changes without a commit, and handling the JVM issues that most often cause trouble.

We will discuss:

1. Jenkins log analysis: the System Log and agent logs.
2. The Replay Pipeline feature to test syntax changes without a new commit.
3. The Snippet Generator at /pipeline-syntax to produce Groovy syntax.
4. JVM troubleshooting: OOM, thread dumps, and disconnected agents.

#Reading the Jenkins System Log

Every event on the controller is recorded in the System Log, accessed from Manage Jenkins then System Log. There we can filter by level (ALL, SEVERE, WARNING, INFO, FINE) and search for error messages without opening files on the server.

The actual log files live under JENKINS_HOME/logs/. For a recent incident, the most effective filter is to look for SEVERE and WARNING messages:

grep -E "SEVERE|WARNING" /var/lib/jenkins/logs/jenkins.log | tail -50

#Agent Node Logs

When an agent disconnects or fails to connect, the agent log is on each node's page: Manage Jenkins then Nodes then the agent name, then click Log. From there we can see the cause of the connection failure, such as a mismatched SSH key or a wrong host address.

#Replay Pipeline: Testing Without a Commit

The Replay feature is a lifesaver when a pipeline fails at a specific line. Instead of editing the Jenkinsfile, committing, and pushing just to test one line, click Replay on the failed build page. Jenkins opens an editor with the exact same pipeline contents, and we can edit directly, then rerun that build.

A small change via Replay

steps {
    echo "Debug: WORKSPACE = ${env.WORKSPACE}"
    sh 'ls -la ${WORKSPACE}'
}

After the test build succeeds, that fix is then committed to the repository. This speeds up debugging iteration because it removes the slow commit-push cycle.

#The Snippet Generator at /pipeline-syntax

Memorizing the syntax of hundreds of Groovy steps is impossible — and unnecessary. Jenkins provides the Snippet Generator at /pipeline-syntax, which produces complete syntax from a filled-in form.

How to use it:

1. Open the /pipeline-syntax address on the Jenkins instance.
2. Pick a step from the dropdown, for example withCredentials or emailext.
3. Fill in the parameters in the form that appears.
4. Click Generate Groovy, copy the result, and paste it into the Jenkinsfile.

An example output for the sh step with output capture:

script {
    def output = sh(
        script: 'curl -s https://api.example.com/health',
        returnStdout: true
    ).trim()
    echo "Health: ${output}"
}

The Snippet Generator also provides a Declarative Directive Generator for blocks like environment, when, or parameters. This makes the Jenkins UI an interactive documentation that always stays in sync with the installed plugins.

#JVM & Memory Troubleshooting

#Out-of-Memory (OOM)

OOM happens when the controller's JVM heap is not enough for the build load. The symptoms: builds die suddenly, the UI responds slowly, and a java.lang.OutOfMemoryError message appears in the System Log. The fix is to raise the heap limit via JVM parameters when running the controller container:

FROM jenkins/jenkins:lts-jdk17
ENV JAVA_OPTS="-Djenkins.install.runSetupWizard=false -Xmx4g -Xms2g"

#Thread Dump

If the controller feels stuck or slow to respond with no pattern, take a thread dump to see what all threads are doing. Jenkins provides the /threadDump endpoint, which shows a snapshot of every thread. From there, look for threads with a repeated BLOCKED or WAITING status.

curl -k "https://jenkins.example.com/threadDump" | grep -A 5 "java.lang.Thread.State: BLOCKED"

#Disconnected Agents

A disconnected agent is usually not a controller memory issue, but a network or agent process one. The troubleshooting steps:

1. Open the agent log and look for the last connection message.
2. Make sure the agent and controller can ping each other, and the port is not blocked by a firewall.
3. Check agent resources: a build process can kill the agent if memory is full.
4. For JVM-process-based agents, make sure the agent heap is not too small.

For agents that often disconnect due to unstable networks, use container-based agents that restart automatically on failure, and check whether a proxy or firewall is dropping the remoting connection because of idleness.

#Conclusion

In this episode we learned troubleshooting and debugging:

• The System Log and agent logs are the source of truth for finding root causes.
• Replay allows testing pipeline changes without a new commit.
• The Snippet Generator at /pipeline-syntax produces Groovy syntax from an interactive form.
• OOM is handled by raising the heap, thread dumps are analyzed via /threadDump, and disconnected agents are checked through logs and the network.

Now we have all the skills to build and maintain Jenkins. In episode 20, the final episode of this series, we bring everything together into one complete production-grade enterprise pipeline case study — from webhook to production deployment. See you there!