Distributed Builds

Never run builds on the Jenkins controller in production environments. Out of the box, Jenkins is set up to run builds on the built-in controller node. This makes it easier to get started with Jenkins, but, on production instances, builds should only be executed on nodes that are external to the controller.

A build job that uses an agent running on the built-in controller node may be able to access Jenkins configuration files and the workspaces of other builds. An agent could also request information that belongs to other teams or organizations that share the controller. Configuring Jenkins as a distributed build system enhances the security of your Jenkins instance as well as improving its performance and making it more stable. This is because builds that run on the built-in node have the same level of access to the controller file system as the Jenkins process itself. Configuring your Jenkins instance so that builds execute on agents running on separate nodes protects the Jenkins controller from malicious (or just broken) build scripts.

When distributed builds are implemented, the Jenkins controller serves HTTP requests and stores all important information related to the builds. Nodes are separate physical or virtual machines that are configured. The Jenkins controller manages these nodes, but the agents manage the task execution on behalf of the Jenkins controller and supply most of the computing power required to build and test the software. Different nodes can run different operating systems and different tools, enabling one Pipeline to build and test the same software for a variety of platforms.

A job that performs administrative tasks such as backups may run on the controller. To reduce the chance of someone using this executor to run a job, label the executor with a hard-to-guess label and only allow it to be used by jobs that specify that label.

An executor can be temporarily configured on the controller long enough to run the administrative task but no executors should be configured on the controller at other times This ensures that no builds execute on the controller since a clever Pipeline author may be able to work around the label restriction.

You can install the Job Restrictions Plugin to ensure that builds do not run on the controller even if executors are configured.

An agent can also write malicious code to its local disk so that the node is tainted and can infect a later build. For maximum security, run all builds on ephemeral agents in the cloud so that they are destroyed at the end of each build job.

By extension, you also should not run agents on the same Docker host as the controller.

Distributed Components

A Jenkins distributed builds instance includes the following components. For more details about these components, see Managing Nodes.

Jenkins controller: The Jenkins controller is a webserver where Jenkins is installed. It schedules tasks, executes management tasks (configuration, authorization, and authentication). Files written when a Pipeline executes are written to the filesystem on the controller unless they are off-loaded to an artifact repository such as Nexus or Artifactory.

To prevent builds from running drectly on the built-in node, navigate to Manage Jenkins » Manage Nodes and Clouds. Select Built-In Node in the list, then select Configure in the menu. Set the number of executors to 0 and save. Configure clouds or build agents where builds can execute.

Nodes: Nodes are the "machines" on which build agents run. The Jenkins controller itself runs on a special built-in node.
Agents: Agents are small Java client processes that use executors to manage the task execution on behalf of the Jenkins controller. In practice, nodes and agents are essentially the same but it is good to remember that they are conceptually distinct.
Executors: An executor is a slot for execution of tasks; effectively, it is a thread in the agent. The number of executors on a node defines the number of concurrent tasks that can be executed on that node at one time.

Implementing Distributed Builds

To implement distributed builds on your Jenkins instance, you must do the following:

Create nodes and agents where the builds can run.
Prevent builds from running on the controller.

This section also includes other tips and tricks you can use.

Create Nodes and Agents for Builds

To implement a distributed builds architecture for your Jenkins instance, you must configure nodes and agents to use for your builds. See Managing Nodes for instructions.

Prevent Builds from Running on the Controller

You must also prevent builds from running on the built-in controller node. To do this:

Navigate to Manage Jenkins » Manage Nodes and Clouds.
Select master in the list, then select Configure in the menu.
Set the number of executors to 0.
Save the configuration.

A job that performs administrative tasks such as backup may run on the controller, but be sure to label the executor with a hard-to-guess label and only allow it to be used by jobs that use that label. Jenkins should normally be configured to have 0 executors on the controller, and then be temporarily reconfigured to have 1 executor long enough to run the backup or other administrative task. Unless the number of executors is 0, it is possible for a build to run on the controller.

The Agent → Controller Access Control system provides additional protections for the controller.

Another way to prevent builds from running on the controller is to use a plugin such as the Job Restrictions Plugin to limit which jobs can be run on certain nodes (like the built-in controller node), independent of what your less trusted users may use as label expression in their job configurations.

Agent → Controller Security

The Jenkins controller and agents can be thought of as a distributed process that executes across multiple discrete processes and machines. Not building on the built-in controller node is an essential practice to protect the controller from bugs and less sophisticated attackers. However, an agent can stil ask the controller process for information such as the contents of files, and can even request that the controller run certain commands. This means that an agent process taken over by a malicious user could still obtain data or execute commands on the Jenkins controller. To prevent this, Jenkins implements the Agent → Controller Access Control filter that prevents agent processes from sending malicious commands to the Jenkins controller.

This filter may prevent your builds from executing legitimate operations.

If builds are failing because of the agent-to-controller access control, you should first ensure that you have upgraded to the latest version of all plugins, since older versions of plugins may not work correctly with this feature.
You can also tweak the filter’s rules by adding allow/deny rules to specify the commands that you need and the type of access that is allowed for the agent. See Customizing Agent → Controller Security for more information.

In releases prior to Jenkins 2.326 (see Agent → Controller Security Changes in 2.326), this protection could be disabled in the web UI by un-checking the box on the Manage Jenkins » Configure Global Security page, although this was strongly discouraged.

Configure Global Security - Enable Agent ⇒ Controller Access Control

See the documentation for details.

Other Tips and Tricks

If you do not have any other computers on which to run agents, you can run an agent process as a different operating system user on the same system to achieve a similar isolation effect. In this case, be sure that the agent process has neither read nor write file system access to the Jenkins home directory, and that the agent process cannot use sudo or other practices to elevate its own permissions.

User Handbook

Tutorials

Resources