Software developers can work well alone, if they're in control of all their software assets and tests. Things get trickier, however, when they have to work as part of a team on a fast-moving project with lots of releases. A group of developers can contribute their code to the same source repository, like Git, but they then have to run all the necessary tests to ensure things are working smoothly. Assuming the tests pass, they must build those source files into executable binaries, and then deploy them. That's a daunting task that takes a lot of time and organisation on larger software projects.
This is what Jenkins is for. It's an open-source tool that co-ordinates those stages into a pipeline. This makes it a useful tool for DevOps, a development and deployment approach that automates the various stages of building software, creating an efficient conveyor belt system. Teams that get DevOps right with tools like Jenkins can move from version roll-outs every few months to every few days (or even hours), confident that all their tests have been passed.
Jenkins used to be called Hudson, but its development team renamed it after Oracle forked the project and claimed the original name. It's free, and runs on operating systems including Windows, Mac, and Linux, and it can also run as a Docker image.
You can get Jenkins as a downloadable from the Jenkins.io website, but you'll need to run the Java runtime environment to support it. Alternatively, you can install it as a Docker container by following the instructions on the official Jenkins site, which is what we'll do here. Docker takes a little extra work to set up, but the advantage here is twofold: it solves some dependency problems you might run into with Java, and it also enables you to easily recreate your Jenkins install on any server by copying your Docker file and the Docker run command to run it, which we put into a shell script for increased convenience. Jenkins' Docker instructions also install a souped-up user interface called Blue Ocean. If you don't use the Jenkins Docker instructions you can also install Blue Ocean separately as a plugin.
First, we must create a Python program for Jenkins to work with. We created a simple file called test-myapp.py, stored on our Linux system in /home/$USER/python/myapp. It includes a basic test using the Python PyTest utility:
| #test_capitalizationdef capitalize_word(word): return word.capitalize() def test_capitalize_word(): assert capitalize_word('python') == 'Python' |
Create a Github repository for it using git init. Commit the file to your repo using git add ., and then git commit -m "first commit".
Now it's time to start Jenkins using the docker run command in the Jenkins teams' Docker instructions. Once Jenkins is running, you can access it at localhost:8080. It will initially show you a screen with a directory path to a file containing your secure first-time access password. Copy the contents of the file to get logged into the administration screen, and from there it will set up the basic plugins you need to work with the software. Then, you can create a new user account for yourself.
If you're not already in the Blue Ocean interface, click on that option in the left sidebar. It will ask you to create a project. Call it myapp and then select Git as the project type in the Blue Ocean interface.
Blue Ocean will now ask you to create a pipeline. Click yes. We're going to write this pipeline ourselves in a 'Jenkinsfile', which we'll store in our myapp folder.
A Jenkinsfile is a test file describing your pipeline. It contains instructions for the stages of each build. It looks like this:
| pipeline { agent any stages { stage('Build') { steps { <steps for this stage go here> } } stage('Test') { steps { <steps for this stage go here> } } stage('Deploy') { steps { <steps for this stage go here> } } }} |
Each stage reflects a step in the build pipeline and we can have as many as we like. Let's flesh out this template.
Python programs don't need building and deploying in the same way that, say, C++ programs do, because they're interpreted. Nevertheless, Jenkins is useful in other ways. We can test our code automatically, and we can also check its formatting to ensure that it's easy for other developers to read.
To do this, we need to compile the Python program into bytecode, which is an intermediate stage that happens when you run Python programs. We'll call that our build stage. Here's the Jenkinsfile for that step:
| pipeline { agent none stages { stage('Build') { agent { docker { image 'python:2-alpine' } } steps { sh 'python -m py_compile test_myapp.py' stash(name: 'compiled-results', includes: '*.py*') } } }} |
The agent is the external program that runs this stage. We don't define a global one, but we do define one for the individual stage. In this case, because we're using Docker, it's a lightweight Alpine container with a Python implementation.
For our step, we run a shell command that compiles our python file.
Save this in your project folder as 'Jenkinsfile' and then commit it using git add . and git commit -m "add Jenkinsfile".
Back in the UI, ignore Blue Ocean's prompt to create a pipeline. Once it spots the Jenkinsfile in your repo, it'll build it from that automatically. Go into the Jenkins dashboard by clicking the exit icon next to the Logout option on the top right, or clicking the Jenkins name on the top left of the screen, to get to the main Jenkins screen. Look for your new project in the dashboard and on the left, select Scan Multibranch Pipeline Now.
Wait for a few seconds and Jenkins will scan your Git repo and run the build. Go back into the Blue Ocean interface, and all being well you'll see a sunny icon underneath the HEALTH entry that shows the build succeeded. Click on myapp, then Branch indexing, and it'll give you a picture of your pipeline and a detailed log.
Now we will add a test stage. Update your code to look like this:
| pipeline { agent none stages { stage('Build') { agent { docker { image 'python:2-alpine' } } steps { sh 'python -m py_compile test_myapp.py' stash(name: 'compiled-results', includes: '*.py*') } } stage('Test') { agent { docker { image 'qnib/pytest' } } steps { sh 'py.test --verbose --junit-xml test-results/results.xml test_myapp.py' } post { always { junit 'test-results/results.xml' } } } }} |
We're using another Docker container to run a simple PyTest (which we included in the code of our myapp.py file). Save this file and update your repo with another git add a . and git commit -m "add test stage to Jenkinsfile". Then, scan the multibranch pipeline as before. When you drop into Blue Ocean, you'll hopefully see success once again. Note that Docker stores everything it runs in its own volume, along with the results. Although you can work some command line magic to access those files directly, you don't need to; Jenkins shows you those assets in its UI. Click on the latest stage to open the build details, and find the entry that says py.test –verbose –junit-xml test-results/results.xml testmyapp.py. Clicking on that shows you the results of your test:
Everything passed! Now we're going to bring it home with the final stage in our demo pipeline: checking the code formatting. There are specific rules for formatting Python code as outlined in the language's PEP-8 specifications. We'll update our Jenkins file to use a tool called PyLint that will check our code. Here's the full Jenkinsfile for all three stages of our pipeline:
| pipeline { agent none stages { stage('Build') { agent { docker { image 'python:2-alpine' } } steps { sh 'python -m py_compile test_myapp.py' stash(name: 'compiled-results', includes: '*.py*') } } stage('Test') { agent { docker { image 'qnib/pytest' } } steps { sh 'py.test --verbose --junit-xml test-results/results.xml test_myapp.py' } post { always { junit 'test-results/results.xml' } } } stage('Lint') { agent { docker { image 'eeacms/pylint' } } environment { VOLUME = '$(pwd)/test_myapp.py' IMAGE = 'eeacms/pylint' } steps { withEnv(['PYLINTHOME=.']) { sh "pylint ${VOLUME}" } } } }} |
Follow the same steps as before: save the file, commit it to your Git repo so that Jenkins sees it, and then rescan the multi-branch pipeline. Then go into Blue Ocean and look at the result. Oh no!
The pipeline stage failed! That's because our code is badly formatted, and PyLint tells us why. We'll update our test_myapp.py file to make the code compliant:
| """Program to capitalize input"""#test_capitalizationdef capitalize_word(word): """ Capitalize a word""" return word.capitalize() def test_capitalize_word(): """Test to ensure it capitalizes a word correctly""" assert capitalize_word('python') == 'Python' |
Now, save, commit to your repo, and rescan. Blue Ocean shows that we fixed it (note that in our demo it took us a couple of runs at the Python code to get the formatting right).
You could run all these steps manually yourself, but the beauty of Jenkins is that it automates them all for faster development. That makes the tool invaluable for developers working on a fast cadence as part of a team, but even a single freelance dev, or a hobbyist working on open-source projects, can use this to refine their practice.
