Jenkins for Python Web Apps Part 2

Part 2: You've now got Jenkins, you have a Python web app.... Now what?

Following on from Part 1, for the purposes of this demonstration I've come up with a really simple Bottle application, which lives in a single file, app.py:

from bottle import Bottle, run,response, request

app = Bottle()

@app.route('/echo/<text>')
def echo(text):
    return {"Your Path Param":text}

@app.route('/queries')
def query():
    q = request.query_string
    if q:
        return {"You queried":q}
    else:
        response.status = 400
        return "Please supply a query string"


if __name__ == "__main__":
    app.run(host='localhost', port=8080)

The application has got two endpoints, which are both just 'echo' servers, one grabbing a path param and the other a query param. The query param endpoint throws an error if one isn't supplied, which makes testing more interesting. In order to assist our testing, I am making use of the WebTest package, meaning we don't have the overhead of running a live HTTP server in order to test the web app, (the package is actually recommended in the Bottle Docs). I've used this in conjunction with PyTest, so my test_app.py file is listed below. Note the test_client fixture which returns the WSGI app, wrapped in the testing interface.

from webtest import TestApp,AppError
import app
import pytest


class TestApplication:

    @pytest.fixture
    def test_client(self):
        return TestApp(app.app)

    def test_successful_echo(self,test_client):
        client = test_client
        response = client.get('/echo/hello')

        assert response.status_code == 200

    def test_echo_no_path_param_supplied(self,test_client):
        client = test_client
        with pytest.raises(AppError):
            client.get('/echo')

    def test_successful_query(self,test_client):
        client = test_client
        response = client.get('/queries?foo=bar')

        assert response.status_code == 200

    def test_query_no_params_supplied(self, test_client):
        client = test_client
        with pytest.raises(AppError):
            client.get('/queries')

A quick check on my laptop to ensure all is working well: 

(venv) $ pytest
=========================================================================== test session starts ============================================================================
platform darwin -- Python 3.8.9, pytest-7.0.1, pluggy-1.0.0
rootdir: /Users/petersimpson/PycharmProjects/basic-bottle
collected 4 items                                                                                                                                                          

test_app.py ....                                                                                                                                                     [100%]

============================================================================ 4 passed in 0.19s =============================================================================

Cool! All the tests pass. I'll get that up-to a Github repository and then its time to turn our attention back to Jenkins. 


We are going to create a Jenkins ‘freestyle’ project. If you are coming to the Jenkins documentation for the first time, there are many options and the difference between freestyle and other types of Jenkins projects may not seem that clearly defined. For the purposes of this demo, the key takeaway should be that this its effectively the most simple type of CI job you can have, just to get started. Click on ‘New Item’ from the left hand menu and then give the project a name, selecting the ‘Freestyle Project’ option.

You’ll see that there is a lot we could potentially be configuring here, but at the moment, all we want to do is enable Jenkins to pull the code from a Github repository and then give it a series of commands to set up the environment to get the app to be able to have our suite of tests run against it.


The ‘Source Code Management’ section of my project looks like this:

My repository is a public one, so no credentials needed, but that can be configured if required. The only other thing we are doing is giving Jenkins a series of commands to execute after its gotten the source code:

That's creating and activating a virtual environment, installing the application's requirements and then running the tests. It might look a bit different to what you expect and thats because the Jenkins shell doesn't necessarily always play nicely with virtual environments. Some discussion can be found HERE. I'll admit that there was a lot of pain during this step. There is a Jenkins plugin call Shining Panda, which is supposed to make this easier, but it doesn't look to be maintained, (its used in the CI chapter of Harry Percival's excellent Obey The Testing Goat book). Later on we’ll be using Docker to improve upon this.

Save the config and then when you are on the page for the project, hit the 'Build Now' button from the left hand menu. All being well; a new entry will appear in the build history table and you can inspect it and examine the console output. 

So from Jenkins we can now hit a button, and it will grab our source code, install our dependancies and then run the tests. That's obviously great but ultimately we want more automation and the potential for a more complex sequence of actions. When more than 1 person is contributing to a code base, it's likely there will be code formatting, lining, static type checking and potentially different types of tests, not to mention tests against different operating systems and Python versions. These could be run automatically at defined intervals or in response to each new commit to the codebase.


For these more involved use-cases, Jenkins has the concept of a Pipeline. Pipelines can be defined via a Jenkinsfile which is checked into source code, giving developers full control over what goes on in Jenkins in a simple and declarative way.  It's much cleaner than the hacky shell commands from earlier, to wrangle a virtualenv into shape. A quick read of the Jenkins documentation of Docker Pipelines should give you an idea of how powerful it can be.

SSH back into you Digital Ocean Droplet (or other), and get docker installed. We then need to get the Jenkins user added to the docker group.

$ sudo usermod -aG docker $USER

We now need to restart Jenkins from the command line (doing it from the Jenkins UI won’t be sufficient here).

$ sudo service jenkins restart

Once that’s complete, we need to add the Docker and Docker Pipeline plugins to Jenkins. Manage Jenkins > Manage Plugins.

I Dockerize my Bottle app, so that we now have a Dockerfile defining the environment that the app will run in, along with copying the code into it. This is an overtly simplistic example for my toy project, but a more detailed discussion of Docker and Python can be found here in the article 'Creating the Perfect Python Dockerfile' by Luis Sena.

FROM python:3.8-alpine

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

WORKDIR /app

COPY requirements.txt /app/
RUN pip install -r requirements.txt

COPY . /app/

And my Jenkinsfile is as simple as:

pipeline {
  agent {
    dockerfile { filename 'Dockerfile' }
  }
  stages {
    stage('test') {
      steps {
        sh 'pytest'
      }
    }
  }
}

The Jenkinsfile initially creates the container we define in our Dockerfile as an ‘agent’, which it will use in the different stages of the Pipeline. Now that all of the environment setup is encapsulated in our Dockerfile, all Jenkins needs to do for our ‘test’ stage is simply run the 'pytest' command!

Outrageously simple, but hopefully you can see that underneath 'stages', there could be multiple 'stage' declarations, (and then each of those can themselves contain many steps). This is where the complexity can be built out.

Log back into Jenkins, click on 'New Item', but this time ensure you select 'Pipeline' rather than freestyle project. Under the Pipeline configuration, select 'Pipeline script from SCM' and you can then hook into the same GitHub repository. Jenkins will assume that the Jenkins file is checked into the root folder, but this can also be overridden.  

You get a much nicer and more detailed UI here, I think. Here is a successful build: 

The inidvidual stages can be inspected seperately from one another: 

So there you have it! I cannot stress enough this is only the beginning. These two posts are intended as merely an ‘up and running’ tutorial. I’ve not really talked about CI more generally, let alone many of the cool and exciting things you can do with Jenkins. These include,(but are by no means limited to): different kinds of triggers, parameterization, interactivity, etc, etc, etc. Also, what is Blue Ocean!?

Said Bostandoust’s Tutorial Series looks like a good place to get an overview of more advanced functionality. The ever reliable Full Stack Python site also has a CI page

You may also like: