In our previous post, How to Deploy with Jenkins: Best Practices we discussed security, backup and other topics. In this post we will provide the instructions required to use Jenkins to build and deploy Python web applications. We will use a UNIX-based OS, and in this specific case, Ubuntu Server 15.10 and latest version of Jenkins.
To enable the Jenkins installation (on Ubuntu) to proceed smoothly, make sure you have installed JDK and JRE on your machine. The versions that are suggested on the Jenkins site are openjdk-7-jre and openjdk-7-jdk (Learn how to install JDK and JRE).
The commands to install Jenkins on Ubuntu
Proceed to install Jenkins, which should be pretty easy using the following 4 commands:
wget -q -O – https://jenkins-ci.org/debian/jenkins-ci.org.key |
sudo apt-key add –
sudo sh -c ‘echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list’
sudo apt-get update sudo apt-get install jenkins
By default the service will run on port 8080. After installing Jenkins, enter the IP address of the machine where Jenkins is installed. You will notice the message about booting up Jenkins and preparing for work, as in this image:
Now we’re ready to move to the next step, which involves applying one of the basic and well known practices to secure Jenkins.
Under Manage Jenkins there is a Configure Global Security section where it is possible to configure different types of authentication. For this use case we will use the Jenkins user database after enabling security. Of importance here is the option “Allow users to sign up” where you can enable or disable new users to use your Jenkins. As mentioned above, Jenkins should be deployed in a private network and be restricted to a specific set of users. One option is for the administrator to add users. Another option is to use GitHub and similar organizations. You can add GitHub users to your organizations on the GitHub site and then you can configure the Jenkins plugin. This plugin reads the organization’s structure and allows specific GitHub users to login into Jenkins.
Under Authorization there are several ways of configuring the permissions that you want to provide to particular users. In this section we will not go into detail about securing Jenkins, instead we will use the Matrix-based security where you can easily enable or disable permissions with simple checkboxes. More details can be found on the Jenkins site – standard security setup.
Now let’s move to create the jobs that will prepare and and deploy your web application. Creating jobs is possible right after installing Jenkins on your server, but Jenkins support distributes builds in master-slave manner; more information is available via this link.
While your integration and deployment processes should be divided into small jobs, since this example is relatively a small project we will not show this step. For this use case, just create the job for pulling the latest code from the repository, start the tests and then deploy and start it on your test server. Also this use-case has a simple architecture with Jenkins and the application located on same server. With complex architectures we might use the scripts or plugins to move the builds between servers.
The first job which we created is the job which will execute shell commands, to create a virtual Python environment for your web application.
Enter the Jenkins dashboard and click New Item on the left menu as shown below:
The form below displays; in this form, enter the item name and select the type of the project. For our purposes, the freestyle project is one which we selected and has the item name we used for PythonCreateEnv. The freestyle project provides enough options and features to build the complex jobs that you will need in your projects.
The next step is to configure your job. The structure of the web application which we used isn’t presented here, but there is a file inside the folder /envs where we put the list of dependencies (pip style) for this project. This file will be used as source list, and notes which pip to use to install the required libraries for this project.
Before configuring the shell script, we will enter the code Repository URL under the Source Code Management –
The next and final step under the Build section is to enter the shell script (below) which will create or recreate (this depends on the previous state of your virtual environment), the virtual environment for your web application and install the required libraries. This job also can be avoided in the Jenkins job and executed in your terminal, but also we want to show that jobs like this can be automated with Jenkins.
(here is the script that you can copy/paste)
if [ –d $PYENV_HOME ]; then
rm –rf $PYENV_HOME
virtualenv —no–site–packages $PYENV_HOME
pip install –r envs/req.txt
After saving this item, the job is ready to create the virtual environment and will appear on the Jenkins dashboard. Our next move is to create a job to start the tests and interpret the results.
Jenkins supports interpreting and presenting different kinds of unit tests for various programming languages. In our project we used the nose tool for Python testing. If we pass –with-xunit flag, nose will create the XML file as a report of the test cases which were covered. Jenkins knows how to read this file and it will create nice charts for test results, so let’s create the job.
After clicking ‘New item’ from the menu and choosing the name and the type of the project, there are two additional steps to flow:
- As in the previous job, you will have to enter the URL of the repository under the Source Code Management.
- The next step is to create the shell script which Jenkins will execute. This shell script will start a new virtual environment, and then execute the nose unit test.
virtualenv —no–site–packages $PYENV_HOME
nosetests —with–xunit tests
For more information about how to user virtualenv, see: https://www.google.com/url?q=https://virtualenv.pypa.io/en/stable/
The last step is adding the post build actions.
Under the section Post-build Action, click the Add post-build action button.
Select the Publish JUnit test result report, then two new input fields will appear. In first one which is labeled Test report XMLs, enter the filename of generated report. The second field is optional – health report amplification factor, which allows you to specify a percentage of failed tests. Using this field you can measure the quality of the release.
Now we’re done with configuring this job; after saving, it will appear on the Jenkins dashboard. The outcome results charts are generated after the job is done and will appear in the detailed view of the job, after clicking on the job name in the Jenkins jobs table on the dashboard.
You can start the jobs by clicking the icon on the far right, which is located in same row as the project on the Jenkins dashboard. Jenkins allows you to monitor the console output for each job once a job is done. From the Jenkins dashboard you can get the console output by clicking the job which appears in the Build Executor Status on the left side, or by clicking the number which appeared in the Last Success or Last Failure column in the Jenkins jobs table.
The last job to close this use-case is the job which will start the web application. Starting a web application written in Python is as simple as creating the shell script which will start the HTTP server.
The web application which we have written for this use case is implemented with the Flask framework which has an integrated development server. Starting the web application with the development server via the command line is accomplished with the command python run.py &.
The development server isn’t a recommended way of deploying web applications for a production environment, usually one of the WSGI Servers is used for this purpose.
One more thing, it’s possible to schedule jobs periodically as an automation process so that you can avoid starting the jobs manually. In Jenkins, the build can be triggered in several ways like using the cron jobs or via service hooks, and the configuration settings for build triggers is located under the Build Triggers section of job configuration page.
One of the popular ways to trigger a job is after each commit to the repository. One of the easiest ways to do this is via the GitHub Jenkins plugin. Once you have this plugin installed, a new option Build when a change is pushed to GitHub will appear under the Build Triggers section which we discussed above.
In order to complete this way of triggering, a webhook needs to be added, so go to your GitHub repository then click Settings (see the image below).
Then click Add Service and find the Jenkins (GitHub plugin). Now in the Jenkins hook url field enter the your Jenkins URL (see the image below).
Final Notes: Jenkins 2.0
Using Jenkins is the simplest and most common way to implement CI in today’s R&D environments. In both parts of this article (part 1 can be found here) we discussed the theoretical side and best practices, as well as give you a hands-on example. However, there are a number of suitable alternatives to Jenkins, such as TravisCI, Go and TeamCity.
In addition, Jenkins 2.0 was just released, presenting an important new capability – the Pipeline. The new version takes Jenkins beyond CI and CD, and includes new capabilities which enable users to plan, orchestrate and visualize their entire delivery pipeline.
“For me as a practitioner and Jenkins administrator, the out-of-the-box defaults we’ve changed in this release are also important, especially for new users. New users of Jenkins are going to find it much easier to get set up with the tools they need, and Jenkins will be more secure out of the box.” Said R.Tyler Croy, Jenkins Community Leader
We also encourage you to get involved and join the active Jenkins community. Contribute by creating your own plugins, or at least extend the ones you are using. Fix bugs and share the fixes with the community; stay tuned to the Jenkins mailing lists and IRC. After all, Jenkins is open source.
Download the Ultimate Guide to deploying, managing and scaling Kubernetes