Jenkins and Continuous Test Execution

Continuous test execution is an essential part of automation testing specially in regression run. If you need to start execution manually means your return of investment is not as good as needed. There are multiple ways to do this. Developers in many companies use Continuous Integration (CI) tool like Hudson, Rational Team Concert for building application multiple times in a day where codes are checked in from multiple location. You can use same feature of building application to perform testing as well. You can build your own in-house application to perform the same task. I will give idea for the same at the end of the blog.


I use WebDriver for automation testing in .NET framework.  I heard that many companies use Jenkins for CI and even for testing so thought to implement my tests written in C#. After some research on internet I was able to implement a basic skeleton for continuous testing. In order to understand and to customize Jenkins we will go Step by Step.

  1. Download Jenkins.
  2. Run and verify Jenkins.
  3. Install as a Window Service.
  4. Configure Jenkins as a slave. (Optional. Read only if you want to learn.)
  5. Create Job to execute test.

1.       Download Jenkins:

In order to download Jenkins, Go to http://jenkins-ci.org/ and download ‘Latest and greatest’ version of Java web archive (.war) file and save it to some location.

2.       Run And Verify Jenkins:

Open Command window and navigate to the location where you have downloaded *.war file. Execute following command (I saved Jenkins.war file in D drive)

  • Java –jar jenkins.war

You will see following output

image001

If you read all lines you will see that HTTP Listeners started at port 8080.

image002

3.       Install Jenkins as Window Service:

Node: If you install as window service, your tests will run in background.

Perform Step 2 if not yet done. Click ‘Manage Jenkins’ you will see following scree with various option.

image003

Click on ‘Install as Windows Service link. You will see following screen.

image004

Click on Install button. Once installation complete you will see following:

image005
Go to Window Services and verify if you can see there or not. image006 From now, you will not require to start Jenkins site on command line. You can directly start http://localhost:8080/ if your Jenkins service is running. NOTE: if you are using only window OS then you can directly install native package from http://jenkins-ci.org/ image007 You need to unzip the folder and install msi file. That will install and get ready with step 3.

4.       Configure Jenkins as a Slave:

This is most interesting part of Jenkins. Here you can configure your slave attached to Master which is ready in step 3. You can assign tasks to various slave to perform like test execution etc. Start Jenkins Website and click on link ‘Manage Jenkins’. You will see following option among various other options. image008 Click on the above option which will take you to the following screen. image009 Now you need to create Node that connected to master (Just like GRID (HUB-NODE)). To do that you need to click on ‘New Node’ link that should you to following screen. image010 Give some Name of Node and Click on Dumb Slave radio button that will active the OK button. Click on OK button. This will take you to the following screen. image011 You can click on Help icon image012 to understand more about the fields. I am giving following options # of executor : 1 ,  Remote FS root:  D:\Shared. Launch method: Save this. It will take you the following screen. image013 As you can see above, you can start that node by given three options. It is up to you, to choose but I prefer first option. Click on the Orange button with Launch. This will ask you to download slave-agent.jnlp. Please save this to the location where you have created your slave i.e. D:\Slave. Open command prompt and change directory to the location where you have saved this file and execute  following command Javaws slave-agent.jnlp aafter some processing you may see following windows: image014 This will disappear in few second and you will see Jenkins slave Agent screen with ‘Connected’ text on the screen. image015 If you go back to jenkin site and refresh page in Manage Node screen, you will see following screen. image016 You can install this slave into machine as window services using File menu given in ‘Jenkin salve agent’ screen so in future you will not require doing the whole process of starting slave via jlnp image017 image018 Click ‘OK’. This may ask your permission to run the program. image019 Will see that Jenkins slave agent window disappear and if you check in ‘services’ you will be able to see slave as services. image020 Go back to Jenkins site and Click on slave link the page and make sure that you can see slave running. image021 Now your slave is connected to your master and ready to server your for execution by added execution job assigned by you as given  in next section.

5.       Create JOB to Execute Test:

Once your slave node is running, go to home page of Jenkins and click on ‘New Job’. image022 This will take you to the following window. Provide Job name as per your choice and select radio button. ‘Build a free-style software project’. And hit OK. image023

This will take you the detailed screen and you can fill information as you needed. Few points are very important which you should consider.

  1. Execute test at specific Node.

image024

2. Regression Run repletion :

How frequent you want to execute test.

image025

There are various command you can give to give frequency of test execution ie. Every hour or at ever day at some specific time frame. You can Google the commands or I can write separate article about this if needed.

3. Execute Tests from various options (batch command preferred at least in my case)

image026

4. Post Execution:

Configure to send email or prepare report as NUnit/Junit repot etc.

image027

Once you save this job this will take you to the following screen.

image028

Click on  [Back to Dashboard] link that will take you the following screen.

image029

To execute test you can click on image030  . You will see following screen.

image031

I think much detail covers the basic understanding of Jenkins configuration for test execution. I hope this will give you are direction to configure Jenkins in your project. I use Webdriver in .NET environment and configure Jenkins for continuous execution and  it was successful implementation but later we decided to build our own internal application to specific for our own application which can do continuous execution and other tasks as well which is not possible via Jenkins like customized internal website with customized graph and tables but that’s another story.

I am going to write another article specifically how to execute tests using msbuild, msTest and Nunit and another article for JUnit.

14 Comments

  • gabriel

    August 14, 2014

    It is possible to specify what tests have to be started for an certain build ?
    To give you an example, if my project contain 100 files but for the current build only 10 files were modified, I need to run only the tests related to those 10 files and not all the tests.

    Reply
  • Anonymous

    September 9, 2014

    Getting error as [SSH] Opening SSH connection to :22.
    ERROR: Failed to authenticate as jenkins. Wrong password. (credentialId:4af42382-5f62-4141-ba2d-d37e33ac986f/method:keyboard-interactive)
    [09/09/14 15:45:32] [SSH] Authentication failed.
    hudson.AbortException: Authentication failed.
    at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1143)
    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:648)
    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:642)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
    [09/09/14 15:45:32] [SSH] Connection closed.
    [09/09/14 15:45:32] Launch failed – cleaning up connection

    Reply
  • Aditya

    September 9, 2014
    Reply
  • Vani

    September 28, 2014

    Hi,
    Just a small confirmation.We can use Jenkins for selenium webdriver C# testcases right.As i have searched net but could not get clear information whether jenkins can be used for C#.Can you please confirm.

    Thanks in advance,
    Vani

    Reply
    • Aditya

      September 30, 2014

      hi Vani,
      You can run selenium testing written in C# suing command line. Once you are able to run them, you can use the same command using jenkins. Let me know if you able to run your test. if not then let me know.

      Reply
  • Rajesh

    June 4, 2015

    Hi Aditya,

    I am very new on jenkins and want know how to setup testing job for my web application.
    I am able to build my web project which is developed in asp.net, classic ASP and back-end as SQL server.
    To run this project we host website into IIS.
    I want to execute some basic test for this application through Jenkins. Please help me to understand how we can setup basic testing for web application which require IIS. Please show me example like this blog. I don’t know what plugin I should use. Please help.

    Reply
  • debasish

    May 16, 2016

    How to do integration with server based jenkins(which one present in Linux platform) to selenium codes, those are available in stash repository).

    Reply
  • siva

    June 23, 2016

    can you send me the link of how to execute tests using msbuild, msTest and Nunit and another article for JUnit.

    Reply
  • Mandar

    August 4, 2016

    Hi Aditya,

    i have few queries related to my project , We have BAT concept (build Acceptance Testing) , We usually get the build once or twice a week , there is a separate dedicated team for that , They deploy the latest builds on Manual , Automation and Performance testing servers . So post this build on Automation Servers , we need to execute some BAT scripts which verifies Crud operations , Hardly there are 20-30 test cases . We run this suit of BAT Scripts through automation Simply by selecting the suit and selecting environments(Schemas) and Run . I want this to be done automatically … So could you please help me with this , Below are my requirements

    1) Need to run the BAT scripts automatically as soon as the build is deployed to Automation Server completely
    2) How this things can be achieved ?

    We are using below technologies
    Selenium Web Driver , C# , Visual Studio , tortoise SVN , MB Unit Framework

    Reply
  • Aditya

    August 5, 2016

    Hi Mandar,
    Thanks for reaching us. Did you try CORN Operation (https://en.wikipedia.org/wiki/Cron#CRON_expression) with jenkins?
    http://stackoverflow.com/questions/12472645/how-to-schedule-jobs-in-jenkins

    My personal approach would be first try to find a way how can I run all these test online. Once I that command then move that command into ***.bat file and then schedule this job in Jenkins.
    Refer(https://wiki.jenkins-ci.org/display/JENKINS/Schedule+Build+Plugin)
    Let me know if that is helps? Else reach me on learn@learnseleniumtesting.com, We can talk more in detail about your issue and see if we can solve or not.

    Reply
  • Kumar

    March 3, 2017

    Hi,
    I pulled Selenium maven project from gitHub, and build and run from jenkins, Execution working excellent but browser activity is not displayed. I searched but didn’t get solution, could you please help me.

    Reply
  • sonia

    April 10, 2017

    ii run test in windows firefox, but jenkins give me the following error:
    Unable to connect to host 127.0.0.1 on port 7055 after 45000 ms. Firefox console output:
    ox\\browser\\extensions\\{972ce4c6-7e08-4474-a285-3208198ce6fd}.xpi”,”installDate”:1262300400000,”updateDate”:1262300400000,”applyBackgroundUpdates”:1,”skinnable”:true,”size”:22012,”sourceURI”:null,”releaseNotesURI”:null,”softDisabled”:false,”foreignInstall”:false,”hasBinaryComponents”:false,”strictCompatibility”:true,”locales”:[],”targetApplications”:[{“id”:”{ec8030f7-c20a-464f-9b0e-13a3a9e97384}”,”minVersion”:”47.0.1″,”maxVersion”:”47.0.1″}],”targetPlatforms”:[],”seen”:true}
    1491833617073 addons.xpi DEBUG getModTime: Recursive scan of {972ce4c6-7e08-4474-a285-3208198ce6fd}
    1491833617074 DeferredSave.extensions.json DEBUG Save changes
    1491833617075 addons.xpi DEBUG Updating database with changes to installed add-ons
    1491833617075 addons.xpi-utils DEBUG Updating add-on states
    1491833617076 addons.xpi-utils DEBUG Writing add-ons list
    1491833617078 addons.xpi DEBUG Registering manifest for C:\Program Files (x86)\Mozilla Firefox\browser\features\e10srollout@mozilla.org.xpi
    1491833617079 addons.xpi DEBUG Calling bootstrap method startup on e10srollout@mozilla.org version 1.0
    1491833617079 addons.xpi DEBUG Registering manifest for C:\Program Files (x86)\Mozilla Firefox\browser\features\firefox@getpocket.com.xpi
    1491833617080 addons.xpi DEBUG Calling bootstrap method startup on firefox@getpocket.com version 1.0.2
    1491833617082 addons.xpi DEBUG Registering manifest for C:\Program Files (x86)\Mozilla Firefox\browser\features\loop@mozilla.org.xpi
    1491833617083 addons.xpi DEBUG Calling bootstrap method startup on loop@mozilla.org version 1.3.2
    1491833617116 addons.manager DEBUG Registering shutdown blocker for XPIProvider
    1491833617117 addons.manager DEBUG Provider finished startup: XPIProvider
    1491833617117 addons.manager DEBUG Starting provider: LightweightThemeManager
    1491833617117 addons.manager DEBUG Registering shutdown blocker for LightweightThemeManager
    1491833617117 addons.manager DEBUG Provider finished startup: LightweightThemeManager
    1491833617118 addons.manager DEBUG Starting provider: GMPProvider
    1491833617128 addons.manager DEBUG Registering shutdown blocker for GMPProvider
    1491833617129 addons.manager DEBUG Provider finished startup: GMPProvider
    1491833617129 addons.manager DEBUG Starting provider: PluginProvider
    1491833617129 addons.manager DEBUG Registering shutdown blocker for PluginProvider
    1491833617129 addons.manager DEBUG Provider finished startup: PluginProvider
    1491833617130 addons.manager DEBUG Completed startup sequence
    1491833617604 addons.manager DEBUG Starting provider:
    1491833617604 addons.manager DEBUG Registering shutdown blocker for
    1491833617605 addons.manager DEBUG Provider finished startup:
    Crash Annotation GraphicsCriticalError: |[0][GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0 (t=11.6273) [GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0
    1491833617878 DeferredSave.extensions.json DEBUG Write succeeded
    1491833617879 addons.xpi-utils DEBUG XPI Database saved, setting schema version preference to 17
    1491833617879 DeferredSave.extensions.json DEBUG Starting timer
    1491833617895 addons.repository DEBUG No addons.json found.
    1491833617896 DeferredSave.addons.json DEBUG Save changes
    1491833617900 DeferredSave.addons.json DEBUG Starting timer
    1491833617938 addons.manager DEBUG Starting provider: PreviousExperimentProvider
    1491833617938 addons.manager DEBUG Registering shutdown blocker for PreviousExperimentProvider
    1491833617939 addons.manager DEBUG Provider finished startup: PreviousExperimentProvider
    1491833617942 DeferredSave.extensions.json DEBUG Starting write
    1491833617967 DeferredSave.addons.json DEBUG Starting write
    1491833618016 DeferredSave.extensions.json DEBUG Write succeeded
    1491833618031 DeferredSave.addons.json DEBUG Write succeeded
    Crash Annotation GraphicsCriticalError: |[0][GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0 (t=11.6273) |[1][GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0 (t=20.9973) [GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0

    Command duration or timeout: 46.02 seconds
    Build info: version: ‘2.53.1’, revision: ‘a36b8b1cd5757287168e54b817830adce9b0158d’, time: ‘2016-06-30 19:26:09’
    System info: host: ‘sesjknboa01’, ip: ‘127.0.0.1’, os.name: ‘Linux’, os.arch: ‘amd64’, os.version: ‘3.10.0-514.6.1.el7.x86_64’, java.version: ‘1.7.0_07’
    Driver info: org.openqa.selenium.remote.RemoteWebDriver
    Caused by: org.openqa.selenium.remote.ErrorHandler$UnknownServerException:
    Unable to connect to host 127.0.0.1 on port 7055 after 45000 ms. Firefox console output:
    ox\\browser\\extensions\\{972ce4c6-7e08-4474-a285-3208198ce6fd}.xpi”,”installDate”:1262300400000,”updateDate”:1262300400000,”applyBackgroundUpdates”:1,”skinnable”:true,”size”:22012,”sourceURI”:null,”releaseNotesURI”:null,”softDisabled”:false,”foreignInstall”:false,”hasBinaryComponents”:false,”strictCompatibility”:true,”locales”:[],”targetApplications”:[{“id”:”{ec8030f7-c20a-464f-9b0e-13a3a9e97384}”,”minVersion”:”47.0.1″,”maxVersion”:”47.0.1″}],”targetPlatforms”:[],”seen”:true}
    1491833617073 addons.xpi DEBUG getModTime: Recursive scan of {972ce4c6-7e08-4474-a285-3208198ce6fd}
    1491833617074 DeferredSave.extensions.json DEBUG Save changes
    1491833617075 addons.xpi DEBUG Updating database with changes to installed add-ons
    1491833617075 addons.xpi-utils DEBUG Updating add-on states
    1491833617076 addons.xpi-utils DEBUG Writing add-ons list
    1491833617078 addons.xpi DEBUG Registering manifest for C:\Program Files (x86)\Mozilla Firefox\browser\features\e10srollout@mozilla.org.xpi
    1491833617079 addons.xpi DEBUG Calling bootstrap method startup on e10srollout@mozilla.org version 1.0
    1491833617079 addons.xpi DEBUG Registering manifest for C:\Program Files (x86)\Mozilla Firefox\browser\features\firefox@getpocket.com.xpi
    1491833617080 addons.xpi DEBUG Calling bootstrap method startup on firefox@getpocket.com version 1.0.2
    1491833617082 addons.xpi DEBUG Registering manifest for C:\Program Files (x86)\Mozilla Firefox\browser\features\loop@mozilla.org.xpi
    1491833617083 addons.xpi DEBUG Calling bootstrap method startup on loop@mozilla.org version 1.3.2
    1491833617116 addons.manager DEBUG Registering shutdown blocker for XPIProvider
    1491833617117 addons.manager DEBUG Provider finished startup: XPIProvider
    1491833617117 addons.manager DEBUG Starting provider: LightweightThemeManager
    1491833617117 addons.manager DEBUG Registering shutdown blocker for LightweightThemeManager
    1491833617117 addons.manager DEBUG Provider finished startup: LightweightThemeManager
    1491833617118 addons.manager DEBUG Starting provider: GMPProvider
    1491833617128 addons.manager DEBUG Registering shutdown blocker for GMPProvider
    1491833617129 addons.manager DEBUG Provider finished startup: GMPProvider
    1491833617129 addons.manager DEBUG Starting provider: PluginProvider
    1491833617129 addons.manager DEBUG Registering shutdown blocker for PluginProvider
    1491833617129 addons.manager DEBUG Provider finished startup: PluginProvider
    1491833617130 addons.manager DEBUG Completed startup sequence
    1491833617604 addons.manager DEBUG Starting provider:
    1491833617604 addons.manager DEBUG Registering shutdown blocker for
    1491833617605 addons.manager DEBUG Provider finished startup:
    Crash Annotation GraphicsCriticalError: |[0][GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0 (t=11.6273) [GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0
    1491833617878 DeferredSave.extensions.json DEBUG Write succeeded
    1491833617879 addons.xpi-utils DEBUG XPI Database saved, setting schema version preference to 17
    1491833617879 DeferredSave.extensions.json DEBUG Starting timer
    1491833617895 addons.repository DEBUG No addons.json found.
    1491833617896 DeferredSave.addons.json DEBUG Save changes
    1491833617900 DeferredSave.addons.json DEBUG Starting timer
    1491833617938 addons.manager DEBUG Starting provider: PreviousExperimentProvider
    1491833617938 addons.manager DEBUG Registering shutdown blocker for PreviousExperimentProvider
    1491833617939 addons.manager DEBUG Provider finished startup: PreviousExperimentProvider
    1491833617942 DeferredSave.extensions.json DEBUG Starting write
    1491833617967 DeferredSave.addons.json DEBUG Starting write
    1491833618016 DeferredSave.extensions.json DEBUG Write succeeded
    1491833618031 DeferredSave.addons.json DEBUG Write succeeded
    Crash Annotation GraphicsCriticalError: |[0][GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0 (t=11.6273) |[1][GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0 (t=20.9973) [GFX1-]: [D3D11] create swap chain failed: 0x887a0022, 0

    Build info: version: ‘2.53.1’, revision: ‘a36b8b1cd5757287168e54b817830adce9b0158d’, time: ‘2016-06-30 19:26:09’
    System info: host: ‘sesjknboa01’, ip: ‘127.0.0.1’, os.name: ‘Linux’, os.arch: ‘amd64’, os.version: ‘3.10.0-514.6.1.el7.x86_64’, java.version: ‘1.7.0_07’
    Driver info: driver.version: FirefoxDriver

    Reply

Leave a Reply