Browse Author: Tejas Modi

Selenium Testing beyond GUI Browsers.

“Selenium automates browsers”. So goes the introductory line on Seleniumhq.org. To what extent is this a limitation?

The topic of the blog provokes questions – Can we structure Selenium-based test framework to test beyond GUI browsers? If not, at the very least, how can we improve test-effectiveness by extending an existing Selenium test-suite to work with other test-actions that do not use browsers.

The first question above needs in-depth technical discussion. We focus on the second one here which can be addressed at a more conceptual level.

We have an example below which illustrates how one can improve the efficiency and coverage of automation by going beyond the Browser GUI and enabling backend operations.


Consider a simple web-application which has Selenium test suite containing hundreds of automated scripts that need to run daily. In addition, there are test-scenarios that need to be selectively executed during the day using some of the scripts. The framework is robust and the application relatively stable.

All scripts need to necessarily navigate across the web-application to get to specific web-pages. There are often many test scenarios that need to be executed on a specific web-page. The time for the script to get to that specific web-page depends on the performance of the application and latency in loading the intermediate web-pages. This would in turn increase the overall time needed to execute the automation suite when the script count is in hundreds. How can this be reduced?

Problem Statement: how do we limit the steps of navigating the web front-end using Selenium scripts so as to reduce the overall test-execution time?

Refer the illustration below.

Selenium Beyond GUI

Total Execution Time – 2 minutes.

We have the above scenario wherein there are parameters to be set on Web-Page-B which are necessary for validations on Web-Page-C. Every individual script logs-out and logs-on to the application, and total time taken is close to 2 minutes.

Now if there are 50 scripts that set different parameter combinations, the Selenium suite that runs scripts on the browser GUI would potentially take 50 * 2 = 100 minutes just for navigating back and forth on the web pages, especially if we need to log in and log out after every script.

The actual verification point however is only on Web-Page-C for every kind of parameter setting.

Selenium Beyond GUI img 2

Solution:

The parameter setting could be handled by Python or Perl scripts running in the backend. This would then cut down the navigation on the GUI.

Selenium Beyond GUI -img 3

The test flow is handled as below.

·       We test the end-to-end GUI navigation one time. The first test scenario covers this part.

·       At the same time, we trigger a script that directly accesses the backend. There could be several ways to do this – server side scripts, API calls, database queries… This depends on the application architecture and what is being tested.

·       The parameter is set at backend, the validation is done by the Selenium script on web-page C, this step is iterated over 50 parameters to be set for the 50 scenarios to be tested

The key here is to enable the automation framework to detect when the backend parameter is set to progress onto GUI validation, and then continue iterating between the two steps.

Total execution time now comes down drastically since webpage navigation is no longer needed.

This is an example of how existing Selenium suite can be extended with backend operations that improve automation efficiency and overall test effectiveness. The concept is proven; the implementation is heavily dependent on application architecture and specific test scenarios.

Selenium versus HP LeanFT

Selenium has carved a niche in the software testing tools world, and has a dedicated user base with consistently increasing adoption in the last few years. While this tool was always popular with Open Source enthusiasts since the RC days, we now have increased acceptance in enterprises as well. In the last couple of years, quite a few Fortune 500 companies and banks have diversified their skill-base and tool portfolio with Selenium, in addition to traditional HP toolset.


Let’s look at a recent product offering in the market-place, which could be very relevant to continued growth of Selenium user base. This post describes a brief analysis of HP LeanFT with respect to Selenium. Unsure if Learn-Selenium blog is the right medium for comment on HP tool, but it seems that this tool is HP’s response specifically to counter the increasing popularity of Selenium in the testing world.

Selenium – what has worked

1. Cost. Cost. Cost.

Cost by far remains the biggest differentiator for Selenium. Being open-source, this becomes the automation tool of choice for browser based applications in small and medium enterprises. Vibrant user community and strong support base help mitigate concerns around open source usage in enterprises. HP licensing is disproportionately expensive.

2. Object Identification

As web technologies get advanced, we have third party toolsets that cause issues in object identification during automation with HP QTP-UFT. AngularJS, Ajax, Oracle Forms are examples. While HP keeps refining with every version, there are easy alternatives. Selenium uses XPATH, and identifies objects where we face challenges in detecting unique properties using QTP.

3. The Buzz around Dev-Ops

Dev-Ops is the approach of leveraging test assets and automation in Development and Operations. With increasing agile adoption, the lines blur between traditional roles of developer, functional tester, and test automation specialist.

Application Development Leads and Architects are interested in test-automation for continuous integration, build sanity, and unit-testing. This is a community with expertise in Java/C#, and very comfortable with IDEs like Eclipse. They find it difficult to digest that anything worthwhile can be done with VBscript. These stakeholders are often key influencers in management decision-making on Enterprise Tool Usage, leading to increased acceptability for Selenium in large enterprises.

4. Multi-browser Testing

Even today, Selenium is a clear winner in cross-browser testing against UFT. Multiple UFT add-ins have to be tried for different browser versions and we have compatibility issues. Examples – UFT 11.5 does not support Chrome v40, you need to downgrade to Chrome v36 for automating scripts, which would not be in sync with production. HP license upgrades do not keep pace with browser version changes.

Selenium – where it falters

1. End-To-End Automation

Large Enterprises have multiple applications under test and end-to-end testing flows that traverse more than one application. Any tool restricted to browser testing would limit coverage of automation. Example – A very common scenario in banking systems would be transaction initiated on front-end web application that would have validation step on mainframe and backend database.

2. Object Identification and Script Build Productivity

While XPATH usage helps identify problematic objects, we have lot of instances where QTP can easily get unique property index, which may be cumbersome in Selenium. HP QTP/UFT are feature rich and easy to use. Invariably, script build productivity is higher as compared to Selenium, although this would vary based on application under test.

3. Skill-base and staffing

HP QTP has been the industry leader since ages, and sourcing experienced automation testers skilled in coding with VBscript is relatively easier. In comparison, ramping a project team on Selenium skills may be more of a challenge. Note- this is a snapshot as of mid-2015, things change very fast.

HP LeanFT – What’s on offer

Circa 2015 July, HP has introduced LeanFT along with the UFT 12.5 upgrade. Refer the figure below.

image

    Figure: Reproduced from datasheet on HP home-site

We have detailed below features and observations of LeanFT, which bear relevance to the analysis above.

1. LeanFT provides Support for multiple IDEs (Eclipse, Visual Studio) and coding languages (Java, C#).

HP keeps pushing features every few years to retain its dominance (and premium licensing), BPT was introduced to sell the concept of BA–Tester, now LeanFT is built to whet the interest of the Dev-Tester. IDE and language flexibility would make the tool popular with the Application Developers community.

2. Dev-Ops and CI support. LeanFT supposedly integrates well with standard SCMs, build/deploy tools and approaches, as compared to UFT which is heavily ALM centric. Selenium was an easy choice compared to QTP in building CI/CD solutions closely integrated with dev-workflows. This may change with LeanFT and needs to be investigated.

3. Object identification & Multi-browser support. LeanFT has an object identification engine similar to Spy in UFT which is installed as plug-in to the IDE. This is an advantage and could be quite powerful. HP datasheet indicates LeanFT to be light weight tool with good cross browser support. This could potentially address a gap in HP toolset where Selenium has an edge over QTP.

4. Cost: LeanFT licenses are free for HP UFT12.5 users, this helps penetrate the existing user base in large organizations and halts the move to Selenium.

5. We have clean integration of LeanFT with UFT12.5, which aids in end to end automation. This permits automation beyond browser based applications, a clear advantage over Selenium in organizations like banks and insurance companies.

6. QTP-skilled staff have no learning curve to start automation using LeanFT. Existing resources can be used for automation, as against Selenium projects which need well-thought out staffing and training strategy.

To Conclude:

It seems that LeanFT has been specifically targeted at the Selenium user base – would be very interesting to see how this pans out in the marketplace in the next 12 months. Much of the recent increase in Selenium projects has been because of large organizations seeking to diversify their tools portfolios. Technology trends are extremely dynamic and you prepare today for anticipated changes or risk obsolescence. Would LeanFT stem the tide? – We would watch the events and follow up on this in another six months.