16th January 2018
Razvan Vuşcan, one of our Senior Test Automation Engineers at Lola Tech, discusses one test automation framework model using Selenium Webdriver that aims to prevent certain defects and allows for scalability and cross-browser testing.
Working on test automation is both challenging and rewarding. In the pursuit of improving test scripts, covering as many scenarios as possible or creating rock solid test automation frameworks we are in a continuous cycle of research and experimenting. Learning automation can be easy, but learning to create a solid test automation framework that follows good case practices can be difficult. There are many pitfalls which should be avoided, otherwise we might end up with an unorganized, flaky test suite containing unclear or repetitive test methods that follow no pattern, where everything seems to spiral out of control and maintenance becomes a nightmare. In the following pages of this article we are going to propose one test automation framework model using Selenium Webdriver that aims to prevent these defects and allows for scalability and cross-browser testing.
Context and Needs
All in all, there's a lot of nice features that Selenium Webdriver offers for web automated testing. Books like “The Selenium Guidebook” by Dave Haeffner or “Mastering Selenium Webdriver” by Mark Collin – which I strongly encourage you to read - provide complete examples of how these features work along with other ins and outs. Over the following pages, I'd like to talk about how my team and I tried to extract some of these teachings and mix them with our collective experience in order to create one of the test automation framework models that we're currently using on a live project, and how we managed to overcome the flakiness associated with building a framework supporting several major browsers. Bear in mind though that what I will be discussing is just one approach that suits specific project needs, and that you are certainly free to explore and build a Webdriver test automation framework tailored to your needs.
In order to provide some context about the chosen solution I'd first like to go in brief over the needs of the project. There were three primary issues that the test automation framework needed to solve.
The second issue was related to having a clear, organized and scalable project structure that could meet business requirements. Until now I've had the good fortune of seeing several test automation frameworks, some clearer and more structured while others amorphous, having all tests or multiple page's elements in one single class, undocumented methods with unclear variables which belong to a specific page's class would be in another, etc. So in order to avoid these pitfalls, structure was needed which would allow clear separation of pages and their specific elements, features, and actions; allow for scalability by permitting the inclusion of methods and elements for new pages as well as associated tests, new modules or components; and eliminate code duplication by respecting the DRY principle (do not repeat yourself). Some methods needed to be carefully planned with the potential changes that can occur to a page's state in mind, in order to avoid having several similar functioning methods that only differed by a parameter (such as when performing a search and sort on a specific list of entities that could be labeled either active, inactive, or draft).
What we ended up with was a project structured centered around the Page Object Model where each page of the application would have its own Java class with methods and web elements dedicated solely to that page. The web elements defined in these Page Objects (with the @FindBy annotation) are initialized using the Page Factory Class which is also used to instantiate the Page Object. Each page objects has its own steps class that groups these methods under human readable steps. The scenarios themselves which are written using the human readable steps are laid out in Cucumber's Feature files,fl which in turn are grouped under the “resources” directory in specific packages pertaining to a module or feature of the application. Finally, a Webdriver initialization class which determined the type of browser required by each test, as well as several supporting classes for logging, working with databases, and reading property files were added to the framework.
Illustration 1: Steps class example for the Login Page Object. The first arrow points out the usage of Page Factory, while the second arrow shows how methods can be grouped under human readable steps which can accept custom parameters.
As the business was closely involved in the development process, tests needed to be easily readable and understandable by non-technical staff. In an effort to make the framework as accessible as possible to project newcomers we chose to build it using widespread, familiar technologies like Java, JUnit, Maven, and Cucumber on top in order to write the acceptance tests in a behavior-driver development style, make their steps human readable, group and customize them more easily, and generate complete and visually appealing reports.
But one thing to keep in mind here, which I also mentioned in the first part of the article, is the pitfall of automating every single test. It's pretty common knowledge that the business would like every possible scenario to be covered and have an associated automated test, however such a behavior can cause a lot of time to be spent on creating tests that are not going to be run that often or are for features which are non-critical and will rarely be used by end users. In the rush to automate everything testers start losing focus of the critical parts of the application - the “money-makers” - and leads to the stacking of too many tests that eventually become hard to maintain and sort out. Repetition and redundancy can set in if the framework is not organized.
An article by Mike Wacker on the Google Testing Blog suggests that only 10% of the automated testing done on a project should be End to End tests which simulate a real user's interactions with the application. 70% should be Unit tests, while 20% should be Integration tests. This can greatly reduce the flakiness and execution time associated with running tests which simulate real user scenarios. But this does not mean that we should fall in the other extreme where only the happy flows should be automated and nothing else. Supporting functionalities should also be covered by separate tests, even if they are not critical for users to reach their end goal. Negative scenarios also contribute to determine if the application is behaving as expected. All in all each test automation team needs to find a balance; one where there aren't hundreds of tests which become hard to maintain and track, nor where there are only a few End to End tests that check if the user can get from point A to point B without really checking for anything else.
The last issue that the test automation framework needed to address was how to reduce the number of resources involved in the testing process. This objective was intended for both testing staff and physical machines on which the tests ran. By increasing test coverage across several browsers, meant that manual testers would not have waste time testing on secondary supported browsers while the automated tests ran on the main browser. This meant that their efforts could be channeled into testing new features or performing other activities. Additionally, by using Jenkins as a CI tool, separate jobs were created for each module of the application so that when a change was made in a specific module the appropriate test suite could be ran first and detect any potential faults in the build. This, in addition to Cucumber's flexible Tags feature, allowed for the grouping on tests into user acceptance, smoke or full regression packages which could be run either on demand or on a fixed schedule. Another step included switching over from using the classic Selenium Grid, which ran over several physical machines, to Docker containers running Firefox and Chrome. Only one machine running a Windows environment was kept for running native tests on Internet Explorer while all other tests were ran using Docker containers. Practically each Jenkins job checks whether an image of the Selenium Hub is running, creates separate Selenium Node containers for Firefox and Chrome, runs the test suite on both browsers, and then tears down these two Node containers. Ideally the Selenium Hub container is kept alive so that a finishing job will not destroy it while another job might still be using it.
Using custom methods to overcome flakiness and support running tests across several browsers
The Webdriver initialization class lies at the core of the test framework and contains a custom method used to launch a specific driver type based on the browser argument passed by the tags in the Cucumber Feature files.
Illustration 2: Tag defining and associated Webdrvier initialization methods
In turn, these Feature files are run using the test classes located under Maven's “test” directory, which include the feature packages that should be picked up, where their corresponding steps are located, what type of report to generate, and the tags that should be included.
Illustration 3: Example test class which indicates which features should be run and for which browsers
A scenario annotated with the @Firefox tag will launch an instance of the Firefox driver, one annotated with @Chrome will launch an instance of Chrome Driver, etc. The advantage of using tags in Cucumber lies in the fact that test scenarios can be customized for the specific browser they are about to run on. Each feature file also supports a Background section which acts similar to the @Before annotation in JUnit.
Illustration 4: Feature file example showing the Background section run before each test annotated by the appropriate tag is run
Cucumber also allows for the easy grouping of tests depending on their intended category - for instance user acceptance tests can be annotated with an @Acceptance tag while smoke tests can have a @Smoke annotation alongside the browser annotation - in which case a Maven command like -Dcucumber.options=”--tags @Acceptance” will run only the scenarios in feature files which have that annotation, regardless of the specified browser annotation. This makes it really easy to scale tests in order to support additional browsers, specify a custom flow for any given scenario based on the browser it is currently running on, or run a specific sub-set of tests depending on the need.
In addition to the custom driver initialization method, the core class contains other customized clicking, get and set text methods, as well as Fluent Wait methods. The reason we opted not to settle for using the bare, vanilla methods that the Selenium API provides is that when running across different browsers the same script might not always work. As I mentioned before, different browsers, different loading speeds. This became especially obvious when trying to click an element immediately after navigating to a new page. Sometimes the element would not appear in the DOM for a certain amount of time, and the test would break. This mean that a way to effectively wait until an element was available in the DOM was needed. Using Thread.Sleep() or Implicit Waits was not an option. The first because it's forces the test to wait for a given amount of time regardless of conditions on the page (which can stack up the execution time of a test); the latter because it forces the test to wait for a given amount of time when trying to find an element that is not available immediately in the DOM. This will remain in place for the duration that the browser is open so that means that additional searches for elements could become affected by the Implicit Wait. Additionally, combining both Implicit and Explicit waits in a test automation framework can lead to some undesirable behaviors regarding the wait times.
As such, we opted for the use of Fluent Waits which set a maximum amount of time to wait for a condition to be fulfilled (i.e. an element to become available before being clicked) while periodically checking if the condition is met. That means that if the timeout is set for 10 seconds, but the element appears in two, the condition will be fulfilled and the test will move on without waiting for the remaining eight seconds. In addition, Fluent Waits can ignore specific exceptions while waiting for the condition to be met. This added an extra degree of fluency in our framework as well as mimicked human behavior since a real person will always wait for the page to load and the element to be visible before clicking it – average users don't immediately start compulsively clicking on the buttons of a page that hasn't even finished loading. Once the element would be available, the click action would finally be performed.
Illustration 5: Example of Fluent Wait method
In regards to getting the text of elements we opted for creating another customized method which would make sure that the returned String would not be null or blank. A text can come out blank when the actual String value is stored in the “value” attribute of the element. In this case WebDriver's getText() method, which gets the visible innerText of the element, will return an empty String. So a condition was added to the custom text retrieval method which would check if by using the default getText() method the String would be empty. If it is, then a second attempt to collect the text value would be made, this time using the .getAttribute(“value”) method. In general these two steps will work in most situations. The image below illustrates this example.
Illustration 6: Custom get text method example
However, an instance where the returned String might be null is when trying to retrieve the text from a box in a table. For example, if we are talking about a table containing users and their e-mail addresses. Usually, an e-mail address is not a mandatory field that needs to be filled out in a form. As such, some users from the table might not have an e-mail address associated. So when trying to select a random e-mail for the column in the table we might end up with a null String being returned. In this instance neither Webdriver's getText nor getAttribute methods are at fault – if there is no text at all in the table box nothing will be retrieved no matter which of the two methods are used. So in order to prevent this, a condition needed to be set which checks that while the returned String is null, a new table box should be selected until a valid e-mail String is returned.
The final example of a custom method that I want to discuss is one for setting text in input fields that have autosuggest result drop-downs. Depending on how fast a page would load and depending on the running browser, a set of jQuerry auto complete results could take a variable amount of time to become available. So in order to make sure that the list of results was populated and updated accordingly to what was being typed in the input field, a different way than setting the text directly using the sendKeys method needed to be used. This is because setting the text in one single shot would often times prevent the autocomplete results to even appear at all. A for loop was used which sent each character of the desired String into the input field one character at a time, emulating a real user typing. This made the auto complete results set to appear and update as characters were being typed. Finally, the text value set in the field would be retrieved and compared to the original desired value in order to make sure that no letters were left out.
Illustration 7: Set text method example
The final thing I'd like to touch in this article is related to including ways of contextually testing the pages of an application while tests are running. By this I mean including steps that help check if images are correctly loaded, files can be downloaded or URLs are broken or not. The beauty of adding such steps in automated tests is that these types of actions can be performed quickly and they can support in part the work done by the manual testers. This will help ensure that visual elements or links in the application are checked and are working as expected. This can be especially useful since from experience such testing sometimes tends to become overlooked over the course of the testing process.
This sums up the main aspects proposed in our test automation framework model. Of course, there will always be a way of making things better faster or more reliable. Perhaps you will find some of the examples illustrated over the course of this article useful, a match for your current needs, or perhaps you will feel the need to look elsewhere for answers to your challenges. There is no universal blueprint that is applicable in every situation, rather a set of guidelines and practices to keep in mind when building your own solution. My final piece of advice is to never stop exploring for new ways and possibilities that can improve your framework.
This article was originally posted in issue 6 of the Quality Matters magazine.
About the Author
Razvan Vuşcan is a Senior Test Automation Engineer at Lola Tech, responsible for writing automated test scenarios and other scripts that are used in the testing process. Lola Tech is a pioneering travel software business leading the field in travel tech innovation. With big-name clients globally, their talented team is adept at creating awesome systems to increase your uptime and conversion rates or to provide you with that innovative edge to outshine the competition.