Last year, our Gumtree Australia team started to use behavior-driven development in our agile development process. Initially, we searched for an open-source BDD framework to implement for BDD testing. Since we work on a web application, we wanted our automated BDD tests to exercise the application as a customer view, via a web interface. Of course, many web testing libraries do just that, including open-source tools such as Selenium and WebDriver (Selenium 2). However, we discovered that no existing BDD framework supports web automation testing directly.
We decided to simply integrate the open-source BDD tool JBehave with Selenium, but soon learned that this solution doesn’t completely fulfill the potential of automated BDD tests. It can’t drive the project development and documentation processes as much as we expected.
Different stakeholders need to view BDD tests at different levels. For instance, QA needs results to show how the application behaved under test. Upper managers are not so interested in the finer details, but rather want to see the number and complexity of the features defined and implemented so far, and if the project is still on track.
We needed a testing framework that could let us express and report on BDD tests at different levels, manage the stories and their scenarios effectively, and then drill down into the details as required. And so we started to develop a product that matches our needs. We named this product the Behavior Automation Framework (Beaf) and designed it to make the practice of behavior-driven development easier. Based on JBehave as well as more traditional tools like TestNg, Beaf includes a host of features to simplify writing automated BDD tests and interpreting the results.
Beaf’s extensions and utilities improve web testing on WebDriver/Selenium2 in four ways:
- Supplying a Story web console, where PMs/QAs can manage all the existing stories, divide stories into different categories or groups, and create/edit stories using the correct BDD format.
- Organizing web tests into reusable steps, and mapping them back to their original requirements and user stories.
- Generating reports and documentation about BDD tests. Whenever a test case is executed, Beaf generates a report. Each report contains a narrative description of the test, including a short comment and screenshot for each step. The report provides not only information about the test results, but also documentation about how the scenarios under test have been implemented. Reports serve as living documentation, illustrating how an application implements the specified requirements.
- Incorporating high-level summaries and aggregations of test results into reports. These overviews include how many stories and their scenarios have been tested by automated BDD tests. Taken as a set of objective metrics, test results show the relative size and progress of each feature being implemented.
When executing a test, whether it be with JUnit, jMock, or another framework, Beaf handles many of the Selenium 2/WebDriver infrastructure details. For example, Beaf can run cases cross-platform, including desktop (Firefox, Chrome), mobile (iPhone), and tablet (iPad). Testers can opt to open a new browser for each test, or use the same browser session for all of the tests in a class. The browser/device to be used can also be set in the Beaf configuration.
Using common steps provides a layer of abstraction between the behavior being tested and the way the web application implements that behavior. This level of abstraction makes it easier to manage changes in implementation details, because a desired behavior will generally change less frequently than the details of how it is to be implemented. Abstraction also allows implementation details to be centralized in one place.
@When("posting an Ad in the \"$category\" category")
public void postingAd(@Named("category") String category) throws Throwable
{
// 1. PageFactory will generate ad post pages under specified category.
// 2. Page status will be sent to next step by ThreadLocal.
postAdPage.set(PostAdPageFactory.createPostAdPage(category, AdType.OFFER.name()));
}
In addition to test reports, Beaf supplies a very useful web module called the Beaf Dashboard, which provides a higher-level view of current status. The dashboard shows the state of all of the stories, both in terms of their relative priorities and in terms of how many P1/P2/P3 stories and scenarios are fully, partially, or not automated. This information gives a good idea of the amount of work involved in implementing different parts of the project. The dashboard also keeps track of test results over time, so that users can visualize in concrete terms the amount of work done so far versus the estimated amount of work remaining to be done.
Beaf facilitates QA people joining projects in stages. It provides PD/QA with the detailed information required to test and update code, while giving business managers and PMs the more high-level views and reports that suit their needs. However, the greater potential of Beaf is the ability to turn automated web tests into automated web acceptance testing, in the true spirit of BDD.