In the Artificial Intelligence-Human Language Technologies team at eBay, we work on software that powers eBay’s conversational bot, ShopBot. We ship software daily to production that makes our bot intelligent, smarter, and more human. As a crucial part of this effort, we have to make sure any regressions are caught quickly and fixed to help keep our customers doing what they love – making purchases on ShopBot.
ShopBot’s backend is built on a polyglot suite of Scala, Java, and Python-based microservices that work in unison to provide ShopBot’s functionality. Hence, many of the crucial services need to be regression tested before we can release a new version to production.
To help with that effort, we built and are open sourcing our regression testing tool, Regressr.
Why Regressr
We looked at the common approaches that are widely used in the industry today to build an automated regression testing suite. In no particular order, they are listed below.
- Comprehensive JUnit suite that calls two versions (old and new) of the service and compares the minutiae of the responses – JSON elements, their values and the like.
- Using SOAP UI’s Test Runner to run functional tests and catch regressions as a result of catching functionality failures.
- No regression tests. Wait for the front-end to fail as a result of front-end regression tests in dev or test, and trace the failure to the backend.
We also looked at Diffy and were inspired by how simple it was to use for catching regressions.
We had some very unique requirements for testing eBay ShopBot and found out that none of these tools provided the features we wanted:
- Super-low ceremony: Must quickly be able to productionize and gain significant value without too much coding or process.
- Low conceptual surface area: An engineer should be able to grok what the tool does and use it quickly without going through bricks of manuals and frameworks.
- Configurability of comparisons: We want to able to specify how the response should be compared. Do we want to ignore JSON values? Do we want to ignore certain elements? What about comparing floating point numbers, precision, etc.?
- Compare at high levels of abstraction: We want to capture high-level metrics of the responses and then perform regression testing on them. For example, we would like to be able to say the number of search results in this response were 5 and then use that number to compare against future deployments.
- Low maintenance overhead: We want maintenance of the regression suite to have low or negligible coding effort. Once every deployment is approved for release, we just want the suite to automatically capture the current state of the deployment and use that as a reference for future deployments.
- CI/CD Integration: Finally, we wanted this suite to be hooked into our CI/CD build.
We built Regressr specifically to solve these requirements, so that the team can focus on the important stuff, which is serving great experiences and value to our customers who use ShopBot.
Regressr is a Scala-based command line tool that tests HTTP services, plain and simple. We built it to be really good at what it does. With Regressr, you can use the out-of-the-box components to get a basic regression test for your service up and running quickly and gain instantaneous value, while coding regression tests that will cover close to 100% of the functionality in a more delayed fashion as time permits. Finally, Regressr doesn’t even need the two services to be up and running at the same time, as it uses a datastore to capture the detail of the baseline.
Regressr works in two modes:
- Record – Use Record when you want to capture the current state of a deployment to be compared as the baseline for later deployments. A strategy file is specified that contains the specifics of what needs to be recorded.
- Compare/Replay – Compares the current state of a deployment with a baseline and generates a comparison report.
The image below captures what is done in these two flows.
The Strategy File
The strategy file is the configuration that drives what happens during a record and a compareWith execution.
An example strategy file that posts two requests and performs regression testing is specified below:
service:
baseURL : http://localhost:9882/endpoint
commonHeaders:
Content-Type : application/json
requests:
- requestName: say_hello
path: /say_hello
method: GET
recorder: org.ebayopensource.regression.internal.components.recorder.SimpleHTTPJSONRecorder
comparator: org.ebayopensource.regression.internal.components.comparator.SimpleHTTPJsonComparator
- requestName: shop for a pair of shoes
path: /shopping
method: POST
requestBuilder: org.ebayopensource.regression.example.ExampleRequestBuilder
dataInput:
conversationId : 12345
keyword : Show me a pair of shoes
mission_start : yes
recorder: org.ebayopensource.regression.internal.components.recorder.SimpleHTTPJSONRecorder
comparator: org.ebayopensource.regression.internal.components.comparator.SimpleHTTPJsonComparator
- requestName: say goodbye
path: /goodbye
method: POST
requestBuilder: org.ebayopensource.regression.internal.components.requestBuilder.StringHTTPBuilder
dataInput:
payload : '{"mission" : "12345", "keyword" : "good bye", "mission_start" : "no" }'
recorder: org.ebayopensource.regression.internal.components.recorder.SimpleHTTPJSONRecorder
comparator: org.ebayopensource.regression.internal.components.comparator.SimpleHTTPJsonComparatorThe Components
The important parts of the strategy file are the different components, RequestBuilder, Recorder, and Comparator.
RequestBuilder is used to specify how the request should be built in case of a POST or a PUT request.
The interface for RequestBuilder accepts a Map of Strings and outputs the payload that will be sent in the request.
abstract class RequestPayloadBuilder {
def buildRequest(dataInput: Map[String, String]): Try[String]
}Recorder is used to specify what parts of the response should be recorded for future comparison. Regressr injects all parts of the response to the Recorder during this time.
The interface for Recorder accepts a List of HTTPResponses (most of the time this will be one) and return a RequestRecordingEntry.
The RequestRecordingEntry is a holder for a value that will be recorded in Regressr’s datastore. The response code can be stored in a RequestRecordingEntry. Similarly a JSON response can be stored in a RequestRecordingEntry. You can also do some computation on the JSON and store a number (like the number of search results).
The interface for Recorder looks like the below.
protected def record(responses: Seq[HTTPResponse]) : Try[RequestRecordingEntry]Finally, the Comparator is used to specify the details of comparison during the compareWith mode. How do you want to compare JSON’s? What about strings?
The interface for Comparator looks like the below. It accepts both the recorded RequestRecordingEntry and the current one and returns a List of CompareMessages which will be included in the comparison report.
abstract class Comparator {
def compare(recorded: RequestRecordingEntry, replayed: RequestRecordingEntry): Try[Seq[CompareMessage]]
}Regressr comes with out-of-the-box components that can be plugged in to provide significant value instantaneously for many common types of services. However, you can write your own components implementing these interfaces and include them into Regressr (Use ./regressr.sh -r to build everything)
The comparison report is generated at the end of the compareWith lifecycle and looks like this:
Testing HATEOAS services
HATEOAS (Hypermedia As The Engine Of Application State) is where some classes of RESTful services tend to go to, especially when there are lightweight GUIs in front of them that mimic the conversation which happens to the service. Regressr also supports simple and efficient breadth first traversal of HATEOAS resources for regression testing.
We support this through the use of a new component class called as Continuations.
Let’s imagine you have a shopping cart service exposed at a URL such as /shoppingCart/items.
When issued a GET request on this URL, if the services is modeled on HATEOAS principles the results will be similar to:
{
"items": [
"item1": "http://<host>/shoppingCart/items/<item-id>/",
"item2": "http://<host>/shoppingCart/items/<item-id>/",
"item3": "http://<host>/shoppingCart/items/<item-id>/"
]
}As you can imagine, these new requests are non-deterministic and cannot be modeled with the help of Regressr’s configuration files, because the data may change over time.
That is where Continuations come in. With continuations, the tester can specify how many new requests should be created programmatically based on the response of a previous service call.
This allows the tester to write a continuation generically that creates new requests based on how many items were present in the response of the /items call.
An example of continuations is here.
What’s Next
- Maven plugin that attaches to Regressr that can be used in a CI/CD build.
- Jenkins plugin for Regressr report.
- Global comparators that can be used to capture global metrics across requests and compare them.
Conclusion and Credits
We have found Regressr to be a very useful regression testing tool for lean and low ceremony engineering teams that wish to minimize effort when it comes to regression testing of their services.
There were many people involved in the design, build and testing of Regressr without which this could not have been possible. Recognizing them: Ashwanth Fernando, Alex Zhang, Robert Enyedi, Ajinkya Kale and our director Amit Srivastava.
Comments and PRs are welcome with open hands at https://github.com/eBay/regressr.