In a microservices architecture, services are typically integrated through remote procedure calls or asynchronous messages. The traditional way of testing microservice integration is through end-to-end integration tests. Unfortunately, the integration environments may be unstable due to external dependencies, which makes end-to-end testing brittle and less efficient. This is quite common in real-world scenarios.
Another challenge for our Notification Platform team at eBay is that our APIs are consumed by many domain teams. Maintaining compatibility with all consumers while evolving our service APIs is a fundamental principle for us. In this article, we will introduce how at eBay we addressed the above challenges by adopting contract testing, and the tools we have built to deeply integrate the consumer-driven contract testing workflow into eBay’s development ecosystem.
Seeking a Solution
Our main objective is to explore a reliable way of evolving APIs with backwards compatibility. Since our API is based on the OpenAPI specification, we evaluated the possibility of ensuring API compatibility via OpenAPI schema evolution with semantic versions. This approach is wholly managed by the service provider. For example, even if a data attribute rename would not break the consumer’s data consumption (e.g. deserialization), we still need to be conservative in making the change, as we do not know whether a customer uses this particular data attribute. The OpenAPI specification on its own does not reflect how the API consumers depend on the schemas. Sometimes the service provider has to opt for data attribute redundancy to ensure safety. This solution also doesn’t address the fragility of the end-to-end testing problem.
We needed a way to make the consumer’s dependencies formal, just like the provider’s API specification. We first looked at the BDD (Behavior Driven Development) approach of formalizing the API consumer’s requirements into behavioral specifications with a domain specific language (Gherkin, in our case). Based on the BDD specification, collaboratively defined by the API consumers and providers, we can implement provider-side test cases to cover the verification of the API behaviors in our unit tests. This seemed to fill the information gap between the consumer and provider. But what if the consumer changes the requirements without updating the behavioral specification? How can we guarantee the consumer’s behaviors are always consistent with the specification? Using BDD to ensure API backward compatibility still largely relies on the technical personnel who are performing the process. This approach only works in those ideal cases where everything goes exactly as planned (i.e., API provider’s functional test cases can always cover all consumer behaviors).
Given the flaws of the BDD approach, we looked into another approach to solve this problem: contract testing. A contract, in this case, is a minimal agreement of API behaviors between the consumer and provider. Depending on different implementations, the consumers may explicitly set up the API expectations against mocks (or stubs) in their test cases, and these expectations can be later translated to a programming-language-agnostic intermediate file which describes the interactions of the contract. The API provider, however, needs to satisfy all the consumer contracts in the provider’s verification tests. By utilizing contract testing, we can establish a systematic workflow to make the process enforceable and detect compatibility issues early.
Contract testing promotes the idea of isolated unit tests against the integration point based on a predefined contract rather than a real end-to-end interaction, making it comparatively fast and stable.
Contract Testing Frameworks
Within the Notification Platform team, we evaluated two popular contract testing frameworks: Spring Cloud Contract and Pact.
A contract can be provider-driven or consumer-driven. In a provider-driven approach, the contract is defined by the API provider; the API consumer-side unit test relies on the API stubs derived from the contract. This is sometimes useful in isolated tests in a multi-component system, where the application developer is in control of both the provider and consumer. Interactions between components can be simulated with stubs, and there is no communication gap between the two parties.
A consumer-driven contract records each interaction from the consumer’s perspective. Different consumers may have different requirements, and the provider has the obligation to fulfill all the contracts. Compared to producer-driven contracts, this is a more widely accepted service testing paradigm to evolve services while maintaining backward compatibility.
The Consumer-Driven Workflow
Spring Cloud Contract was initially built as a provider-driven framework, but can achieve consumer-driven contract testing through a predefined workflow:
The service consumer clones from the shared contract repository, and then adds a new contract or modifies the existing contract in a feature branch.
The service consumer generates stubs from the defined contract, installs the stubs into the local file system, and writes test cases using local generated stubs.
After the test cases pass the contract verification, the service consumer can create a pull request of the contract from the feature branch to the main branch.
The service provider implements the API to satisfy the consumer-defined contracts, writes the verification test to make sure the implementation fulfills the contract and then merges the pull request.
The service provider can then publish a final version of the generated stubs into the remote stubs repository.
The consumer updates their stub dependency from the feature branch to the released version.
As illustrated in the above diagram, we see that the workflow entails multiple steps and involves back-and-forth communication between the two parties. Most importantly, the contracts are manually managed and maintained. For an API with multiple consumers, all the teams need to collaborate on a shared contract repository and follow a predefined folder structure to organize the contract files. This adds extra complexity and effort in communication and maintenance.
The workflow in Pact is much more straightforward:
The service consumer defines the service expectations using Pact-provided mock DSL.
The Pact contract definition DSL also generates a working mock that can be used in a unit test. After the consumer has passed the unit test, the mock file is uploaded to the Pact broker.
The Pact Broker replays all the stored contracts against the service provider and compares the responses with the contracts.
Pact introduces a contract management system called the Pact Broker (the commercial version is called Pactflow). The broker functions as a contract store with many other features. The Pact Broker solves many drawbacks of the manually managed shared contract repository in Spring Cloud Contract and makes contract testing more applicable in cross-team collaboration scenarios.
In the Notification Platform team at eBay, we first implemented all the test cases in Spring Cloud Contract. However, after learning about Pact and evaluating both frameworks, we found the workflow in Pact is much more suitable for our collaboration scenarios, and we reimplemented all the test cases using Pact.
In Spring Cloud Contract, the user usually defines the HTTP service contract in Groovy (or Yaml, Kotlin, or Java) DSL. The Groovy contract DSL is very expressive and flexible, but the IDE tooling support is relatively weak (as in auto-completion for the contract definition API); the developer needs to refer to the documentation for a slightly complicated use case. Spring Cloud Contract primarily targets the JVM technology stack and is more closely integrated with the Spring ecosystem. Although it claims to support other technology stacks through Docker containers, the user experiences can be quite different.
On the other hand, Pact provides native API bindings for major programming languages. API consumers use the Pact contract DSL to define API expectations in unit tests. The Pact DSL translates API behavior expectations into mock services that are available locally and can be used in test cases. After passing the test case, the recorded contract file will be transferred to the Pact Broker. Providers will replay all the contracts to ensure none of them break. Pact’s native API bindings lead to a better IDE tooling support, but there is a bit of a learning curve.
Pact also provides a state management API for controlling the pre- and post-conditions of each interaction. Unlike Spring Cloud Contract, provider validation tests are not automatically generated, but Pact provides libraries to automate much of the work.
From our experiences, Spring Cloud Contract is like a helper library that enables you to achieve contract testing in Spring. It has tighter integration with Spring and is based on popular libraries from the Java ecosystem. Pact, however, is a full-featured contract testing solution. It has its own set of libraries and contract management tools and specifications.
Spring Cloud Contract supports a wide range of integrations: Apache Camel, Spring Integration, Spring Cloud Stream, Spring AMQP, Spring JMS and Spring Kafka, to name a few. The messaging support is to a large degree based on the Spring messaging abstraction. From the consumer side, generated stubs can be triggered by a method or a message, or manually triggered through the StubTrigger interface. If a user specifies triggering by a message in the contract, the consumer’s test cases can actually be triggered by a real message, which more closely simulates the real integration.
Pact, however, takes a different approach: it abstracts away the message medium, focusing on unit-testing of the message handler logic with the mocked message. There could be a gap between the actual handler expected parameter data types and the mocked message data types.
To summarize the message contract testing support, Spring Cloud Contract provides seamless integration for Spring messaging abstractions and can also simulate real message interactions, whereas Pact is flexible but requires a little bit of manual integration effort.
Contract Testing at eBay
The Pact Initializer Project
A properly implemented contract testing workflow requires interaction with Pact Broker at various stages throughout the application development process. The Pact initializer project, built by eBay's Application Platform team, is a set of bootstrapping services for integrating Pact contract testing framework into eBay's development ecosystem.