Introduction
In a microservices architecture, services are typically integrated through remote procedure calls or asynchronous messages. The traditional way of testing microservice integration is through end-to-end integration tests. Unfortunately, the integration environments may be unstable due to external dependencies, which makes end-to-end testing brittle and less efficient. This is quite common in real-world scenarios.
Another challenge for our Notification Platform team at eBay is that our APIs are consumed by many domain teams. Maintaining compatibility with all consumers while evolving our service APIs is a fundamental principle for us. In this article, we will introduce how at eBay we addressed the above challenges by adopting contract testing, and the tools we have built to deeply integrate the consumer-driven contract testing workflow into eBay’s development ecosystem.
Seeking a Solution
Our main objective is to explore a reliable way of evolving APIs with backwards compatibility. Since our API is based on the OpenAPI specification, we evaluated the possibility of ensuring API compatibility via OpenAPI schema evolution with semantic versions. This approach is wholly managed by the service provider. For example, even if a data attribute rename would not break the consumer’s data consumption (e.g. deserialization), we still need to be conservative in making the change, as we do not know whether a customer uses this particular data attribute. The OpenAPI specification on its own does not reflect how the API consumers depend on the schemas. Sometimes the service provider has to opt for data attribute redundancy to ensure safety. This solution also doesn’t address the fragility of the end-to-end testing problem.
We needed a way to make the consumer’s dependencies formal, just like the provider’s API specification. We first looked at the BDD (Behavior Driven Development) approach of formalizing the API consumer’s requirements into behavioral specifications with a domain specific language (Gherkin, in our case). Based on the BDD specification, collaboratively defined by the API consumers and providers, we can implement provider-side test cases to cover the verification of the API behaviors in our unit tests. This seemed to fill the information gap between the consumer and provider. But what if the consumer changes the requirements without updating the behavioral specification? How can we guarantee the consumer’s behaviors are always consistent with the specification? Using BDD to ensure API backward compatibility still largely relies on the technical personnel who are performing the process. This approach only works in those ideal cases where everything goes exactly as planned (i.e., API provider’s functional test cases can always cover all consumer behaviors).
Given the flaws of the BDD approach, we looked into another approach to solve this problem: contract testing. A contract, in this case, is a minimal agreement of API behaviors between the consumer and provider. Depending on different implementations, the consumers may explicitly set up the API expectations against mocks (or stubs) in their test cases, and these expectations can be later translated to a programming-language-agnostic intermediate file which describes the interactions of the contract. The API provider, however, needs to satisfy all the consumer contracts in the provider’s verification tests. By utilizing contract testing, we can establish a systematic workflow to make the process enforceable and detect compatibility issues early.
Contract testing promotes the idea of isolated unit tests against the integration point based on a predefined contract rather than a real end-to-end interaction, making it comparatively fast and stable.
Contract Testing Frameworks
Within the Notification Platform team, we evaluated two popular contract testing frameworks: Spring Cloud Contract and Pact.
A contract can be provider-driven or consumer-driven. In a provider-driven approach, the contract is defined by the API provider; the API consumer-side unit test relies on the API stubs derived from the contract. This is sometimes useful in isolated tests in a multi-component system, where the application developer is in control of both the provider and consumer. Interactions between components can be simulated with stubs, and there is no communication gap between the two parties.
A consumer-driven contract records each interaction from the consumer’s perspective. Different consumers may have different requirements, and the provider has the obligation to fulfill all the contracts. Compared to producer-driven contracts, this is a more widely accepted service testing paradigm to evolve services while maintaining backward compatibility.
The Consumer-Driven Workflow
Spring Cloud Contract was initially built as a provider-driven framework, but can achieve consumer-driven contract testing through a predefined workflow:
Figure 1: Consumer-driven contract testing workflow in Spring Cloud Contract
-
The service consumer clones from the shared contract repository, and then adds a new contract or modifies the existing contract in a feature branch.
-
The service consumer generates stubs from the defined contract, installs the stubs into the local file system, and writes test cases using local generated stubs.
-
After the test cases pass the contract verification, the service consumer can create a pull request of the contract from the feature branch to the main branch.
-
The service provider implements the API to satisfy the consumer-defined contracts, writes the verification test to make sure the implementation fulfills the contract and then merges the pull request.
-
The service provider can then publish a final version of the generated stubs into the remote stubs repository.
-
The consumer updates their stub dependency from the feature branch to the released version.
As illustrated in the above diagram, we see that the workflow entails multiple steps and involves back-and-forth communication between the two parties. Most importantly, the contracts are manually managed and maintained. For an API with multiple consumers, all the teams need to collaborate on a shared contract repository and follow a predefined folder structure to organize the contract files. This adds extra complexity and effort in communication and maintenance.
The workflow in Pact is much more straightforward:
-
The service consumer defines the service expectations using Pact-provided mock DSL.
-
The Pact contract definition DSL also generates a working mock that can be used in a unit test. After the consumer has passed the unit test, the mock file is uploaded to the Pact broker.
-
The Pact Broker replays all the stored contracts against the service provider and compares the responses with the contracts.
Figure 2: Consumer-driven contract testing workflow in Pact
Pact introduces a contract management system called the Pact Broker (the commercial version is called Pactflow). The broker functions as a contract store with many other features. The Pact Broker solves many drawbacks of the manually managed shared contract repository in Spring Cloud Contract and makes contract testing more applicable in cross-team collaboration scenarios.
In the Notification Platform team at eBay, we first implemented all the test cases in Spring Cloud Contract. However, after learning about Pact and evaluating both frameworks, we found the workflow in Pact is much more suitable for our collaboration scenarios, and we reimplemented all the test cases using Pact.
HTTP Integration
In Spring Cloud Contract, the user usually defines the HTTP service contract in Groovy (or Yaml, Kotlin, or Java) DSL. The Groovy contract DSL is very expressive and flexible, but the IDE tooling support is relatively weak (as in auto-completion for the contract definition API); the developer needs to refer to the documentation for a slightly complicated use case. Spring Cloud Contract primarily targets the JVM technology stack and is more closely integrated with the Spring ecosystem. Although it claims to support other technology stacks through Docker containers, the user experiences can be quite different.
On the other hand, Pact provides native API bindings for major programming languages. API consumers use the Pact contract DSL to define API expectations in unit tests. The Pact DSL translates API behavior expectations into mock services that are available locally and can be used in test cases. After passing the test case, the recorded contract file will be transferred to the Pact Broker. Providers will replay all the contracts to ensure none of them break. Pact’s native API bindings lead to a better IDE tooling support, but there is a bit of a learning curve.
Pact also provides a state management API for controlling the pre- and post-conditions of each interaction. Unlike Spring Cloud Contract, provider validation tests are not automatically generated, but Pact provides libraries to automate much of the work.
From our experiences, Spring Cloud Contract is like a helper library that enables you to achieve contract testing in Spring. It has tighter integration with Spring and is based on popular libraries from the Java ecosystem. Pact, however, is a full-featured contract testing solution. It has its own set of libraries and contract management tools and specifications.
Message Integration
Spring Cloud Contract supports a wide range of integrations: Apache Camel, Spring Integration, Spring Cloud Stream, Spring AMQP, Spring JMS and Spring Kafka, to name a few. The messaging support is to a large degree based on the Spring messaging abstraction. From the consumer side, generated stubs can be triggered by a method or a message, or manually triggered through the StubTrigger interface. If a user specifies triggering by a message in the contract, the consumer’s test cases can actually be triggered by a real message, which more closely simulates the real integration.
Pact, however, takes a different approach: it abstracts away the message medium, focusing on unit-testing of the message handler logic with the mocked message. There could be a gap between the actual handler expected parameter data types and the mocked message data types.
To summarize the message contract testing support, Spring Cloud Contract provides seamless integration for Spring messaging abstractions and can also simulate real message interactions, whereas Pact is flexible but requires a little bit of manual integration effort.
Contract Testing at eBay
The Pact Initializer Project
A properly implemented contract testing workflow requires interaction with Pact Broker at various stages throughout the application development process. The Pact initializer project, built by eBay's Application Platform team, is a set of bootstrapping services for integrating Pact contract testing framework into eBay's development ecosystem.
Figure 3: A overview of the Pact Initializer architecture
-
The Pact Initializer Portal collects configuration metadata for contract testing environment setup. After the developer enters the required configuration, the backend initialization service is triggered.
-
The backend initialization service parses the configuration metadata, generates corresponding initialization tasks and triggers the events.
-
The task executor will also send the execution result to the analytics backend for monitoring and further analysis.
Unified Provider Verification Service
When consumers change contracts, providers only need to verify compatibility with the new changes. Pact provides a webhook to bind each consumer to its provider validation CI/CD jobs. However, for each consumer, they need to repetitively set up service accounts and authentication configurations for Pact Broker to trigger the verification process. We can use a proxy service to manage the common steps of each validation and free the consumer from managing these configurations.
Figure 4: A unified verification service for triggering provider side verification process
-
Service consumers use the Pact Initializer Portal to initialize the environment.
-
The verification service stores the configuration metadata.
-
Service consumers publish the contract changes to the Pact Broker.
-
Pact Broker detects the changes and forwards the changes to the verification service.
-
The verification service reads the target verification job information from the datastore and triggers the CI jobs.
Contract Testing Best Practices
Choose the Appropriate Framework
Both frameworks can do a fairly good job in implementing contract testing workflow. There are a couple of considerations of choosing the best framework:
-
Your application stack. Non-Java applications may directly go to Pact, it provides native API bindings, and better user experience. If your application is Spring based microservices, Spring Cloud Contract may be useful and have tighter integration with Spring’s ecosystem.
-
Scale of the collaboration. If you’re working with multiple teams or applications, the Pact broker can save you from the contract management headaches.
-
Consumer-driven or provider-driven. Spring Cloud Contract allows you to define a provider-driven workflow and sometimes this makes sense: you either own both the consumer and provider application, or the provider can singly define the API behaviors (like public API).
The Robustness Principle
Be conservative in what you send, be liberal in what you accept. In the context of contract testing, this means the consumer code should only send request parameters they really depend on to the provider, and extra data attributes from the provider’s response should not break the consumer.
Focusing on the Communication Contract Rather Than the Function
Contract testing sits between unit testing and service integration testing. It does not try to replace functional tests, it replaces the dependency on a real provider’s response to verify the communication data format. Contract testing should only focus on the integration point, not the business logic (i.e., whether the provider’s response conforms to the business requirement). These areas should be covered by the provider's functional tests.
BDD Is a Good Complement to Contract Testing
As mentioned in the previous paragraph, contract testing should not test provider’s features or functions. BDD is a great methodology to formalize the consumer’s requirements and enforce the provider side’s functional test coverage. Therefore, it is a good complement to contract testing.
Avoid Unnecessarily Strict Assertions
The consumer should verify the data attributes loosely rather than strictly to enhance the robustness of the test cases. For example, if you don’t rely on the length of a data attribute, asserting the attribute is not empty is more maintainable than asserting the attribute with a fixed length. Usually, the consumer relies on the structure of the response, rather than the value content.
Conclusion
In this article, we have compared different contract testing frameworks and shared our experiences of the adoption at eBay. We believe contract testing is valuable for testing integration points in microservices architecture and can help evolve services with confidence.