The search for a new Specification–the WHYs
Implementation architecture has undergone a radical shift in the last two decades, leading us to a landscape that is more services oriented than ever before and certainly more diverse in terms of technology choices.
The transition from a legacy monolithic centralized architectural paradigm to the distributed lightweight micro-services-driven world has proved liberating in the choice of technology stacks, allowed greater consistency of experience, and improved efficiency and reduced TTMs. However, it has also surfaced the need to evolve the way service contracts are explored, tested, published, and integrated.
The realization that API contracts would need to meet the needs of seamless exploration and integration across a diverse technology stack, be industry standard, and be feature rich to complement our Technical Standards and governance models necessitated the exploration for a new specification.
The primary criteria was a specification that was both human and machine readable, language agnostic, vendor neutral, and open source.
The OpenAPI specification was the unanimous choice.
Introduction to OpenAPI–the WHATs
“The OpenAPI Specification (OAS) defines a standard, language-agnostic interface to RESTful APIs that allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection. When properly defined, a consumer can understand and interact with the remote service with a minimal amount of implementation logic.”
The OpenApi Specification is driven by the OpenAPI Initiative, created by a consortium of forward-looking industry experts who recognize the immense value of standardizing on how REST APIs are described. As an open governance structure under the Linux Foundation, the OAI is focused on creating, evolving, and promoting a vendor neutral description format.
The open source ecosystem that supports OpenAPI specification is one of the most advanced and feature rich compared to any other contemporary specification. Some of the open source tooling complementing OpenAPI includes interactive UIs that allow API exploration. The code generation support, incidentally, currently has the widest SDK generation capability across 40+ languages to boot strap both server side and client side integration.
The OpenAPI tooling ecosystem (a.k.a. swagger) is powered by one of the largest active open source community in its category.
While mature as a specification with strong open source tooling support, it is also an evolving standard led by industry leaders.
The relevance of OpenAPI–a few exciting specifics
OpenAPI scores over our legacy and, indeed, many other contemporary specifications in multiple ways.
Before discussing the possibilities that OpenAPI brings, this section highlights a few features of interest:
- Structural integrity and richness: To call out a few examples, the specification requires response definitions within the context of http status codes, has fairly rich support for data types and formats (for example: RFC3339 for date and date-time), and constraints (for eg: RFC3986 (reserved character support), ECMA 262 regex support), etc.
The specification also supports extensibility at almost every level. Also, quite importantly, it allows for fragmented contracts using remote references. This opens up the possibility of storing a set of governed common definitions and referencing those in a language-agnostic manner across different APIs.
With OpenAPI 3.0 GA Release, we now have support for multiple base paths and environments.
- Futuristic and evolving: OpenAPI 3.0 introduced the ‘links’ concept that allows for call associations with semantics that allow programmatic references back to a previous request or response structure (full Augumented BNF support). The support for webhooks was recently introduced as well.
- Tooling support: From open source interactive UIs for API exploration to code generation across 40+ technology stacks for client and server side to testing and mocking capabilities, OpenAPI ecosystem is one of the most feature rich in the industry.
- OpenAPI tooling opens up both code first and contract first approaches to API design and development.
- Finally, OpenAPI offers a fully open source and therefore a fully customizable stack.
We, however, need to acknowledge the risks associated with too much non-generic customization, which can make any implementation eventually resemble more of a homegrown solution.
A moderated approach that includes a loop contributing back to the open source community any generic patterns or tooling of relevance to the Industry is the recommended path forward.
The Opportunities—a glimpse at the possibilities ahead
This section discusses a few high-level opportunities that increased OpenAPI adoption brings.
First, OpenAPI decouples contracts from the implementation stack and allows contracts to exist and be shared, published, explored, and integrated in a stack neutral way.
OpenAPI-based tooling enables contract-first development and boot strapping development through customizable code generation templates and tooling.
To call out a few other opportunities across different aspects:
- Standards and Compliance: The rich data types and format support complements the new emerging eBay Technical Standards, and opens up possibilities for very detailed automated governance through compliance checks and reviews.
- Bootstrapping development across technology stacks: The only truly polyglot machine-readable specification, this makes service development and integration trivial across multiple languages.
- Downstream Service Integration: OpenAPI contracts make downstream service integration so much easier through interactive UIs for exploring the contracts and code generation of client artifacts.
- Continuous Delivery: Opens up possibilities for language agnostic test code authoring. Pipelines leverage OpenAPI code generation capabilities to execute automation suites.
- Application Security: Contracts with constraints defined for every inbound field for example could potentially be used to scan and filter out malicious payloads by upstream tiers way before it reaches the resource servers.
- External Development Community: OpenAPI provides seamless tooling and fast integrations support for our external developer community.
The opportunities discussed here are by no means exhaustive, but just a glimpse of opportunities that could make a huge difference in the days ahead.
A true micro-services-driven world would be hard to sustain without leveraging the machine-readable and language-agnostic nature of a contract specification.
A note on early days–the challenges and tactical tooling
We moved swiftly once the decision was made to adopt OpenAPI in April 2017, leading to our first publication of OpenAPI contracts to our external community in July 2017.
The challenge in those early days was introducing a new specification in an ecosystem that was heavily reliant on our legacy contracts without any disruption or rework on any of the existing tooling or processes.
The tactical solution was therefore to create an OpenAPI contract generator service that takes in a legacy contract and transforms it to an OpenAPI contract. Any aspirational OpenAPI-based tooling, therefore, would just need a simple pass through the transformation layer, as depicted in the diagram below:
Incidentally, because the idea was not limited to OpenAPI contracts for internal use, but also to the external world via our published Public API contracts, the generator needed capabilities to sanitize the contract for external releases.
Some of the key features required for external releases included defaulting serialization risks as well as having the generator strip out any other internal usage markers and clean up any resulting orphaned definitions.
For validating the contracts, we leveraged the swagger code-gen to redo our testing pipelines. The approach is detailed in the diagram below:
Conclusion
From a conversation in April 2017 to the first CD pipeline powered by OpenAPI in June 2017 to our first ever external Public API contract published in July 2017, we have made steady progress. The infographic below highlights some of our milestones from 2017:
eBay joined the OpenAPI organization in August 2017. With this move, we are happy to partner with the OpenAPI Initiative and contribute towards its evolution.
In terms of fully harnessing OpenAPI capabilities and contributing back, we do have some way to go, but the path is exciting, with many tangible benefits to us and the community at large.
References
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md