New & Improved Terapeak Research 2.0 in eBay Seller Hub

Terapeak has been the seller’s darling when it comes to providing key insights for businesses of all sizes and utilized heavily to evaluate sourcing opportunities, pricing guidance, market trends and analyze their competition. These excellent attributes made Terapeak a match made in heaven for eBay. This post chronicles the journey of integrating this excellent tool into eBay SellerHub making it available to the millions of eBay sellers and describes the technical challenges, transformations and architectural changes performed to achieve the milestones in this transformative journey.

Terapeak, acquired by eBay in 2017, is a search tool that allows sellers to derive insights on what to sell, when to sell and how to sell a product on eBay.  Using Terapeak, sellers can determine the best price for their listings, the best shipping policy, the expected sell-through rate, and whether or not the product their selling is subject to seasonal trends. They can gain insights into the keywords that top-selling listings include in their titles, providing them with keywords suggestions that will surface their listings in the search results, putting them directly in the buyer’s line of sight.

These features make Terapeak a perfect addition to the Seller Hub, which serves to help sellers make better decisions by directly providing them with eBay’s data, including inventory management, performance insights, and growth opportunities. Sellers can read more about Terapeak on the Terapeak help page and it can now be found in eBay’s Seller Hub.

Product Research 1.0

From the day eBay acquired Terapeak, eBay’s seller growth team began working hard to integrate the core capabilities of Terapeak into eBay's Seller Hub. This work began with re-building Terapeak's large data backend on eBay’s core technology stack in preparation of exposing the actual features in Seller Hub for sellers to use.

Last year at eBay Open 2019, the initial features formerly part of Terapeak was officially launched in Seller Hub to all US sellers who had an eBay Store Subscription. The first release supported these capabilities:

• Conduct a keyword search for matching titles

• Conduct a product ID search for MPN, UPC, EAN, ISBN and EPID

• Search across any eBay site 

• Select from pre-set date ranges of 7, 30, 90 and 365 days

• Filter their results based on price, condition, format and buyer or seller location

• View aggregate metrics for Average Price, Shipping Policy Coverage, Shipping Cost, Total Items Sold and Total Sales 

• View detailed listing-level insights for all matching listings

• View real-time information that provides insights up to the most recent hour for all eBay transactional data.

Following closely from the successful launch at eBay Open, Terapeak spent the remainder of Q3 2019 developing features that brought the new version closer to parity with the legacy application. These included:

• Custom date picker that allows sellers to define and research any date range over the past 365 days

• Aggregate metrics for the Number of Sellers 

• Design updates that allow for easier search input and more intuitive insights

For the holiday release in 2019, Terapeak capabilities in SellerHub gained critical features such as type-ahead keywords/category suggestions and the category selector breadcrumb. Looking forward to 2020, the team is feverishly working towards delivering critical trends and metrics information to further drive feature parity and ensure a seamless transition to the SellerHub application for sellers.

 

Additionally, the team is working on further data enhancements that provide unsold listing data for the complete 365-day date range. Not only will this provide the sell-through rate demonstrating the percentage of listings sold, but will also prove a major improvement over the legacy application by providing this sell-through rate for any date range, instead of only the last 90 days. The team is also working to add more data visualizations similar to the ones found in the legacy application’s Trends tab, while providing additional visuals and insights where possible. 

In 2020, the team will also launch significant improvements to the way Terapeak search works in the Seller Hub, providing sellers with the ability to search using item specifics such as color, size, style, material and other important details that may not be included in the listing title. 

Technical Journey of bringing Terapeak to Seller Hub

The architecture of the legacy Terapeak application was built on two core infrastructures. First, a robust Elasticsearch cluster for transforming data to a more retrieval-optimized structure enabled fuzzy matching and quick retrieval. Second, the application layer for Terapeak was based on a cloud native architecture with a React based frontend application and a JVM-based backend service to communicate with the data layer.

Legacy Terapeak Data Platform

Above is a simple overview of the data platform architecture utilized by the legacy Terapeak application. The ingestion pipeline consumes the data from eBay and populates the Elasticsearch clusters. The data feed processor is responsible for validation and conversion of the ingested data for the optimized document format with PostgreSQL being utilized for storing and retrieving metadata such as category trees, exchange rates, etc. The API nodes represent the application backend services utilized for reading the data.

Modern Data Migration/Ingestion Architecture 

eBay hosts a myriad of application types managed using Kubernetes. eBay’s internal platform team specializes in managing Elasticsearch workloads, ensuring high availability and scalability of these clusters. With the advantage of a fully managed infrastructure and deeper integration with the eBay ecosystem, the data platform evolved to become faster and nimbler to handle changes better than before. 

With the platform team handling all the operations aspects of the cluster management, the engineering teams could focus more on application building with deeper insights and generating complex models based on near real-time data inputs. The modern architecture after the migration of Terapeak capabilities on to eBay tech stack consists of a tighter integration for data ingestion directly from the item-based data sources detailing the transaction information and direct access to the data warehouse for gathering category and other related information. eBay leverages the Openstack Swift storage system for quick loading of data files and transformation using Apache Spark for historical data. Apache Kafka is utilized for message resiliency to record real-time updates. All this means sellers get analytics data that is minutes old vs days old as was the case on the external Terapeak.com site.

With this new and improved architecture and its closer proximity to the data source, Terapeak is able to provide same day transactional data to the user generating near real-time insights. With increased storage and processing capabilities, the new system has the ability to analyze data over a full year rather than just 3 months provided by the legacy system. With nearly 2 Billion documents, the new system is one of the biggest ES clusters within eBay serviced by around 130 nodes per cluster housing around 3.8 TB of RAM in just one data center. The near real time pipeline processes up to 6 million events in a day updating the cluster with more up to date information  to get these valuable insights into the seller's hands.

Legacy Application Architecture

The legacy Terapeak application stack was constructed as an evolving application stack with various technologies based on their relevance at the time of development.  It is loosely constructed as a multi-tier, service-based architecture with the critical analytical data being stored in Elasticsearch clusters with a PostgreSQL-based database for all metadata and configuration. While it met the needs of the past, it was showing its age internally in terms of flexibility, opportunities to scale,  and total overall cost of ownership.

Modern Application Architecture

With the integration into core eBay architecture, some of the key changes were the transformation from a simple service-based architecture to a microservice environment. This transformation was crucial to enable a multi-device strategy and provide future integration capabilities for multiple form factors. The microservice architecture we used was based on the Experience Services framework, which eBay has started adoption site wide to accelerate multi-screen deployments. Evolving into this Backend for Frontend (BFF) design pattern allows us to develop self-contained modular code that can be easily made available in various parts of the user experience with enriched feature sets at the same time using shared modules.

Monolith vs. Microservice

The original backend service was built as a sheer monolith where the service was responsible for multiple actions to build the data response. This inhibited the cycle time for feature releases even for small changes involving schema updates to the Elasticsearch cluster or adding new data elements from the generated data responses.  

eBay Seller Hub was established with a few architectural guidelines that helped resolve this tight coupling and also helped disintegrate the monolith into self-contained microservices. This also provided a good mechanism to decouple the user experience from the data layer, allowing us to spin off public APIs for these data points in the future as part of our API strategy. The eBay platform services enable end-to-end detailed tracking and robust authentication capabilities. The Seller Hub infrastructure services provide the ability to generate the skeleton modules personalized for users based on their preferences, and subscription services enable quick lookup to ascertain the sellers’ subscriptions to enable targeted experiences.

This architecture enables Terapeak to be bootstrapped into Seller Hub quickly and also without the need to replicate multiple core functionalities. The Experience services are built using Scala, which eases asynchronous programming and makes the orchestration of various services a breeze to implement. The Central Application Logging (CAL) system provided by the eBay infrastructure ensures complete logging capabilities while the global config management system helps migrate all application configurations as a simple one-time exercise.

Asynchronous Progressive Web Application

Seller Hub inherently processes heavy data loads to provide meaningful statistical guidance to the users. To ensure that users do not wait for a long period of time, one of the key design aspects  of Seller Hub architecture is to progressively render the application with data whenever they are made available. Terapeak fits hand-in-hand with this strategy and uses MarkoJS progressive rendering and client-side rendering for asynchronous loading on the browser, while utilizing Scala/Java futures to provide the data from the services side. This helps provide a seamless user experience with the search results and aggregated data being generated and rendered progressively so the user can visualize and interact with them right from the first response byte.

Seller Hub on the other hand has adopted a more modern approach to use server-sent-events (SSE) to push updates from the server as soon as the data is available. This has many advantages over regular long polling as the data is immediately available, the data size is typically smaller (as SSE has no headers,) and using SSE provides users the ability to inspect the data immediately. 

Note: The SSE architecture is used for the front-end server to communicate with the experience service and they are restricted to communicate only with the front-end server to prevent any unwarranted access.

A simple depiction of how SSE is used to build the new UI rendering for the Terapeak Product research page on Seller Hub is provided below.

Learn more about the differences between async responses and SSE in this Stack Overflow post.

Migrating from React to MarkoJS

Another key change in this migration is the adoption of MarkoJS for the frontend application stack. This decision was based on factors including:

1. Seller Hub follows the design standard defined by eBay skin.

2. UI core components were readily available and work hand-in-hand with eBay Skin.

3. These features allow us to build modular experiences that can be easily plugged in various flows.

4. These features include constant support and updates.

MarkoJS also allows us to iterate faster and integrate seamlessly with the existing components, helping us deliver features faster and with more completeness with regards to accessibility, speed, and compatibility across devices and platforms.

Overhead with a caveat

There were few components such as the datepicker that were not readily available as part of the eBay core component system that needed to be built from the ground up using MarkoJS.  This provided us the opportunity to build reusable feature-rich components for the whole eBay ecosystem and contribute back to the larger developer community. It also helped us gain crucial insights into the internal workings of MarkoJS and a deeper appreciation of its core architecture.

Pros and Cons

• Clean syntax

• Fast rendering

• HTML first language (as opposed to JavaScript first JSX)

• More established frameworks already have date-picker component

• Smaller community

For more details see https://markojs.com/docs/marko-vs-react/.

Exciting Future

The  migration from Terapeak’s legacy implementation to a new modern approach on eBay’s technology stack  positions us for a faster and more feature-rich set of capabilities than Terapeak had ever been able to offer before. This includes the ability to offer more detailed and near real-time insights with higher levels of fidelity along with improved quality,  reliability, and more sustainable operational support. With the entire Terapeak data platform entirely rebuilt and the foundational features of Terapeak now hosted on a modern, more nimble, optimized tech stack atop eBay's technology platform, it has positioned the team to deliver not only the existing value sellers had come to love and expect from Terapeak, but also an ability to quickly deliver an onslaught of new capabilities for sellers that eBay will be rolling out heavily over the first half of 2020. Stay tuned for more updates on Terapeak in Seller Hub in future editions of this blog.