Our approach is that hardware platforms must be developed for eBay’s specific data applications. It is not optimal to simply choose the most high-end premium and powerful hardware for data-driven applications.
Starting in 2017, as part of eBay’s replatforming initiative, eBay customized hardware to develop the most reliable cost effective platforms for eBay’s applications.
What that means is to serve up to 10,000 queries per seconds at 10 milliseconds latency, the team had characterized the workload, apply new CPU technology to develop a system of choice for eBay search application. The rack-mounted system design fits into 1U standard 19” racks with density 2 server nodes per 1U.
Other features include:
-
System cooling and power supplies that are shared between server nodes. It is designed with N+1 fans and dual redundant AC feeds. System power at full workload utilization is within a 1800W power consumption limit.
-
Dual-socket CPU server nodes with Intel (or equivalent) Skylake 6138 processors for optimum cost/performance.
-
An optimized, efficient storage subsystem for distributing a bulk dataset with 2T SATA HDDs in software RAID 10 mode.
-
An embedded 10G SFP+ network adapter per server node with out-of-band BMC management. The system has robust BMC resilient firmware features and a hard reset path to BMC in addition to the standard one. This design resolved common BMC failure/hang issues in a DC environment.
This system is designed based on a Skylake 20C, 2GHz CPU at 128GB in 1U form factor, and it is capable of holding the eBay search index database.
Figure 1. System Block Diagram of the distributed search performance server
eBay's hyperscale high-density, lower-power, cost-efficient storage server
Because the demand for storage capacity increases as the image database grows, the new storage server is designed to meet the required latency at the lowest cost and power, and it optimizes CPU, memory, and storage ratios for system design with an acceptable failure domain. eBOD (Ethernet attached Bunch Of Disks) is an eBay initiative to adopt a new cost-effective storage infrastructure approach of using a custom design to support the independent scaling of compute and storage, an independent refresh of compute and storage, and to provide an acceptable blast radius for storage nodes.
The new motherboard is designed based on System on Chip (SoC), with embedded network and storage controllers in 1U form factor fit into standard 19” rack. The system is equipped with dual AC power feed and N+1 fans for redundancy.
Platform throughput is targeted at 3200 MB/s for system power of less than 300W. The system design can be scaled to higher capacity drives, which can drive system storage costs lower for the same performance and power.
Figure 2. System Block Diagram of low-cost, high-density image storage servers
The new ODM (Original Design Manufacturer) approach that entails developing customized eBay platforms for specific eBay applications, enables eBay to control business, technical outcomes and life cycles of its hardware infrastructure. eBay has extended its ODM platforms to cover high-performance servers and storage servers, as well as artificial intelligence/machine learning and distributed data analytic servers.