Developer Tools in Engineering

Merchant Integration Platform (MIP)

By: Dilip Varadarajan

Merchant Integration Platform (MIP)

In the fall of 2010, we began discussing ways to reduce eBay’s on-boarding cost and time for merchants. Merchants were spending too much time for the following reasons:

  1. No support for account-level profiles such as Return Policy, Shipping, Payment, and so on
  2. No support for item description HTML templates
  3. No concept of items being out-of-stock
  4. No support for products or product relationships (i.e. product ‘A’ is a variation of product ‘B’)

In addition, merchants wanting to list their inventory on eBay have to understand eBay-centric terminology such as “Unsold”, “Relist”, and “GTC” (Good Til Canceled).  The Merchant Integration Platform (MIP) has been designed to address these pain points and more.

Where We Started

We began the MIP project by thinking through what’s best for our larger sellers. Two key points stood out:

  1. Merchants should be able to use all of their existing feeds. We should not require merchants to change their feeds to fit with eBay
  2. Integrators should be able to use a simple scripting language to translate merchants’ feeds to the eBay format

So, we built a layer external to eBay and used eBay’s existing services to communicate with the site. Below is a high level diagram that shows the role of MIP from a merchant perspective:

MIP High Level Diagram

Before delving into the details, it’s important to understand some terminology:

  1. Product – an intangible entity that describes what a merchant has for sale. An earlier blog post by David Goldberg talks more about the distinction between products and eBay items.
  2. Variation – a type of relationship between products. For example, a merchant who sells shirts may offer the same shirt in two colors, blue and red. Different merchants model variations in different ways. Two most commonly observed modeling methods are (i) a parent record that defines the product and two child records that indicate the variation in color and (ii) two product records with a relationship value indicating that these two products are variations of one another
  3. Item – a tangible instance of a product that can be bought or sold
  4. Listing – an item that’s for sale on eBay
  5. SKU – Stock Keeping Unit. This can be thought of as the primary key in the merchant’s product table
  6. Feed – a CSV or XML file. Any file that is transferred from the merchant to eBay is an ‘Inbound’ feed. Any file that gets generated on eBay and sent back to the merchant is an ‘Outbound’ feed
  7. Product feed – An inbound feed that contains product information such as SKU, title, description, UPC, and attributes
  8. Inventory feed – An inbound feed that contains updates to price or quantity for a set of SKUs
  9. Order feed – An outbound feed of orders generated on eBay.  Contains information about each order such as SKU, buyer’s shipping address, and so on
  10. Shipment feed – An inbound feed that contains shipment information such as tracking number, carrier, and so on for orders generated on eBay

In the future, we plan on supporting other feed types such as a Refund feed and a Payment feed.

Building a Data Model

Given our goal of allowing our merchants to use their view of the product and item universe, we needed to augment eBay to support the common concepts used by our sellers. We need to do this because eBay is largely listing based, it doesn’t have a strong notion yet of products and items being instances of products.

For example, consider the following scenario:

  1. A merchant has a product feed
  2. A merchant has an inventory feed for the products
  3. We need to combine the information from the product and inventory feeds to create a listing on eBay
  4. eBay may reject the listing (for example, the title may be longer than 55 characters). In this case, the inventory feed has to be “rolled back” for the SKU that did not get listed on eBay

This isn’t easily supported within the eBay data model, so we created a database schema to store merchant feeds.

    This particular scenario is solved by creating draft tables in a MIP database. Draft tables contain the data from the merchant feeds. This data gets promoted to the main tables only when eBay has successfully accepted the changes.

    We also have challenges with products with variations. For example, a merchant may be selling different colored shirts in different sizes made from different materials. In this case, color, size, and material will identify the variation that they’re dealing with. Another merchant may be selling wedding rings. In this case, ring size, and metal may identify the variation. To add to the complexity, eBay supports variations in some but not all categories. Given these constraints, we were left with the following choices:

    1. Model every variation as a standalone product. This is closer to how most merchants model variations
    2. Model every variation as an instance of a product with distinguishing attributes. This is closer to how eBay treats variations

    We decided to go with the latter: decoupling products from product instances allowed us to avoid duplication of data and, more importantly, allowed us to integrate with catalog vendors. Data from catalog vendors is used by eBay sellers.

    Supporting Integrators with Scripts

    As we discussed, MIP supports integrators who map merchants’ feeds to eBay’s schema. Our goal in adding this support was to chose a simple, flexible programming language and model that’s easy to learn, widely understood, and easy for our merchants to hire for.

    We evaluated Smooks, an open source framework that did a nice job of what we needed to support integrators who map merchants’ feeds to eBay’s schema. We did not adopt it because of its lack of JavaScript support and customer support. We also looked at Extract-Transform-Load (ETL) solutions such as Talend. These are too complex to use by our integrators. With these options ruled out, we decided to build our own wrapper around Mozilla’s Rhino. We chose Rhino because it’s popular and JavaScript based.

    The script writer uploads their scripts as a zip file through an admin tool. The script writer has a base implementation that they will subclass for merchant specific changes. Below is an example snippet:

    In the  base file (mapping-base.js), we have the following boiler plate code:

    function transform(){
	//...
	PRODUCT.Price = getPrice();
	//...
}

    In the merchant’s extension (mapping-custom.js), the script writer can add the following code:

    function getPrice() {
	if( INVENTORY_INRECORD.Inventory.length() == 1 ) {
		if( INVENTORY_INRECORD.Inventory.Sale.length() == 1 ) {
			return trim( INVENTORY_INRECORD.Inventory.Sale.SalePrice.toString() );
		}
		return trim( INVENTORY_INRECORD.Inventory.StandardPrice.toString() );
	}
	return null;
}

    In the above snippet, PRODUCT refers to the MIP product object. INVENTORY_RECORD refers to the record in the inventory feed. The inventory feed could either be XML or CSV. The script writer only sees them as JavaScript objects.

    The script writer also has access to a mapping service that is a collection of utility JavaScript functions to open an Excel file and parse its contents. This is mainly used for data mapping. For example, let’s suppose that eBay has defined a color called “Metallic” whereas the merchant has the colors “Gold” and “Silver”. By using the mapping service, the script writer can easily carry out the translation of the attribute values.

    The script writer also has access to a logger to write messages into a log file that is accessible from the admin tool.

    Workflow

    Overall, the MIP platform has five separate modules with well defined inputs and outputs.

    1. Transportation module: This module is responsible for moving files from the client desktop to our cloud storage and vice versa. For the first version, we achieve this by using WebDAV. This allows merchants to map their local drive to a web drive and transfer files by simply dragging and dropping them to this mapped drive. We are planning to support secure ftp in future versions
    2. Request translation module: This module is responsible for retrieving feeds from our cloud storage, running them through the scripting engine and persisting the results in our draft tables. Any errors while executing the script are saved in an errors table
    3. Request builder module: This module has all the business logic (examples: when to end the listings on eBay, when to relist, when to send in variations). It uses eBay’s Large Merchant Services (LMS) to send requests to eBay. eBay’s LMS expects an xml file with a well defined schema. When a file is uploaded to LMS, it returns a reference id which is saved to one of our tables
    4. Response builder module: This module polls eBay’s LMS at a configurable interval to determine if the result file is available. If it is available, it streams the file through a SAX parser and for successful SKUs, promotes the draft table entries to the main tables. For unsuccessful SKUs, the module writes the LMS error messages to an error table. Using the draft and error tables, it builds the response file that will be the input for the next step
    5. Response translation module: This module is responsible for getting the feeds from the Response builder module and performing the necessary translation for the outbound feeds

    These five modules interact using eBay’s internal Business Event Stream mechanism. Every module does its work and when done produces an event for the next module to consume.

    We have also built an Admin application that allows merchants to do tasks like setting up an account for MIP, setting up account level profiles, looking at the history of feeds, uploading JavaScript files etc.

    Below is a diagram that shows this workflow and the interaction between the modules:

    MIP Flow Diagram

    What MIP Supports

    Profiles

    Profiles are the store policies for a merchant. For example, a merchant may have a policy of accepting returns only within seven days. Another merchant may have a shipping policy of not offering expedited processing. Almost all merchants have only a handful of shipping profiles, return policy profiles, and payment profiles. In MIP, these profiles are configured as JSON files.

    When a profile changes, the changes must be propagated to all relevant eBay listings. Let’s say a merchant changes their return policy profile from 10 days to 7 days. Now all the listings on eBay that were associated with this profile should be revised to reflect this change. Since we keep the association between the merchant SKU, item id on eBay, and the profile id, we can generate a LMS file and upload it to eBay to reflect these changes.

    Description templates

    To be successful on eBay, it is important to have a good description in the listing. Most of our merchants create only a few HTML templates with tokens that need to be substituted at run-time. For example, different products in the same category may use the same HTML template with tokens such as ${PRODUCT.Title} or ${PRODUCT.CustomFields.Brand}.

    If the description template or any of the token values in the template change, we need to change the listing descriptions on eBay. To accomplish this, we store two values along with the description template. One is a checksum of the description template and the other is a list of token names from the template when the template was first uploaded.

    1. Description template changes: we compare the checksum values. If they are different, we regenerate the description and send it to eBay
    2. Token values used in the template changes: for example, the title of the product may change as part of a product feed. We check to see if any of the modified fields is in the list of token names. If there is even one, we regenerate the description and send it to eBay

    Out of stock handling

    eBay does not support the concept of Out of stock, therefore MIP handles this on behalf of the merchants.

    Products and relationships

    MIP supports only one type of relationship in our first release: variations.

    eBay supports variations in some categories (for example, shirts) but not in others (for example, computer hard drives). MIP hides this complexity:

    • One listing with multiple variations is generated for categories where eBay supports variations
    • Multiple listings, one for each variation, is generated for categories where eBay does not support variations

    The response file contains the SKU, and a code to indicate whether the listing was created successfully.

    In future versions, we intend to extend support to other forms of relationships such as Fitment (for example, the automobiles that are compatible with a particular tire) and Merchandising (the set of products to be used for cross promotion when a buyer buys a specific product)

    Summary

    MIP is available as a Beta release for a few merchants. The Beta phase gives us a chance to iterate on the platform, and our integrating partners an opportunity to become familiar with the system and give us feedback. We plan to deliver the public version 1.0 by late summer.

    We recently successfully integrated our first merchant, Reeds Jewelers. You can find their store at: http://stores.ebay.com/Reeds-Jewelers.

    Dilip Varadarajan
    Senior Manager – Product Development

    Tag: Developer Tools