Beats @ eBay - Collectbeat - A Journey where Company and Community Come Together

In the beginning…

In early 2016, the Monitoring Special Interest Group (SIG) ventured into solving the problem of logs and metrics shipping from Tess.io (eBay’s Kubernetes ecosystem). Kubernetes, as one may be aware of, is a container management system. Users have the flexibility to drop in Docker containers and let Kubernetes manage them. These Kubernetes clusters, or Tess clusters inside of eBay, are multi-tenanted. We have many customers running their workloads at any given time. The multi-tenanted aspect of our Tess clusters brings about some interesting problems. Some of them are:

All logs and metrics need to be logically grouped (namespaced) based on customer/application type.
Workloads are ephemeral and tend to move across Nodes.
Additional metadata is required to search/query logs and metrics from Pods.
Metrics and logs need to be exposed/collected in a cloud native fashion.
Ability to self onboard logs and metrics to the centralized system.

Our first offering…

With these problems in mind, we wanted to offer a solution that allowed users to drop their Pods into Kubernetes and obtain their logs and metrics in the simplest possible way. If a user’s Pod logs to stdout/stderr, we should be able to collect the logs, append all the Pod metadata to each log line. If a user exposes metrics in a well-known HTTP endpoint, we should be able to collect it.

Knowing these challenges and the goal in mind, we embarked on the problem, attempting to solve one issue at a time.

Let us take logs as the first example and see how we attempted to solve it. Docker allows users to write to logs stdout/stderr, which is taken and placed in a well-known file path of:

/var/lib/docker/containers/<id>/<id>-log.json

If we were to listen to /var/lib/docker/containers/*/*-log.json, we would be able to collect all the logs that are being generated by all Pods in a given Node. Configuring Filebeat to listen to that path is simple and is exactly what we did. When we collected all these logs, we needed a way for users to be able to query based on Pod name, Namespace name, etc. Kubelet also started exposing these files in a symlink of:

/var/log/containers/pod_namespace_container_<id>.log

It would be easy to write a processor on Filebeat to split the source value in the payload and extract pod, namespace, and container name. But, we realized that pod labels also carry significance in querying an entire deployment’s worth of logs and that information was not present. To solve this, we wrote our own custom Beat called Annotatebeat which can:

Listen on lumberjack protocol for Beat events
Look for the source field and extract the container ID
Look up the Kube API server for metadata of all pods in a given node
Use the container ID to append all the remaining metadata onto the event
Send it to a prescribed destination

As long as a user writes an application that can write to stdout/stderr, Docker would pick up the log and place it in a well-known log file. Filebeat tails the logs, sends it to Annotatebeat, which annotates the log message with pod metadata and ships the logs out. At this time, the Beats community wasn’t fully invested in Kubernetes, so we built some of these features internal to eBay.

Seeing how simple it was to write logs and have them shipped, we wanted a simple experience for metrics as well. At Elastic{ON} 2016, the Elastic folks announced Metricbeat as a new offering they were coming up with. Metricbeat has the concept of “modules” where a module is a procedural mechanism by which metrics can be collected from a given application. If Metricbeat is configured to listen to localhost:3306 for module type “mysql”, the MySQL module knows that it should connect to the host:port and run a `SHOW GLOBAL STATISTICS` query to extract metrics and ship them out to the configured backend.

This concept appealed to us because it allows “drop in” installations like MySQL, Nginx, etc. to be monitored out of the box. However, we needed users, who write their own code and deploy applications into Kubernetes, to also be able to monitor their applications. We hence came up with Prometheus/Dropwizard modules for users to expose their metrics via the above formats as HTTP endpoints, so that we could collect metrics from them and ship them. However at the time of Metricbeat creation, it was designed to be tailored for specific applications like MySQL, Apache, and Nginx and not for generic frameworks like Prometheus or Dropwizard. Hence our PR was not initially accepted by the community, and we managed the module internally.

The discovery is something that is not supported by Beats out of the box. We had to come up with a mechanism that says “given a node on Kubernetes, find out all the pods that are exposing metrics and start polling for metrics.” How do we find the pods that are poll worthy? We look for following metadata found as annotations:

io.collectbeat.metrics/type - the type of metrics exposed (Metricbeat module name)
io.collectbeat.metrics/endpoints - ports to look at
io.collectbeat.metrics/namespace - namespace to write metrics into

As long as these three mandatory annotations are present, we should be able to start polling for metrics and write them into the configured backend. This discovery module uses Kubernetes’ controller mechanism to keep watching for updates within the node and start polling configured endpoints. This discovery module resided in a custom Beat that we lovingly call Collectbeat. To sum up, we used Collectbeat for collecting metrics from pods and Filebeat for collecting logs. Both sent their data to Annotatebeat, which appended pod metadata and shipped it to the configured backend. We ran this setup internally for about a year on version 1.x. Then Beats came out with 5.x.

Challenges in managing an internal fork…

When we were ready to upgrade to Beats 5.x, most of the interfaces had changed, and all of our custom code had to be upgraded to the newer interfaces. By this time, the Beats community had evolved Metricbeat to support generic collectors like Prometheus and several other changes for which we had written changes in our internal fork were available upstream. The effort to upgrade to 5.x would be substantial.

We had two options in front of us. One was to keep going down this path of managing our internal fork and invest a month every major release to pull in all the new features and make necessary changes to our internally owned features. The second option was to open source anything that was generic enough to be accepted by community. On taking stock of all the features that we had written, 90% of them were features applicable to any Kubernetes cluster. The remaining 10% was required to ship data to our custom backend. Hence, we took a decision to upstream that 90% so that we don’t have to manage it any longer.

Be one with community…

In Elastic{ON} 2016 we met with the Beats community and came to an agreement to open source as much as we can with regards to the Kubernetes use-case, since we already have expertise monitoring Kubernetes internal to eBay, in return for faster PR reviews.

The first thing that we decided to get rid of internally was Annotatebeat, which did the metadata enrichment. Today in libbeat there is a processor called add_kubernetes_metadata, which was a result of that decision. We took all the logic present in Annotatebeat and converted it into a processor with the help of Carlos Pérez-Aradros, a member of the Beats community. We also took our internal Prometheus implementation and used it as a reference to update the community-available version to cover a few missing use cases. Dropwizard, Kubernetes Metricbeat modules, were something we used internally that we also open sourced.

Eventually we got to a point where we could run both Filebeat and Metricbeat as available upstream without any necessary changes. With go1.8 out, there was also support for plugins and we offloaded all our custom code internal to eBay. It is managed independent of stock Beats.

We realized the hard way that it is impossible to keep up with the rapid pace of an open source community if we have custom code residing in our internal fork. Not having a custom fork internally has helped us to be on the most recent version of Beats all the time and has reduced the burden of pulling in new changes.

It is always easier to make progress when we work with the community on features that not only benefit us today, but may also benefit someone else tomorrow. More thoughts and ideas on the code can always make it better. A good working relationship with the Beats community has helped us not only with code management, but also with features that were required internally that ended up getting built by the community. Today, eBay contributes the most amount of code outside of Elastic itself to the Beats product. This has not only benefited the product, but also eBay as well. With the combined effort of eBay and the Elastic, Beats will have native Kubernetes support in 6.0.

A new day…

Removing all of our custom code improved our agility to think of newer use cases. We wanted to increase coverage for the number of applications from which metrics can be collected. We realized that writing Metricbeat modules for every application is an impossible task and that going after protocols is a more scalable option.

One protocols that has tremendous coverage is the plain text protocol understood by Graphite. Tools like CollectD and StatsD can write to destinations that understand the Graphite protocol. We then implemented “Graphite server” as a Metricbeat module and contributed it back to Beats. This module inside of Collectbeat’s Kubernetes discovery helped us support use cases where customers can annotate their Pods with a parsing rule, and Collectbeat would receive metrics and parse them to split the metric name and tags before ingesting them to the desired backend. Another similar protocol that we went after was vanilla HTTP, where users can send metrics as JSON payloads to Metricbeat, and it would be shipped to the desired backend.

Being able to discover metrics inside of a Kubernetes environment is a big win in itself. The benefits were quite huge, and we saw the need to do the same for logs as well to support two use-cases:

Being able to stitch stack trace-like log patterns
Being able to read logs that are not being written into stdout

Because Kubernetes clusters inside of eBay are multi-tenanted, it becomes impossible to configure a single multiline pattern on Filebeat for all Pods inside of the cluster. We applied our learnings from metrics to log collection and decided to expose annotations that users can use to define multi-line patterns based on how Filebeat expects multiline to be configured. A user can, at a container level, configure multiline via annotations, and Collectbeat ensures that the required Filebeat prospectors are spun up to stitch stack traces.

A long standing problem that we have seen in our Kubernetes clusters is that, since we heavily rely on docker’s JSON log driver, performance is always a concern. Letting Filebeat decode each log line as a JSON payload is quite expensive. Also, there are a lot of use cases where a container may expose one of its many log files via stdout, but all others are written in specific file in the container.

One such example is Apache Tomcat, where catalina.out’s logs are written into stdout, whereas access logs are not. We wanted to solve both these problems with an unconventional solution. Collectbeat was rewritten to accept log paths in the Pod’s annotations, and based on what is the underlying Docker file system, Collectbeat would spin up prospectors by appending the container’s filesystem path to the file path. This would let us tail log files present inside of the container, and helps us to not rely on JSON log file processing. We can also collect log files from different files written by a container.

Where we are today…

Collectbeat has become our defacto agent that sits on every node through DaemonSets in our Kubernetes clusters to collect logs and metrics. Collectbeat runs in both Filebeat mode and Metricbeat mode to be able to tail log files and collect metrics respectively. This is what our Node looks like:

What are the features that Collectbeat has today? We are able to:

Collect metrics from any Pod that exposes metrics that abide to all supported Metricbeat modules
Collect logs written to stdout or files inside the Docker container
Append Pod metadata on every log and metric collected
Allow Pods to push metrics through Graphite protocol and parse them uniquely
Stitch stack traces for application logs

Today we run Collectbeat on over 1000 nodes shipping more than 3TB of logs and several billion data points per day. Our end goal is to put Collectbeat on every host in eBay and be able to collect logs and metrics from any application that is being deployed.

Are we there yet? No, but we are slowly, but surely, getting there. There are still several more features that we have yet to crack, like being able to give QoS for all Pods so that all Pods are treated equally when shipping logs and metrics. We also want to be able to provide quotas and throttle workloads when applicable.

We have greatly benefited from Collectbeat, and with great excitement we are happy to announce the open sourcing of Collectbeat. Putting our code out in the open will help us get feedback from the community and improve our implementation at the same time help others who are trying to solve the same problem as we are. So, go get github.com/ebay/collectbeat and let us know your feedback.

Credits…

A big shout out to all the folks in eBay who made this a reality:

Also, a big shout out to the Elastic folks from the Beats community who have helped us along the way:

Tags: Cloud, Kubernetes, OSS