Why We Switched from bcc-tools to libbpf-tools for BPF Performance Analysis

Wenbo Zhang

Distributed clusters might encounter performance problems or unpredictable failures, especially when they are running in the cloud. Of all the kinds of failures, kernel failures may be the most difficult to analyze and simulate.

A practical solution is Berkeley Packet Filter (BPF), a highly flexible, efficient virtual machine that runs in the Linux kernel. It allows bytecode to be safely executed in various hooks, which exist in a variety of Linux kernel subsystems. BPF is mainly used for networking, tracing, and security.

Based on BPF, there are two development modes:

  • The BPF Compiler Collection (BCC) toolkit offers many useful resources and examples to construct effective kernel tracing and manipulation programs. However, it has disadvantages.

In this post, I’ll describe why libbpf-tools, a collection of applications based on the libbpf + BPF CO-RE mode, is a better solution than bcc-tools and how we’re using libbpf-tools at PingCAP.

Why libbpf + BPF CO-RE is better than BCC

BCC vs. libbpf + BPF CO-RE

BCC embeds LLVM or Clang to rewrite, compile, and load BPF programs. Although it does its best to simplify BPF developers’ work, it has these drawbacks:

  • It uses the Clang front-end to modify user-written BPF programs. When a problem occurs, it’s difficult to find the problem and figure out a solution.

By contrast, BPF CO-RE has these advantages:

  • When you implement BPF CO-RE, you can directly use the libbpf library provided by kernel developers to develop BPF programs. The development method is the same as writing ordinary C user-mode programs: one compilation generates a small binary file.

For more details, see Why libbpf and BPF CO-RE?.

Performance comparison

Performance optimization master Brendan Gregg used libbpf + BPF CO-RE to convert a BCC tool and compared their performance data. He said: “As my colleague Jason pointed out, the memory footprint of opensnoop as CO-RE is much lower than opensnoop.py. 9 Mbytes for CO-RE vs 80 Mbytes for Python.

According to his research, compared with BCC at runtime, libbpf + BPF CO-RE reduced memory overhead by nearly nine times, which greatly benefits servers with scarce physical memory.

How we’re using libbpf-tools at PingCAP

At PingCAP, we’ve been following BPF and its community development for a long time. In the past, every time we added a new machine, we had to install a set of BCC dependencies on it, which was troublesome. After Andrii Nakryiko (the libbpf + BPF CO-RE project’s leader) added the first libbpf-tools to the BCC project, we did our research and switched from bcc-tools to libbpf-tools. Fortunately, during the switch, we got guidance from him, Brendan, and Yonghong Song (the BTF project’s leader). We’ve converted 18 BCC or bpftrace tools to libbpf + BPF CO-RE, and we’re using them in our company.

For example, when we analyzed the I/O performance of a specific workload, we used multiple performance analysis tools at the block layer:

The analysis results helped us optimize I/O performance. We’re also exploring whether the scheduler-related libbpf-tools are helpful for tuning the TiDB database.

These tools are universal: feel free to give them a try. In the future, we’ll implement more tools based on libbpf-tools. If you’d like to learn more about our experience with these tools, you can join the TiDB community on Slack.

Originally published at www.pingcap.com on Dec 3, 2020



PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store

PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap