Why We Switched from bcc-tools to libbpf-tools for BPF Performance Analysis

  • libbpf + BPF CO-RE (Compile Once — Run Everywhere) is a different development and deployment mode than the BCC framework. It greatly reduces storage space and runtime overhead, which enables BPF to support more hardware environments, and it optimizes programmers’ development experience.

Why libbpf + BPF CO-RE is better than BCC

BCC vs. libbpf + BPF CO-RE

BCC embeds LLVM or Clang to rewrite, compile, and load BPF programs. Although it does its best to simplify BPF developers’ work, it has these drawbacks:

  • You must remember naming conventions and automatically generated tracepoint structs.
  • Because the libbcc library contains a huge LLVM or Clang library, when you use it, you might encounter some issues:
    When a tool starts, it takes many CPU and memory resources to compile the BPF program. If it runs on a server that lacks system resources, it might trigger a problem.
    BCC depends on kernel header packages, which you must install on each target host. If you need unexported content in the kernel, you must manually copy and paste the type definition into the BPF code.
    Because BPF programs are compiled during runtime, many simple compilation errors can only be detected at runtime. This affects your development experience.
  • Libbpf acts like a BPF program loader and relocates, loads, and checks BPF programs. BPF developers only need to focus on the BPF programs’ correctness and performance.
  • This approach minimizes overhead and removes huge dependencies, which makes the overall development process smoother.

Performance comparison

Performance optimization master Brendan Gregg used libbpf + BPF CO-RE to convert a BCC tool and compared their performance data. He said: “As my colleague Jason pointed out, the memory footprint of opensnoop as CO-RE is much lower than opensnoop.py. 9 Mbytes for CO-RE vs 80 Mbytes for Python.

How we’re using libbpf-tools at PingCAP

At PingCAP, we’ve been following BPF and its community development for a long time. In the past, every time we added a new machine, we had to install a set of BCC dependencies on it, which was troublesome. After Andrii Nakryiko (the libbpf + BPF CO-RE project’s leader) added the first libbpf-tools to the BCC project, we did our research and switched from bcc-tools to libbpf-tools. Fortunately, during the switch, we got guidance from him, Brendan, and Yonghong Song (the BTF project’s leader). We’ve converted 18 BCC or bpftrace tools to libbpf + BPF CO-RE, and we’re using them in our company.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store


PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap