Porting TiDB to ARM64 for Greater Flexibility

Siddon Tang

The Intel x86–64 is the most widely used processor in server architectures. However, ARM, previously known as Advanced RISC Machine, is attracting more and more attention, and many companies have migrated their products from x86–64 to ARM.

At PingCAP, we also want to run TiDB on ARM. This article describes how we compiled and benchmarked TiDB on the Amazon Web Services (AWS) ARM64 platform.

Compiling

The TiDB platform is comprised of three main components: the TiDB server, the Placement Driver (PD) server, and the TiKV server. To compile and build these components, we did the following:

  • Installed supporting software that is compatible with ARM64.
  • TiDB and PD are written in the GO programming language, so we installed go1.12.6.linux-arm64. TiKV is written in the Rust programming language, so we used rustup to install Rust.
  • We installed ARM64-compatible versions of the Grafana and Prometheus monitoring software.
  • Before we compiled, TiKV, we used yum to install the following dependencies:
  • During the TiDB and PD builds, we discovered compatibility issues with a TiDB vendor. We upgraded the vendor software and solved the problem.
  • During the TiKV build, we addressed RocksDB 5450 and Titan 23 compilation issues. We resolved these issues by passing an ARM64 or AARCH64 marco to each component.

By the way, there is a script which can help you build TiDB on ARM64. Enjoy it.

Benchmarking the TiDB ARM64 and x86–64 versions

After we built our TiDB version for ARM64, we benchmarked it to compare its performance with the same configuration on x86–64.

We used Yahoo! Cloud Serving Benchmark (YCSB) for our benchmarking software.

We used two machines for our test setup: one ARM64 machine with 16 cores and 32 GB RAM, and one x86–64 machine with 16 cores and 32 GB RAM. On each machine, we started one PD instance and one TiKV instance with default configurations.

Note: We only use the go-ycsb Raw mode to benchmark TiKV, not TiDB.

We used 100 threads, loaded 1 million rows of data, and executed 1 million operations. The benchmark results were as follows:

As you can see, the performance on ARM64 is much worse than that on x86–64. So we need to do some optimizations. From Optimizing Network Intensive Workloads on Amazon EC2 A1 Instances, we know we can do something to handle the network more efficiently. This article provides a script that makes CPU 0 and 8 handle network interruptions, and evenly distribute received packets to all CPUs. But unfortunately, after we applied this script and did the benchmark again, we ran mpstat and found that CPU 0 and 8 were full.

So we did a little change, to distribute received packets to all the CPUs except 0 and 8:

The result was better on ARM64, but CPU 0 and 8 were still the bottleneck. Specifically, the bottleneck was caused by the network interrupt handling. We’ll try to optimize it later.

You can see that CPUs run more evenly on x86–64 than on ARM64.

Conclusion

This article showed how we built TiDB and performed some benchmark optimizations on ARM. Because of its low power and low cost, I believe ARM can be used more and more widely in servers. Maybe this is the future in China, too.

Originally published at www.pingcap.com on Aug 16, 2019

Written by

PingCAP is the team behind TiDB, an open source MySQL compatible NewSQL HTAP database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store