Lessons from TiDB’s No.1 Bug Hunters Who’ve Found 400+ Bugs in Popular DBMSs

Summary

Finding logic bugs is an important part of building a reliable Database Management System (DBMS). But sometimes the most obvious approach doesn’t work. You can’t just query several databases and compare the results. You need a more sophisticated bug-hunting approach.

Lessons from TiDB’s #1 bug hunters

Transcript

Hello, everyone. My name is Manuel Rigger. I am a postdoctoral fellow at ETH Zurich. I am very grateful to PingCAP for inviting me to introduce myself and my work. I am from Austria, 29 years old. I like to go hiking, go travelling, and play table tennis.

What are logic bugs?

What are logic bugs? Well, I want to explain this on a concrete example. Namely, we have a client application, which sends a SQL query to the DBMS, which is TiDB in our case. Then, the DBMS is supposed to go through all the relevant records. So in this example here, we have three records, two for which the condition — this predicate here — evaluates to TRUE, and one for which it evaluates to FALSE. Consequently, we would expect that the result set that is returned comprises two rows; namely, those for which the condition evaluates to TRUE. However, in some cases it can happen that by sending the query to the DBMS, we trigger a bug, and in such a case it might happen that the result set that is returned is incorrect, such as in this case here, where only a single row rather than two are fetched, and we refer to these kinds of bugs as logic bugs. So those bugs, that result in the computation of an incorrect result set.

Bug hunting methods

How could we tackle this? Well, the most obvious approach would be to use differential testing. Differential testing in this context basically means that we have a query generator, which we use to generate a query that we send to multiple DBMSs. For example, not only to TiDB, but also MariaDB and MySQL, which are the closest, or which are DBMSs with the closest SQL dialect to TiDB. Each of these DBMSs then fetches a result set, and we can compare all the three result sets in this example, and check if they are all the same. If not, we have likely found a bug in one of these systems. Unfortunately, differential testing is not applicable for DBMSs.

The NoREC method: not intuitive, but effective

But let’s focus on NoREC now, which is a simple, but non-obvious approach that I can also explain in a couple of minutes. And it allowed us to find over 150 bugs in widely-used DBMSs.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
PingCAP

PingCAP

PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap