NetEase Games: Why We Chose a Distributed SQL Database Alongside MySQL to Break down Data Silos

Our pain points

In this section, I’ll introduce the MySQL architecture for our billing applications and the bottlenecks we identify. As you’ll see, database scalability and data isolation are our two major pain points.

MySQL architecture for our applications

We deploy more than 10 billing applications to provide services for different game products. A big product employs a dedicated billing application, while multiple small products share the same application. Each billing application applies the standalone MySQL architecture as shown below. (The direction of the arrows indicates the flow of the data or request.)

Standalone MySQL architecture
Standalone MySQL architecture

Bottlenecks

With our applications’ traffic and data volume growing rapidly, our standalone MySQL approach reached the following bottlenecks:

Data silos
Data silos

New database exploration

In this section, I’ll describe the ideal database for our applications, and what we did to find it.

The ideal database for our applications

Considering our billing applications were highly coupled with MySQL, to resolve our issues we needed a new storage solution with the following features:

  • Support for transactions — tasks are executed as transactions or rolled back in the case of error.
  • Support for indexes — especially for secondary indexes.
  • Horizontal scalability — the system can elastically scale while online, including performance scaling and capacity expansion.
  • Stability and reliability.
  • Backup and recovery.
  • Disaster tolerance.

Our options

To meet the requirements above, we listed several optional solutions. These solutions can be grouped into two categories: MySQL-based solutions and NewSQL-based solutions (CockroachDB and TiDB).

Testing

MySQL versus NewSQL

Results of testing MySQL clusters of different node sizes
  • Migrating to the new application architecture was complicated. We would need to modify large amounts of application code.
  • If the application is based on the MySQL database, TiDB may be a better choice; if the application is based on PostgreSQL, CRDB may be the preferred choice.
Performance comparison of reads and writes on a large table
Performance comparison of reads and writes on a large table
  • Note: The above is the testing result for TiDB 2.0.5. In July 2019, PingCAP announced that TiDB 3.0 reached general availability, achieving a major performance boost. Our recent tests showed that in most scenarios TiDB 3.0’s performance was multiple times better than TiDB 2.1. The following test results compare TiDB 3.0.3 and TiDB 2.1.15 in the areas of peak online transaction processing (OLTP) and TPC Benchmark C (TPC-C):
Peak OLTP performance comparison
TPC-C performance comparison
  • TiDB is compatible with the MySQL protocol, which means few code modifications and low migration costs.

Using TiDB for our applications

Throughout our journey with TiDB, we’ve used multiple versions for testing or production, including versions 1.0, 2.0.5, 2.0.11, 2.1.5, 2.1.15, and 3.0.3. We’ve witnessed the great improvements TiDB has made over the years. In this section, I’ll describe how we’re currently using TiDB for our billing applications.

TiDB architecture

We deploy the TiDB cluster in a highly-layered structure as shown below. NGINX is used for load balancing in the front end.

TiDB architecture for billing applications
  • TiKV server is the distributed transactional key-value storage layer where the data persists. It uses the Raft consensus protocol to ensure that data has multiple replicas and will not be lost if a node fails.
  • PD (Placement Driver) server is a metadata cluster powered by etcd that manages and schedules TiKV nodes.

TiDB use cases

Application scenarios

  • TiDB supports data platform services, including reporting, monitoring, operations, generating user profiles, and computing big data.
  • HTAP scenarios that involve both OLTP and online analytical processing (OLAP) workloads.
  • In our production environment, we also have a TiDB 3.0.3 cluster that serves two kinds of workloads: 80% of them are offline big data computing, while the other 20% are online services.

Scale

Conclusion

In this post, I discussed why we adopted a new storage solution for our billing applications and why we chose TiDB over other alternatives. Our current success with using TiDB gives us the confidence to move more of our services to TiDB in the future. This scale-out distributed SQL database enables us to gain insights from data and create value for our users.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
PingCAP

PingCAP

PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap