How We Reduced Multi-region Read Latency and Network Traffic by 50%

  • High read latency. Generally, the cross-region network round trip time (RTT) is dozens or even hundreds of milliseconds (ms).
  • Cross-region network facilities involve high hardware costs. Besides, the service capability for existing facilities is not strong enough. Therefore, the performance for distributed databases is not high. Building or leasing a private network across regions is expensive, and the bandwidth is limited. In a multi-region active-active architecture, replicating data among multiple regions can cause a bottleneck.
  • The read latency for multi-region SQL queries dropped by 50%.
  • The number of cross-region messages decreased by 50%. This means the network traffic was reduced by half.

A sample multi-region architecture

Primary and secondary data centers
Primary and secondary data centers
  1. A TiDB server in City B sends a request to a PD server in City A to obtain timestamp Oracle (TSO) information. It gets the transaction start ID, start_ts. This step takes one round-trip time (RTT).
  2. As shown in the figure below, the TiDB server in City B sends multiple parallel read requests to the TiKV leader in City A for each range involved in this process. This step also takes one RTT.
Read request processing
Read request processing
  • Multi-region network bandwidth is highly occupied.
  • The read latency (two RTTs) is high.

How to reduce multi-region network traffic

Reducing multi-region network traffic with Follower Read

Read request processing with Follower Read enabled
Read request processing with Follower Read enabled

Reducing multi-region network traffic with Follower Replication

Message broadcasting
Message broadcasting
Message broadcasting with a follower delegate
Message broadcasting with a follower delegate
Messages dropped by half
Messages dropped by half

How to reduce multi-region read latency

Before and after optimizing Follower Read
Before and after optimizing Follower Read
Reducing the read latency by half
Reducing the read latency by half
Benchmark in a simulated high-latency network environment
Benchmark in a simulated high-latency network environment

Evaluation for this project

2nd place winners at TiDB Hackathon 2019

Further thoughts

  • Reducing the latency for interactive read transactions to one RTT
  • Implementing cross-DC scalability

--

--

--

PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Greatest Hosting for GSA SER

Apache Spark : How to solve Reverse Mapping Problem in Spark.

From Software Engineering to Venture Capital

✅ Recently our smartcontracts were audited by TechRate and no critical flaws were found.

How to check whether a polyhedron is inside another one?

6 of the Best Niche Platforms to Learn SQL and Python

How to Create an EC2 Instance With an Apache Webserver Through the AWS CLI

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
PingCAP

PingCAP

PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap

More from Medium

Distributed Rate-Limiting Algorithms

Distributed Systems- Modernization

Replication & Partitioning in Distributed Systems

Advantage of layline.io