A Peek into the Future of Database: A Unified Infrastructure to Adapt Intelligently

It’s all around data.

If you think about it, everything we do nowadays is around data. Business owners, big or small, need data to know whether their decisions are sound and adaptive to the challenging markets; software engineers like me, regardless of roles or titles, build and maintain systems that live and breathe data.

Is it possible, a single and real-time source of truth?

When we started PingCAP five years ago, it was impossible. Because as you can see, at the heart of it, is a database platform that could cover all the use cases and different workloads. Before we started, no database solutions on the market could fulfill that. How about Hadoop for big data analysis? How about NoSQL database solutions for scalability? Or even shard your relational database using middleware? They all have one common problem: tradeoff.

Tradeoff for database solutions
The future of data architecture

Unification, adaptiveness, and intelligence

When I talk about unification, I am talking about a unified data platform that overcomes data silos and eliminates frictions between software infrastructure and business value. I would like to use two examples, one is from the system architecture and software stack perspective, and the other is the new design paradigm in the database field in recent years.

The traditional architecture
The HTAP system
  1. Because the OLTP engine and the OLAP engine are in the same system, many details, such as transaction information, won’t be lost during the synchronization process. This means that the internal analysis system can do things that traditional separated OLAP systems can’t.
  2. For the application layer, one less system means a more unified experience and smaller learning and transforming cost.

Adaptiveness: adaptive scheduling would be the core capability of the future database

Ever since the last decade, the biggest change in IT technology has been cloud computing, and this revolution is still in progress. What are the core capabilities of the cloud? One of the most important is that it’s elastic and adaptive. Allocating computing resources has become increasingly easy and on-demand.

  • Can the database perceive the business characteristics and determine the data distribution according to the data access pattern? For example, if the data has obvious geographical features (users in China are likely to visit in China and users in the United States are in the United States), the database automatically places the data in different data centers based on how it is accessed.
  • Can the database perceive the type of query and access frequency to automatically determine the storage medium for different types of data? For example, cold data is automatically transferred to cheaper storage such as S3, hot data is placed on high-end flash memory, and the exchange of cold and hot data is completely transparent to the applications.

Would the next stage be intelligence?

What’s next? I don’t know. But I believe that just as we don’t understand the universe and the ocean, our understanding of data is at best superficial, not to mention they’re a lot of data we haven’t yet recorded. There must be a great mystery hidden in this massive amount of data, but what kind of insights we can obtain from it, or how we can use it to better our lives, I don’t know. I am willing to believe in a future where data can tell us all.

  • Our data processing will no longer be fragmented, and any business system can easily store and obtain data.
  • When we face the uncertainties and the unpredictables in the future, we will have the unified infrastructure to adapt intelligently.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
PingCAP

PingCAP

PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap