Why We Disable Linux’s THP Feature for Databases

What is THP

How Linux manages its memory

  • If the unused physical memory is less than low and more than min, when the user applies for memory, the page replacement daemon kswapd asynchronously frees memory until the available physical memory is higher than high.
  • If the asynchronous memory reclaim can’t keep up with the memory application, Linux triggers the synchronous direct reclaim. In such cases, all memory-related threads synchronously take part in freeing memory. When enough memory becomes available, the threads start to get the memory space they apply for.

When THP causes performance fluctuation

The most typical symptom: sys cpu rises

The indirect symptom: sys load rises

  1. Asynchronous memory compaction
  2. Direct memory reclaim
  3. Direct memory compaction
  4. Out of memory (OOM) collection
  1. View the direct memory reclaim operations taken per second. Execute sar -b to observe pgscand/s. If this number is greater than 0 for a consecutive period of time, take the following steps to troubleshoot the problem.
  2. Observe the memory fragmentation index. Execute cat /sys/kernel/debug/extfrag/extfrag_index to get the index. Focus on the fragmentation index of the block whose order is >= 3. If the number is close to 1,000, the fragmentation is severe; if it's close to 0, the memory is insufficient.
  3. View the memory fragmentation status. Execute cat /proc/buddyinfo and cat /proc/pagetypeinfo to show the status. (Refer to the Linux manual page for details.) Focus on the number of pages whose order is >= 3.
    Compared to buddyinfo, pagetypeinfo displays more detailed information grouped by migration types. The buddy system implements anti-fragmentation through migration types. Note that if all the Unmovable pages are grouped in order < 3, the kernel slab objects have severe fragmentation. In such cases, you need to troubleshoot the specific cause of the problem using other tools.
  4. For kernels that support the Berkeley Packet Filter (BPF), such as CentOS 7.6, you may also perform quantitative analysis on the latency using drsnoop or compactsnoop developed by PingCAP.
  5. (Optional) Trace the mm_page_alloc_extfrag event with ftrace. Due to memory fragmentation, the migration type steals physical pages from the backup migration type.

The atypical symptom: abnormal RES usage

How to deal with THP

  1. View the current THP configuration:
cat /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Join our community

--

--

--

PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Software for Teaching Geometry

Careers in Code Student Success Stories: Eva Carafa

Server side Swift for Laravel developers part 1

Getting started with GraphQL + Golang (in 5 minutes)

DevOps Represent

Pointers in Python v.s. Pointers in Golang

What is a Site Reliability Engineer or an SRE?

Site Reliability Engineering header image

TiDB Hackathon: Reducing Cross-AZ Data Transfer Costs by 89%

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
PingCAP

PingCAP

PingCAP is the team behind TiDB, an open-source MySQL compatible NewSQL database. Official website: https://pingcap.com/ GitHub: https://github.com/pingcap

More from Medium

First impressions of Redpanda

All Logs GoTo Heaven

Advantage of layline.io

My Journey from Traditional Monolithic Architecture to Distributed SQL