Chapter 9. Linux Memory Analysis

This chapter is for anyone wishing to monitor memory usage of processes on Linux, and describes some of the issues that are encountered when trying to get a measure of how much memory a process is actually consuming.

9.1. Issues

Shared memory, swapping and memory-mapped files all make it difficult to accurately assess how much memory a process is actually using. Running “top” for instance, usually shows this kind of output:

top - 13:19:53 up 12 min,  3 users,  load average: 0.14, 0.36, 0.34
Tasks: 225 total,   1 running, 224 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.1 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.1 hi,  0.0 si,  0.0 st
KiB Mem:  12297356 total,  3922792 used,  8374564 free,    79796 buffers
KiB Swap:  6152188 total,        0 used,  6152188 free,  2206980 cached

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  933 root      20   0  200336  69916  54876 S   0.8  0.6   0:19.34 Xorg
 4020 wsinno    20   0 6035584  47176  10876 S   0.3  0.4   0:00.59 java
 4144 wsinno    20   0 7457168 123956  13788 S   0.3  1.0   0:03.79 java
 

In this view, VIRT represents the size of the memory address space allocated by the process, which includes the size of any memory mapped file, the size of any shared memory segment that may be shared with other processes, as well as regular chunks of private memory allocated by the process, some of which may be swapped out. RES shows how much memory associated with the process is currently resident, regardless of what type of memory that is. SHR is the total amount of shared memory mapped into the process’ address space, and can be double counted with any other process also mapping the same shared memory.

9.2. Understanding the Memory Model

A process on Linux can allocate memory in several ways, and there are several ways of classifying the chunks of memory mapped into its address space. These can be broken down as shown in the following table:

Anonymous File Backed
Private malloc() ; mmap(PRIVATE, ANON); stack mmap(PRIVATE, FILE)
Shared mmap(SHARED, ANON) requires fork(); shmget mmap(SHARED, FILE)

Anonymous memory can be swapped out by the operating system to the swap partition. The same mechanism is used for file-backed memory, however, it is written directly to the backing file. Thus, anonymous memory can be in one of two states, either "resident" or "swapped".

Note that it is very common to memory map a file stored in a tmpfs filesystem, which behaves more like an anonymous mapping, since the file is stored in a filesystem that is stored in memory, and has it’s own paging to swap, i.e. the difference between truly anonymous memory and tmpfs backed memory is that in the latter case, it is the filesystem layer that takes care of the swapping.

An additional facet of memory usage in Linux is the extensive use of copy-on-write to implement lazy allocation. For instance, if a process is forked, all the memory of the original process is marked as copy-on-write, and the two processes effectively have the same amount of private anonymous memory, which would show up as such in most tools, but the actual amount of RAM resident for both processes is still only the original amount, which thus results in double counting of memory. An undercount can result from the same facility, in that a call to “malloc” will not result in any memory being used until the memory allocated to is actually written.

Shared memory is counted for each process that is using it, which means that summing up the memory usage for all processes leads to "double counting". There is no "owner" of shared memory, i.e. there is no way to say that a particular piece of shared memory should be accounted for by a given process.

Buffers for files are not accounted for in this document, since this transparently managed by the operating system.

9.3. Tools

top

Commonly used tool, provides many metrics including total address-space (VIRT), size of shared memory (SHR), and size of resident memory (RES).

ps

Also shows resident memory (RSS) with option v. Example: top -p PID v.

/proc/[pid]/statm

Provides info about mem usage (in pages). This is useful for automating measurement of values only available in top otherwise, namely shared memory SHR.

smem

Used to provide additional metrics that aren’t typically available - more expensive to run, since doing queries and computations across /proc filesystem. Includes facility to capture raw output of /proc for later analysis (smemcap).

htop

Provides a very wide variety of metrics related to memory, CPU, and I/O.

9.4. Measuring Different Types of Memory

The total address space of a process is indicated by VSZ.

Total private memory, PM equals VSZ - SHR where SHR is the total shared memory.

Total swapped memory whether in system swap file or in memory-mapped files Swap .

Swapped private memory SPM = PM - USS where USS is resident private memory.

Swapped shared memory SSHR = Swap - SPM.

Resident shared memory RSHR = SSHR - SHR.

In table form:

Resident Swapped Total
Private USS SPM PM
Shared RSHR SSHR SHR
Total RSS Swap VSZ

Note that we aren’t able to break down memory based on whether it is anonymous or file backed, we can just measure whether it is swapped or resident.