Analyse and explain Linux OOM output

OOMAnalyser is a web page to analyse and explain the OOM message of a Linux kernel. The analysis is performed automatically in the browser. You get a summary and a list of (almost) all parameters with their values and a short explanation. Among them are also 2 diagrams that illustrate the memory usage.

OOMAnalyser consists of a web page where the OOM message is copied into the input field. JavaScript code extracts the data and displays the details. All processing takes place in the browser. No data is transferred to any server. Therefore, confidential OOM messages can also be analyzed, and the analysis can be performed in environments without an Internet connection.

This project is written in Python and uses Transcrypt to translate Python code into JavaScript.

Step 1 - Enter your OOM message



Step 2 - Results

Go back to "Step 1 - Enter your OOM message" to run a new analysis.

Summary

The OOM killer was automatically triggered to free memory, because the system couldn't satisfy the memory request. The OOM killer calculates a score for each process and terminates the process with the highest score to satisfy the original memory request.

The OOM killer was manually triggered (e.g. with "echo f > /proc/sysrq-trigger") by a user with root privileges. There is no demand to free memory but the OOM killer started nevertheless. The OOM killer calculates a score for each process and terminates the process with the highest score.

The system has physical memory and swap space. That's total. ( out of ) physical memory and ( out of ) swap space are in use.

The system has physical memory and no swap space. ( out of ) physical memory are in use.

The process "" (PID ) requested a memory chunk of order from the "" memory zone. This is 2order pages == 2 pages == a == memory.

The request failed because the free memory would be below the memory low watermark after its completion. If this requirement was satisfied, the free memory would still be above the low memory watermark. The request failed because there is no free chunk in the current or higher order. The request failed, but the reason is unknown. This memory shortage triggers the OOM process.

The process "" (PID ) with an OOM score of has been terminated. It uses () of the resident memory.

Dynamic memory allocation is used by both the kernel and all applications. This leads to memory fragmentation and is a common behavior. The system memory is heavily fragmented, because all chunks with an order ≥ are in use. Allocation of larger contiguous memory areas will fail.
The amount of fragmentation depends on the software run on the system and the best fix would be to improve the software that causes heavy fragmentation. If that's not possible, compacting the memory (memory defragmentation) can be triggered more often.
The system memory is not heavily fragmented, because chunks with an order ≥ are freely available.

Corrective Actions

Top this list are the actions that have the greatest impact on preventing OOMs.

  1. Use an appropriate memory setup, i.e., configure your server with enough memory to run your workload or configure your application to use only the available memory.
  2. Improve software that causes memory fragmentation, or ask the software vendor to improve it.
  3. The system has a 32-bit kernel. Consider upgrading to a 64-bit kernel as the memory zone setup has been improved.
  4. Trigger memory compaction manually to provide more contiguous blocks of free memory. Write 1 to /proc/sys/vm/compact_memory.
  5. Try different memory tunables to run memory compaction more aggressively. This may postpone the OOM event until later, but of course will not prevent it.
    1. increase /proc/sys/vm/watermark_scale_factor
    2. increase /proc/sys/vm/watermark_boost_factor
    3. enable a more aggressive approaches to memory reclaim by writing 3 to /proc/sys/vm/zone_reclaim_mode
    4. decrease the value /proc/sys/vm/extfrag_threshold to start memory compaction earlier
    More aggressive memory defragmentation can cause latency spikes during its execution.

Details of analysis

The result of the analysis is displayed in three columns. The first column is used to name the property including the original OOM identifier in brackets. The extracted information is displayed in the second column. The last column contains further details and additional information.

Trigger Process
(PID ) This process requests memory and is triggering thereby the OOM situation.
Memory allocation flags
(gfp_mask)
These flags are also called GFP (get free pages) flags. They control the kernel-internal memory allocation.
Some OOM variants include the individual flags in addition to the hexadecimal representation. The flags are calculated if they are not specified.
Node mask to show on which CPU Cores this process can run
(nodemask)
Bit mask indicating the cores on which the process can run.
Requested memory: order The kernel specifies the requested number of pages as an exponent of a power of two.
Requested memory: number of pages (2 pages) a per page Requested memory in number of pages.
Requested memory: size Requested memory in kBytes.
Requested memory: zone Memory zone from which the requested storage chunk should come.
Requested memory: node First NUMA node with memory shortage watermark free < min in memory watermark information.
Assumption that this is the node where the OOM was triggered.
Adjust oom-killer score
(oom_score_adj)
This value is added to the badness score before it's used to determine the process to be killed.
Killed Process
(PID ) Process killed by Linux kernel to satisfy the memory request.
OOM Score
(score)
Programs with the highest OOM score are terminated first.
Virtual Memory
(total_vm)
Virtual memory used by this process.
Total resident anonymous memory
(rss)
All virtual process memory mapped into RAM. Normally, a process always requests more virtual memory than it uses.
TotalRSS = anon-rss + file-rss + shmem-rss
Resident anonymous memory
(anon-rss)
Resident anonymous pages
Part of the virtual process memory mapped into RAM.
Resident file mapping memory
(file-rss)
Resident file mapping pages
Files which have been mapped into RAM (with mmap(2).)
Resident shared memory
(shmem-rss)
Resident shared memory pages
This may include System V shared memory and shared anonymous memory.
Memory Usage
Graphs
RAM Summary
Swap Summary
Swap Usage
No swap space available
Swap Total Total amount of swap space available. [1]
Swap Free Amount of swap space that is currently unused. [1]
Swap Cached Memory that once was swapped out, is swapped back in but still also is in the swap file. (If memory pressure is high, these pages don't need to be swapped out again because they are already in the swap file. This saves I/O). [1]
Swap Used Amount of used swap space w/o cached swap
(SwapUsed = SwapTotal - SwapFree -SwapCache)
Memory Pages
RAM pages Total number of RAM pages
HighMem/MovableOnly Number of pages in the High Memory Area or marked movable for Contiguous Memory Allocator (CMA).
HighMem pages are also counted in the total page number.
Reserved pages Number of reserved pages
CMA reserved pages 0 Pages reserved for Contiguous Memory Allocator (CMA)
Pagetable Cache 0 Number of pages in pagetable cache
Number of pages with hardware errors 0 Pages with uncorrectable memory errors
Memory Usage Details
Active anonymous memory
(active_anon)
Recently used anonymous memory.
These memory pages are not normally swapped out.
Inactive anonymous memory
(inactive_anon)
Least recently used anonymous memory.
These memory pages can be swapped out.
Isolated anonymous memory
(isolated_anon)
Memory isolation is used to separate memory between different virtual machines.
Active Pagecache
(active_file)
Pagecache that has been used more recently and usually not reclaimed unless absolutely necessary.
Inactive Pagecache
(inactive_file)
Pagecache which has been less recently used. It can be reclaimed without huge performance impact.
Isolated Pagecache
(isolated_file)
Memory isolation is used to separate memory between different virtual machines.
Unevictable Pages
(unevictable)
Unevictable memory. It can't be swapped out because the pages are owned by ramfs or protected by mlock(3) / shmctl(SHM_LOCK). Unevictable pages are managed by kernels LRU framework.
Dirty Pages
(dirty)
Memory which is waiting to get written back to the disk. [1]
Writeback
(writeback)
Memory which is actively being written back to the disk. [1]
Unstable
(unstable)
Not yet committed to stable storage.
Slab Reclaimable
(slab_reclaimable)
Slab is an in-kernel data structures cache. Part of Slab, that might be reclaimed, such as caches. [1]
Additional details are listed in slabinfo(5) also.
Slab Unreclaimable
(slab_unreclaimable)
Part of Slab, that cannot be reclaimed on memory pressure. [1]
Mapped
(mapped)
Files which have been mapped into memory (with mmap(2)), such as libraries. [1]
Shared Memory
(shmem)
Amount of memory consumed in tmpfs(5) filesystems. [1]
Pagetables
(pagetables)
Amount of memory dedicated to the lowest level of pagetables. [1]
Bounce
(bounce)
Memory used for block device "bounce buffers". [1]
Free pages
(free)
Free pages
Free per-CPU pages
(free_pcp)
Free number of pages per CPU
Free CMA pages
(free_cma)
Pages reserved but not used by Contiguous Memory Allocator (CMA)
Total Pagecache Total number of pages in pagecache
Operating System
Kernel
Distribution Guessed from the kernel version
Platform Guessed from the kernel version
Page size Extracted from DMA zone buddyinfo Guessed
Available Memory Chunks

                
Memory Watermarks

                
Process Table
pid uid tgid total_vm rss nr_ptes swapents oom_score_adj name notes
Hardware Details

                
Kernel Call Trace

                
Entire OOM Message (click to hide)

                

Go back to "Step 1 - Enter your OOM message" to run a new analysis.

Footnotes

  1. proc(5) - process information pseudo-filesystem (Go Back)

Further Information

  1. Project Page
  2. Source Code (mirrored on GitHub)
  3. Transcrypt
  4. Linux man pages online
  5. How to read and decode OOM messages
    1. Decoding the Linux kernel's page allocation failure messages
    2. Linux Kernel OOM Log Analysis
    3. Out of Memory events and decoding their logging
  6. Kernel Source code starting points:
    1. mm/omm_kill.c at kernel.org or more comfortable mm/omm_kill.c at elixir.bootlin.com
    2. mm/page_alloc.c:zone_watermark_ok() at kernel.org or more comfortable mm/page_alloc.c:zone_watermark_ok() at elixir.bootlin.com
  7. The "Documentation for /proc/sys/vm/" contains explanations to some of the memory compaction settings.

Changelog

(click to show / hide)

Version 0.7.0 - 2024-XX-XX:

General

  1. Extend platform determination for amd64 Debian/Ubuntu kernels (Moritz Schlarb)
  2. Add more platforms to the determination list
  3. Various fixes for a Debian buster OOM block (reported by Alexander Stanzl)
  4. Show kernel upgrade hint only on 32-bit kernels
  5. Extend table of contents to also show h3 headings
  6. Adjust regex "Killed process" for kernels 3.10 and 4.5
  7. Add support for kernel 6.3 to 6.8
  8. ...

Note

See the commit history of the repository for a full list of changes.

Version 0.6.0 - 2023-08-30:

General

  1. Fix memory calculation in summary section
  2. Fix and rework calculation of GFP flags
  3. Add GFP flags for more kernel releases
  4. Display missing memory chunks (buddyinfo) again
  5. Add analysis why the memory request failed
  6. Add check for heavy memory fragmentation
  7. Summary of the analysis revised
  8. Show memory watermarks together will all details
  9. Add support for kernel 6.0 and newer
  10. Add new example ArchLinux, Kernel 6.1.1
  11. Fix handling of line breaks in Mem-Info block
  12. Add section for corrective actions (suggested by Mikko Rantalainen)
  13. Bug fixes

Note

See the commit history of the repository for a full list of changes.

Version 0.5.0 - 2022-01-12:

General

  1. Improve SVG chart colour palette
  2. Add Selenium based unit tests
  3. Fix to allow process names with spaces
  4. Rework removal of unused information
  5. Report uncaught errors to the user
  6. Add support for manually triggered OOM (suggested by Mikko Rantalainen)
  7. Add support for systems w/o swap (suggested by Mikko Rantalainen)
  8. Add support for newer kernels (suggested by Mikko Rantalainen)
  9. Add support for journalctl output (suggested by Mikko Rantalainen)
  10. Add support for newer process table format
  11. Rework memory charts to show all items in legend
  12. Bug fixes

Note

See the commit history of the repository for a full list of changes.

Version 0.4.0 - 2020-12-10:

General

  1. Add a textual summary of the analysis
  2. Fix calculation of requested memory in kBytes
  3. Fix issue that prevents units from being copied
  4. Show additional information in process table
  5. Add sorting process table
  6. Fix: Trigger process isn't part of process table
  7. Update to Transcrypt 3.7
  8. Line "Killed process" can contain the process UID
  9. Add drag-and-drop support for files
  10. Add missing explanations
  11. Allow more characters in program name
  12. Bug fixes

Note

See the commit history of the repository for a full list of changes.

Version 0.3.0 - 2019-11-24:

General

  1. Improve presentation
  2. Separate analysis and visualisation code
  3. Use CSS classes to control the visibility
  4. Strip columns left to the message automatically
  5. Lot if internal improvements and restructuring
  6. Bug fixes

Note

See the commit history of the repository for a full list of changes.

Local Installation

(click to show / hide)
Installing OOMAnalyser is quite easy since OOMAnalyser consists only of two files, an HTML file and a JavaScript file. Both can be stored locally to use OOMAnalyser without an Internet connection.

Installation steps

  1. Download the HTML file and the JavaScript file to the main directory
  2. Open the file OOMAnalyser.html in your favourite browser.

Copyright (c) 2017-2024 Carsten Grohmann mail <add at here> carstengrohmann.de

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.