Flame graphs are a visualization of profiled software, allowing the most frequent code-paths to be identified quickly and accurately. They can be generated using my open source programs on github.com/brendangregg/FlameGraph, which create interactive SVGs. See the Updates section for other implementations.
The following pages (or posts) introduce different types of flame graphs:
The example on the right is a portion of a CPU flame graph, showing MySQL codepaths that are consuming CPU cycles, and by how much.
The x-axis shows the stack profile population, sorted alphabetically (it is not the passage of time), and the y-axis shows stack depth. Each rectangle represents a stack frame. The wider a frame is is, the more often it was present in the stacks. The top edge shows what is on-CPU, and beneath it is its ancestry. The colors are usually not significant, picked randomly to differentiate frames.
Flame graphs can be generated from any profile data that contains stack traces, including from the following profiling tools:
- Linux: perf, SystemTap, and ktap
- Solaris, illumos, FreeBSD: DTrace
- Mac OS X: DTrace and Instruments
- Windows: Xperf.exe
Once you have a profiler that can generate meaningful stacks, converting them into a flame graph is usually the easy step.
My talk for USENIX/LISA13, titled Blazing Performance with Flame Graphs, explains the visualization and summarizes different types:
These slides can also be downloaded as a PDF.
Not long after flame graphs were released:
- Alan Coopersmith generated flame graphs of the X server.
- Dave Pacheco created them with node.js functions.
- Max Bruning has also shown how he used it to solve an IP scaling issue.
- Dave Pacheco has also created stackvis, a Node.js implementation, available on npm.
More Flame Graph news (updated Apr 2013):
- I wrote some documentation for Linux Kernel Flame Graphs, generated using either perf and SystemTap for the profile data.
- Mark Probst developed Flame Graphs for Instruments on Mac OS X.
- Sam Saffron has developed Flame graphs in Ruby MiniProfiler, and shows examples of amazingly deep stacks.
- Bruce Dawson has an excellent post on Summarizing Xperf CPU Usage with Flame Graphs on Microsoft Windows. It includes examples for Visual Studio and Outlook, and a stack folding script to process the Xperf output.
- Google Chrome's performance analysis tool, WebKit Web Inspector, introduced "Flame Charts", inspired by flame graphs. These are a similar visualization, but the x-axis is time, instead of the alphabet. Check out the screenshot, which includes a mouse-over popup that links to the source code. (Update: there is a bug to add flame graphs to Chrome, in addition to flame charts: Chromium 452624.)
- Tim Bunce has been improving and adding features to Flame Graphs, and has included them in Perl's best profiler, Devel:::NYTProf. See his post on NYTProf v5 – Flaming Precision.
More Flame Graph news (updated Aug 2013):
- I wrote a document summarizing four techniques for generating Memory Leak (and Growth) Flame Graphs, which visualize stacks with byte counts, instead of the traditional CPU sample Flame Graphs. I also colored them green to indicate that they are a different type.
- John Graham-Cumming showed how CloudFlare was using SystemTap-generated flame graphs for optimizing their Lua WAF.
- Yichun Zhang showed how Off-CPU Time Flame Graphs (PDF) can solve issues of blocking time. See Off-CPU Analysis for why this is important.
- Igor Soarez wrote How To Make Flame Graphs for node.js analysis, showing all steps involved.
- Paul Irish and Umar Hansa posted an awesome demo of using FlameCharts to investigate time in V8, which includes zooming in and clicking on functions to go to code, and Addy Osmani posted a longer video tutorial. While these aren't Flame Graphs, they show developments in a related visualization: a time-series version that retains sequence and ordering.
More Flame Graph news (updated Jan 2014):
- I gave a talk at USENIX/LISA13 titled Blazing Performance with Flame Graphs, which covered them in enormous detail. See the slides and video. At the last minute this talk became a Plenary when another speaker became ill and had to cancel.
- An article about flame graphs by Joab Jackson was published in PCWorld, InfoWorld, and CIO.
- I wrote documents on Off-CPU Flame Graphs, and Hot/Cold Flame Graphs, based on what I had documented earlier for my USENIX talk.
More Flame Graph news (updated Mar 2014):
- Vladimir Kirillov has developed eflame (github), a Flame Graph profiler for Erlang. Blocking calls are in blue, see the screenshot.
- Trevor Norris posted instructions for creating Linux Node.js CPU flame graphs, by using perf_events and --perf-basic-prof. See the example.
More Flame Graph news (updated Jun 2014):
- Luca Canali provides many examples of Flame Graphs for Oracle databases, and explains what they mean.
More Flame Graph news (updated Nov 2014):
- Facebook's Strobelight uses an inverted flame graph for perf analysis. See the slide in Yannick Brosseau's Using tracing at Facebook scale (PDF) talk.
- The post on Profiling a Meteor app: Telescope includes flame graphs.
- Adrien Mahieux added the Click to Zoom feature to the original FlameGraph software. This is really awesome!
- I wrote a post on Differential Flame Graphs, showing new features I added to flame graphs for performance regression testing.
- I came up with an unusual but useful use for differential flame graphs: CPI Flame Graphs, to highlight memory stall cycles.
- I gave a talk at the 2014 FreeBSD Developer and Vendor Summit on Flame Graphs on FreeBSD (slides).
- Yunong Xiao showed how flame graphs helped solve an important production performance issue for Netflix in the post Node.js in Flames.
- Oozou developed and published RubyProf::FlameGraphPrinter for ruby-prof, to generate folded stacks for flame graph generation.
Thanks to everyone who has written about flame graphs, developed them further, and shared their results! I'll update this page from time to time with more news.