Pyroscope

Pyroscope is PerfShop's continuous profiling service. It receives, in push mode, the CPU, memory allocation, and lock contention profiles produced by the Pyroscope Java agent embedded in the backend, stores them locally, and feeds the Flamegraph panels of the APM dashboards.

Source of truth

This page is taken from pyroscope/pyroscope-config.yml, the backend JAVA_OPTS in the compose files, and the Pyroscope datasource in grafana/provisioning/datasources/pyroscope.yml.

Why Pyroscope?

Continuous profiling is the lesser-known pillar of the "four pillars of observability" (metrics, logs, traces, profiles). Where metrics say "how much", traces say "where in the code", profiles say "which precise line consumes".

Concretely, in PerfShop, Pyroscope answers questions such as:

• "When CPU chaos is active, which Java method is consuming the CPU exactly?"
• "Which class allocates the most objects when the OutOfMemoryError chaos is in progress?"
• "On which synchronized are threads blocking during the Race Condition (A8) simulation?"

Answers are visual: flamegraphs in Grafana, where the width of each rectangle represents the CPU time consumed by the method (or the memory allocated, or the time blocked on a lock).

Architecture

flowchart LR
  subgraph BE["perfshop-app (JVM)"]
    direction TB
    APP["Spring Boot 3.2"]
    PA["Pyroscope agent<br/>pyroscope.jar"]
    APP -.injected.-> PA
  end

  PYRO["perfshop-pyroscope<br/>(grafana/pyroscope:latest)<br/>filesystem backend"]

  GRAF["Grafana<br/>(Pyroscope datasource)"]

  PA -->|"JFR HTTP push :4040<br/>(CPU, alloc, lock)"| PYRO
  PYRO -->|"Pyroscope datasource<br/>(query flamegraphs)"| GRAF

Pyroscope works in push only mode: the agent periodically pushes the profiles, Pyroscope does not scrape. This is different from Prometheus, and it is consistent with the JFR model (one file per time window).

Pyroscope configuration

server:
  http_listen_port: 4040
  log_level: warn

storage:
  backend: filesystem
  filesystem:
    dir: /var/pyroscope/data

scrape_configs: []

Parameter	Value	Effect
http_listen_port	4040	Single HTTP port — used both for push (POST /ingest) and for the query API (Grafana)
log_level	warn	Log level for Pyroscope itself
storage.backend	filesystem	Local storage on the named volume pyroscope-data
storage.filesystem.dir	/var/pyroscope/data	Internal path, mounted on the named volume
scrape_configs: []	(empty)	No target scraped — push-only mode

The default host port is 4040 (variable PYROSCOPE_HTTP_PORT).

No retention configured

PerfShop does not have an explicit retention policy on the Pyroscope side — this is intentional for the pedagogical context (profiles remain available for deferred analysis throughout the duration of a lab). In a real production setup, compactor.retention_period would be set to bound disk consumption.

Pyroscope agent on the backend side

The Pyroscope Java agent is embedded in the perfshop-backend image (path /agents/pyroscope.jar) and activated via JAVA_OPTS:

-javaagent:/agents/pyroscope.jar
-Dpyroscope.server.address=http://perfshop-pyroscope:4040
-Dpyroscope.application.name=perfshop
-Dpyroscope.format=jfr
-Dpyroscope.profiler.event=cpu
-Dpyroscope.profiler.alloc=512k
-Dpyroscope.profiler.lock=10ms
-Dpyroscope.profilingInterval=PT0.02S

Property breakdown

Property	Effect
pyroscope.server.address	Pyroscope endpoint via internal Docker DNS
pyroscope.application.name=perfshop	Identifies the application in Pyroscope — appears as service: perfshop in Grafana
pyroscope.format=jfr	Java Flight Recorder format — the agent uses the JVM's JFR APIs to collect without significant overhead
pyroscope.profiler.event=cpu	CPU profile enabled — this is the main profile
pyroscope.profiler.alloc=512k	Heap allocation profile — a sample is taken every 512 KB allocated (approximately)
pyroscope.profiler.lock=10ms	Lock contention profile — any wait on a lock ≥ 10 ms is sampled
pyroscope.profilingInterval=PT0.02S	CPU sampling every 20 ms (ISO 8601 Duration notation)

Three profiles in parallel

The Pyroscope agent therefore produces three profile streams simultaneously, without interfering with each other:

flowchart TB
  AGENT["pyroscope.jar"]

  AGENT --> CPU["CPU profile<br/>20ms sampling<br/>perf_event or itimer"]
  AGENT --> ALLOC["Allocation profile<br/>1 sample / 512 KB allocated"]
  AGENT --> LOCK["Lock contention profile<br/>blocks ≥ 10 ms"]

  CPU --> P1[("Pyroscope<br/>cpu profile")]
  ALLOC --> P2[("Pyroscope<br/>alloc profile")]
  LOCK --> P3[("Pyroscope<br/>lock profile")]

CPU profile — perf_event vs itimer

Pyroscope automatically chooses the sampling method depending on the environment:

Environment	Method	Notes
Native Linux (VPS, physical server)	perf_event	Samples the hardware CPU, more accurate, but requires specific Linux capabilities
Docker Desktop (Windows / macOS / virtualized Linux)	itimer	Samples via POSIX software timers, works in all containerized environments

This is why the Student and Instructor APM dashboards expose two CPU Flamegraph panels side by side:

• CPU Flamegraph — Linux (perf_event) — empty under Docker Desktop
• CPU Flamegraph — Docker Desktop / Windows / macOS (itimer) — empty on native Linux

One of the two is always empty depending on the environment, but the student can understand the difference and read the one that contains data.

Heap allocation profile

The alloc profile captures the origin of memory allocations — which methods allocate the most objects, at what frequency. This is what the "Heap Flamegraph — memory allocations (F3-OOM)" panel of the Instructor APM dashboard shows.

When the memory backend chaos is active and the heap fills up, the student can:

1. Open the Heap Flamegraph panel,
2. Identify the method at the top of the graph (the widest one),
3. Correlate with the source code to understand which class is responsible for the growth.

Lock contention profile

The lock profile captures threads that wait on a synchronized or a ReentrantLock. The "Lock Contention Flamegraph — JVM locks (A8 Race Condition)" panel highlights contention hot-spots.

It is the main tool for the A8 — Race Condition pedagogical chaos scenario: a thread holds a lock for too long, the others pile up waiting, the lock profile shows exactly where.

Grafana datasource

- name: Pyroscope
  type: grafana-pyroscope-datasource
  uid: pyroscope
  access: proxy
  url: http://perfshop-pyroscope:4040
  isDefault: false
  editable: true

Minimal configuration — the native Grafana plugin grafana-pyroscope-datasource handles everything: profile selection (cpu, alloc, lock), label filtering, comparison between two time windows.

Reading a flamegraph — quick guide

flowchart TB
  R["Root<br/>= 100% of CPU time<br/>over the observed window"]
  R --> M1["Tomcat thread loop<br/>~80%"]
  R --> M2["GC threads<br/>~12%"]
  R --> M3["Others<br/>~8%"]

  M1 --> M11["doFilter (LicenseInterceptor)<br/>~5%"]
  M1 --> M12["doFilter (ChaosInterceptor)<br/>~30%"]
  M1 --> M13["ProductController.list()<br/>~45%"]

  M12 --> M121["CpuChaosScheduler.busyLoop()<br/>~28%"]

Concept	Reading
Width of a rectangle	Proportion of CPU time consumed by the method and all its descendants
Height (vertical axis)	Depth of the call stack
Stacking	A method called by the method above
Color	Decorative (random palette to visually distinguish)
Hot path	The widest path from root to a leaf — that is what to focus on

The classic beginner mistake is to look at the height: it is not an indicator. Only the width matters.

Volumes

Volume	Mount	Content
pyroscope-data (named volume)	/var/pyroscope/data	Profiles stored on filesystem, organized by time series
./pyroscope/pyroscope-config.yml (bind mount)	/etc/pyroscope/pyroscope-config.yml	Pyroscope configuration (read-only)

Ports

Service	Host port	Container port	Env variable
perfshop-pyroscope	4040	4040	PYROSCOPE_HTTP_PORT

To go further

• Overview — four observability signals
• Grafana — Pyroscope datasource
• Shipped dashboards — Flamegraph panels of the Student and Instructor APM dashboards
• Tempo — the other agent embedded in the backend image
• Chaos engineering section — scenarios that exploit continuous profiling (A8 Race Condition, F3 OutOfMemoryError, CPU chaos)