Skip to content

QA Stack — Overview

PerfShop ships a complete test stack running in parallel with the application itself. The pedagogical goal is to give students a realistic environment where they can write, version, run, and steer automated tests — exactly as in a real organization — without installing anything beyond the base Docker Compose stack.

This page provides the overview of the QA workflow. The following pages dive into the details of each component.

Source of truth

All information in this section comes from the compose files (docker-compose.desktop.yml, docker-compose.build.yml), the custom Dockerfiles (test-runner/Dockerfile), the Squash Orchestrator configurations (squash-orchestrator/*.yaml), and the seed scripts (forgejo-seed/seed.py, squash-seed/seed.py).

Eleven services, three families

flowchart TB
  subgraph perf["Performance testing"]
    direction TB
    JM["perfshop-jmeter<br/>(justb4/jmeter:5.5)<br/>idle container"]
    JMUI["perfshop-jmeter-ui<br/>(node:20-alpine)<br/>custom UI"]
    JMUI -->|"docker exec"| JM
  end

  subgraph alm["ALM and functional execution"]
    direction TB
    SQDB[("perfshop-squash-db<br/>postgres:16")]
    STM["perfshop-testmgmt<br/>Squash TM"]
    ORCH["perfshop-orchestrator<br/>Squash Orchestrator"]
    TR["perfshop-test-runner<br/>RF 7 + pytest 8 + SSH"]
    SEL["perfshop-selenium<br/>standalone-chrome"]
    SQDB --> STM
    STM --> ORCH
    ORCH -->|"SSH :22"| TR
    TR -->|"WebDriver :4444"| SEL
  end

  subgraph dev["Scripting and SCM"]
    direction TB
    FG["perfshop-forgejo<br/>self-hosted Git"]
    SUI["perfshop-scripts-ui<br/>editor/launcher"]
    SUI -->|"Git API + token"| FG
    SUI -->|"docker exec"| TR
  end

  FG -.->|"SCM polling<br/>(clone + pull)"| STM

The three families cover three complementary use cases:

Family Role Control tool
Performance testing JMeter load shots, ramp-up scenarios, results-to-backend correlation perfshop-jmeter-ui
ALM and functional execution Squash TM requirements, tests, and campaigns management; orchestrated execution on a remote runner perfshop-testmgmt (Squash TM)
Scripting and SCM Writing, versioning, ad-hoc launching of Robot Framework / pytest scripts perfshop-scripts-ui + perfshop-forgejo

The total is 11 QA services — no consolidation possible, each building block has a distinct responsibility.

Typical pedagogical workflow

Here is an end-to-end scenario that exercises the full QA stack:

sequenceDiagram
  autonumber
  actor S as Student
  participant SUI as perfshop-scripts-ui
  participant FG as perfshop-forgejo
  participant STM as perfshop-testmgmt
  participant ORCH as perfshop-orchestrator
  participant TR as perfshop-test-runner
  participant SEL as perfshop-selenium
  participant BE as perfshop-app

  Note over S,BE: 1. Writing and versioning

  S->>SUI: Opens the web editor
  S->>SUI: Edits login.robot
  SUI->>FG: Push branch (Git API + CI token)

  Note over S,BE: 2. Ad-hoc local execution

  S->>SUI: Clicks "Run"
  SUI->>TR: docker exec robot login.robot
  TR->>SEL: WebDriver session
  SEL->>BE: GET /login + POST /api/auth/login
  BE-->>SEL: Set-Cookie JSESSIONID
  SEL-->>TR: Assertions OK
  TR-->>SUI: Robot output + log.html
  SUI-->>S: Result in the UI

  Note over S,BE: 3. Integration in Squash TM

  STM->>FG: SCM poll (every N minutes)
  FG-->>STM: List of scripts in the repository
  S->>STM: Creates a test case, links login.robot script
  S->>STM: Launches campaign execution

  Note over S,BE: 4. Orchestration and execution

  STM->>ORCH: POST /jobs (JWT RS512 token)
  ORCH->>TR: SSH public key, copies script, runs robot
  TR->>SEL: WebDriver session
  SEL->>BE: Full scenario (login → cart → checkout)
  BE-->>SEL: Responses
  TR-->>ORCH: Exit code + stdout/stderr + log.html
  ORCH-->>STM: Execution result

  Note over S,BE: 5. Inspection and investigation

  STM-->>S: Campaign report (PASS/FAIL per case)
  S->>SEL: noVNC :7900 to watch the Chrome session (if debugging)

This workflow gathers all the QA stack building blocks into a single flow. No step is artificial — this is the normal path a QA engineer follows in a real organization using Squash TM + Forgejo + Robot Framework.

The central role of the test runner

perfshop-test-runner is the convergence point: scripts reach it through two different paths (ad-hoc UI via Scripts UI, or Squash orchestration via SSH), it runs them, and it produces the same artifacts regardless of the origin path.

flowchart LR
  SUI["perfshop-scripts-ui<br/>(docker exec)"]
  ORCH["perfshop-orchestrator<br/>(SSH :22)"]
  TR["perfshop-test-runner<br/>Robot Framework 7.0<br/>+ pytest 8.0<br/>+ OpenSSH server<br/>+ squash-tf-services"]
  SEL["perfshop-selenium:4444"]
  LOGS["/rf-logs<br/>(bind mount)"]

  SUI --> TR
  ORCH --> TR
  TR --> SEL
  TR --> LOGS
  LOGS -.read by.-> PT["Promtail"]
  LOGS -.read by.-> VEC["Vector"]

This dual entry is possible because the test runner embeds both:

  • An SSH server (openssh-server) for the orchestrator
  • A bash shell reachable through docker exec for Scripts UI

Logs produced in /rf-logs/ are collected by both Promtail and Vector, making them visible both in Loki (dashboard dashboard-jmeter → perfshop-test-runner logs panel) and in OpenSearch (index perfshop-qa*).

Key differences between the two execution modes

Aspect Via Scripts UI (ad-hoc) Via Squash TM (campaign)
Who triggers it? Student at the keyboard Instructor or scheduler
Transport Docker socket → docker exec SSH → ssh user@runner
Traceability Logs in /rf-logs, visible in Scripts UI Logs in /rf-logs + structured result in Squash TM (PASS/FAIL per case)
Versioning The script is already being edited in Forgejo The script is pulled from Forgejo via the Squash TM SCM Git plugin
Reporting Raw Robot Framework output + log.html Squash TM campaign with history, dashboard, assignments
Pedagogical use case "I'm writing a script, I want to check that it works" "I'm running a documented non-regression campaign"

The two modes are not mutually exclusive: the same login.robot can be executed ad-hoc during development, then integrated into a Squash TM campaign for scheduled runs.

JMeter — a parallel flow

The JMeter shot follows a completely separate flow from the functional tools (Squash, Selenium, Robot Framework). There is no orchestrator, no SCM, no centralized reporting — just an idle JMeter container and a custom UI that drives it through the Docker socket.

sequenceDiagram
  autonumber
  actor I as Instructor
  participant JMUI as perfshop-jmeter-ui
  participant SOCK as /var/run/docker.sock
  participant JM as perfshop-jmeter
  participant BE as perfshop-app
  participant PROM as Prometheus
  participant GRAF as Grafana

  I->>JMUI: Selects a .jmx scenario
  I->>JMUI: Configures vUsers, duration, rampup
  JMUI->>SOCK: docker exec perfshop-jmeter<br/>jmeter -n -t scenario.jmx -l results.jtl ...
  SOCK->>JM: Launches the shot
  JM->>BE: HTTP flood (per scenario)
  JM->>JM: Exposes /metrics on :9270 during the shot
  PROM->>JM: Scrapes /metrics :9270 (every 5s)
  PROM->>BE: Scrapes /actuator/prometheus :9090 (every 5s)
  GRAF->>PROM: PromQL query (dashboard-jmeter panels)
  GRAF-->>I: Real-time graphs
  JM-->>JMUI: stdout + results.jtl

This is a deliberate choice: a JMeter shot is a one-off, backend-focused activity that does not need the full ALM machinery. The instructor launches a shot, watches the Grafana dashboard-jmeter, correlates with backend metrics, and that's it.

Summary table of the 11 services

Service Image Default host port Role
perfshop-jmeter justb4/jmeter:5.5 — JMeter engine (idle between shots)
perfshop-jmeter-ui node:20-alpine (code mounted) 3005 Custom JMeter control UI
perfshop-squash-db postgres:16 5433 PostgreSQL dedicated to Squash TM
perfshop-testmgmt squashtest/squash-tm:latest 8088 ALM — requirements, tests, campaigns
perfshop-orchestrator squashtest/squash-orchestrator:latest — SSH-based execution control
perfshop-squash-seed python:3.11-slim — Squash TM project init (one-shot)
perfshop-selenium selenium/standalone-chrome:latest 4444 + 7900 Selenium Grid + noVNC
perfshop-test-runner perfshop-test-runner:latest (local build) — RF 7 + pytest 8 + OpenSSH
perfshop-forgejo codeberg.org/forgejo/forgejo:14 3009 Self-hosted Git
perfshop-forgejo-seed python:3.11-slim — CI account + repo + token init (one-shot)
perfshop-scripts-ui node:20-alpine (code mounted) 3008 Script editor/launcher

Startup dependency chain

The startup order of QA services is constrained by several healthchecks and dependencies:

flowchart TB
  SQDB[("perfshop-squash-db<br/>healthcheck pg_isready")]
  STM[("perfshop-testmgmt<br/>healthcheck /squash/isSquashAlive")]
  FG[("perfshop-forgejo<br/>healthcheck /api/v1/version")]
  FSEED["perfshop-forgejo-seed<br/>(one-shot)"]
  SSEED["perfshop-squash-seed<br/>(one-shot)"]
  SEL["perfshop-selenium"]
  TR["perfshop-test-runner"]
  ORCH["perfshop-orchestrator"]
  SUI["perfshop-scripts-ui"]
  JM["perfshop-jmeter"]
  JMUI["perfshop-jmeter-ui"]

  SQDB --> STM
  STM --> ORCH
  STM --> SSEED
  FG --> FSEED
  FSEED --> SSEED
  FG --> SUI
  TR --> ORCH
  TR --> SUI
  SEL --> TR
  JM --> JMUI

Key points:

  • perfshop-squash-seed explicitly waits for the successful completion of perfshop-forgejo-seed, because it needs the Forgejo CI token (stored in the forgejo-token-data volume) to configure the SCM connection between Squash TM and the Git repository.
  • perfshop-scripts-ui waits for perfshop-forgejo healthy, perfshop-test-runner started, and the perfshop-app backend — it consumes all three.
  • perfshop-test-runner starts as soon as Selenium has started, without waiting for any healthcheck (Selenium responds quickly; the test runner can wait for a WebDriver session when needed).

Going further