Scuffed Diagnostics

Scuffed Diagnostics (ScuffedDiag) is an operator tool. It measures tick lag, memory, garbage collection, per-player network traffic and entity populations, writes evidence to disk when something goes wrong, and turns each capture into a shareable report page. It has no player-facing features, so there is no gameplay page. Everything below is for staff.

• Plugin version: 3.13.12 (plugin name ScuffedDiag)

What it does

ScuffedDiag is read-only. It never changes game state. It has three layers:

Every /scuffeddiag subcommand also runs from the server console.

Requirements

Notes:

• Several tools read Linux files (/proc/self/status, cgroup memory files). On a host without them the container memory guard stays idle and the native-memory dump omits those sections.
• ZGC-specific tuning (soft max heap) only applies when the JVM exposes that flag. The boot log states whether the JVM accepted it.
• The heap, JFR and native-memory commands use the JVM's own diagnostic interface inside the process. They need no shell or jcmd, but a host that blocks JFR or the diagnostic MBean will refuse them.

Admin & Staff Commands

Root command: /scuffeddiag. All subcommands are staff-only (see Permissions).

World-scoped commands (entities, entitieschat, interactions) take an optional world name: an exact name, or a substring that matches exactly one world. With no name, a player gets their own world; the console gets every world that has players (busiest first, at most 3), or the first 3 worlds if nobody is online.

Status and live readouts

Recording and captures

report and capture do not run at the same time as another report. A second report while one is in progress is refused.

Unattended capture

Memory and JVM tools

These talk to the JVM directly. Some pause the server, so run them on purpose.

Permissions

Who may run commands:

• The server console is always allowed.
• A player needs OP or scuffeddiag.command, granted to the user or to one of their groups.
• Any other sender (for example a chat bridge) is judged by its own hasPermission("scuffeddiag.command") answer. If a bridge reports every permission as granted, anyone who can reach the bridge's command relay can run these commands, so keep that relay limited to an admin role.

The lag watchdog sends its in-game warning only to online players who have OP or scuffeddiag.command.

Configuration

ScuffedDiag has two files in mods/ScuffedDiag/. Both are created with defaults on first boot.

diag-config.json

One flat JSON object of dotted keys. Keys added by a later version are merged in on load without touching values you changed. /scuffeddiag config reload re-reads the file with no restart. A value of the wrong type falls back to its default, and an unreadable file leaves the file untouched and runs on defaults.

Population census and hotspots

Per-player network watch

Allocation profile on an unattended low-TPS report

Stall evidence retention (mods/ScuffedDiag/stalls/)

Lock dump

Load level

Per-stream network accounting

The two daily caps are counted in memory and start over at every boot, so a day with restarts can write more than one cap's worth. Events and rollups have separate caps so a flood of events cannot stop the rollups.

Network thread gauge

web-config.json

Settings for the report viewer. Written once when absent, so later boots never overwrite your edits.

Applied by /scuffeddiag web reload without a restart: publicHost, publicBaseUrl, autoGenerate, allowBundleDownload. A change to enabled, bindAddress, port, portOffset or linkTtlDays needs a restart; the reload message lists any such keys that differ on disk.

Fixed values (no key)

Detection and capture behavior

Lag watchdog

Polls every 2 seconds, does only cheap reads while the server is healthy, and writes one compact line to <day>/watch-<date>.jsonl when a trigger fires. At most one record per 2 minutes (30 seconds for hangs). Online staff get an in-game warning at most once per 10 minutes. It never starts a capture itself. It stands down while an operator's own recording is on, but keeps running during automatic windows.

A tick-freeze or world-hang also takes a thread and lock-owner dump into stalls/lockdump-<UTC>-<trigger>/threads.txt (plus a locks-*.jsonl line naming deadlock patterns such as a read-to-write lock self-deadlock). The dump skips lock synchronizers on purpose, because collecting them forces a heap-wide safepoint.

Automatic capture windows

Automatic windows never take over from an operator: if recording is on or a capture is running, they do nothing. If a stall window cannot open, the watchdog arms Refixes' meters alone. Refixes' meters record forward, so a stall window explains the next stall in the episode, not the one that opened it.

Baseline folders are <day>/auto-baseline-<UTC>/ (newest 24 kept). Stall windows are stalls/auto-stall-<UTC>/, outside the day folders so the 7-day sweep cannot remove them.

Automatic low-TPS report (TPS guard)

On by default. Every 5 seconds it reads the mean tick interval over the newest 90 ticks of each world that has players (at least 30 recorded ticks). A world whose last tick is 10 seconds old counts as 0 TPS. When the worst world stays at or below the trigger (default 15 TPS) for 15 seconds, it records a 1-minute report into <day>/autoreport-<UTC>/, attaches an allocation profile (see allocAuto.*), and writes a LAGREPORT.txt into the folder. Cooldown between automatic reports is 20 minutes, and the newest 24 autoreport-* folders are kept. It does not run while anything else is recording. Each trigger also adds a line to autoreport-<date>.jsonl.

Heartbeat

Every 5 minutes, whether or not anything is wrong, one line goes to <day>/heartbeat-<date>.jsonl: worlds, instances, idle instances, entity and chunk counts, heap used, heap floor over 10 minutes, post-GC live set (or -1 when untrusted), non-heap, direct buffers, GC count and time, container memory, census summary, top network consumers, load levels and the worst TPS. It keeps writing while the auto report is off and stands down only while a capture is running (the capture's own 30-second cap_worlds series covers it). Use it to see whether instance count, entity count or heap climbs over a day.

Load level

Per world with players, every 5 seconds: smoothed TPS (newest 150 ticks, smoothed) and memory pressure (the larger of the container resident fraction and the 5-minute heap floor fraction). The level is the worse of the two: OK, SOFT or HARD, using the load.* keys. Escalation needs two consecutive polls. Every change is written to load-*.jsonl. ScuffedDiag takes no action on the level; other mods read it through the API.

Container memory guard

If a container memory limit is readable, it checks resident memory (RssAnon plus RssShmem) every 5 seconds against that limit, since the host kills the process on resident memory, not heap. It always publishes a container-aware heap ceiling for the soft-max guard. At 85% it clamps the soft max and writes a container-memory watch row; at 90% it forces a GC through the diagnostic interface and a native heap trim (at most every 2 minutes); at 95% after that it logs that a kill is imminent. With no readable limit it logs that it is idle.

Heap soft-max guard

At boot ScuffedDiag sets the ZGC SoftMaxHeapSize flag to 6 GB (clamped to -Xmx) so the collector reclaims garbage instead of riding to -Xmx. That value is a floor. Every 60 seconds a guard raises the target to the larger of the floor and 1.6 times the live set, never above 75% of -Xmx or the container-aware ceiling, and lowers it again when the retained set falls. There is no config key: use /scuffeddiag softmax <GB> to change the floor until the next restart, or 0 to stop adjusting. The boot log shows whether the JVM accepted the flag.

Data & storage

Everything is under mods/ScuffedDiag/, written as JSON Lines (one object per line, t in epoch milliseconds, times in UTC) by a single background writer thread, so the tick never waits on disk.

mods/ScuffedDiag/
  diag-config.json  web-config.json  reports.jsonl  LATEST.txt  atlas-report.txt
  2026-09-03/                          one folder per UTC day
    session-<UTC>/                     recording started by /scuffeddiag on, report or capture
    autoreport-<UTC>/                  automatic low-TPS report
    auto-baseline-<UTC>/               automatic baseline window
    <reason>-<UTC>/                    window started through the API (for example a remote caller)
    watch-<date>.jsonl  heartbeat-<date>.jsonl  autoreport-<date>.jsonl
    census-<date>.jsonl  hotspots-<date>.jsonl  load-<date>.jsonl  locks-<date>.jsonl
    net-<date>.jsonl  netstream-<date>.jsonl  netev-<date>.jsonl  atlas-<date>.jsonl
    systemtimings_live-<date>.jsonl    attribution armed by the watchdog
    lagreport-<UTC>.txt  heaphisto-*.txt  nmt-*.txt  heapdump-*.hprof.gz  scuffeddiag-leak-*.jfr
  stalls/
    auto-stall-<UTC>/  lockdump-<UTC>-<trigger>/

LATEST.txt holds the path of the newest session folder relative to the plugin folder, so pulling "the newest data" is one read. A flat mods/ScuffedDiag/ from an older version is migrated into day folders at boot.

Reading a session folder

Open SUMMARY.txt first. It lists findings (a warning sign marks anything needing attention), the report link when the viewer is running, and the file list. README.txt repeats the basics. Main files:

Reading the tick numbers: ScuffedDiag measures the interval between ticks, not work time. A healthy world paces at about 33 ms per tick, so a mean near 33 ms is good. TPS and the slow percentage (intervals of 50 ms or more) are the signals; use spark for steady work-time MSPT.

Report viewer

Each capture window mints a link to a self-contained report.html for its folder, served by a small built-in HTTP server (two worker threads, GET and HEAD only, 120 requests per minute). The link is announced when the window opens, so you can watch the page while it records; until the window closes the page shows "Still recording" and refreshes every 20 seconds.

Security model: the link is the access control. Each token is 128 random bits, there is no login and no index page, and a request path is never used to find a file. Anyone holding a link can read that bundle and download its zip, so treat links like the data inside (the bundle can name players). Bind to 127.0.0.1 behind a proxy if that trade is wrong for you, or set allowBundleDownload to false. Links expire after linkTtlDays and stop working once their folder is pruned. They survive restarts through reports.jsonl.

Setup:

• The port defaults to game port plus 2, which usually falls inside the range a hosting panel already opens. Set port to override.
• Set publicHost to the address players use to reach the server. If it is blank, the host is detected when the first report is made (a public address on the server, else the hosting panel's address variable). When only a private address is found, links will not open from outside and /scuffeddiag web says so. Edit publicHost and run /scuffeddiag web reload.
• Use publicBaseUrl instead when a reverse proxy or tunnel serves the viewer and the link must not carry a port.

/scuffeddiag web actions:

The report page

How to read it: world-thread CPU near 100% means expensive work (optimize the owner); a stalled thread near 0% means it was blocked or starved of a core (a lock, I/O, a ZGC allocation stall, or too many threads for the box). A high queue with normal ping points to server-side backpressure, not the player's connection. A JVM-wide pause with no world freeze explains a ping spike for every player at once. A report with no named systems means Refixes was absent or too old.

Integrations

Refixes (optional)

Reached by reflection, so ScuffedDiag runs without it. A capture window enables Refixes' per-system, per-resource and diagnostic meters (queue latency, page events, path-finding, archetype churn), snapshots them every 60 seconds so a cut-short window still has data, and switches them off at the end. Do not run /systemtimings capture alongside a window; the two clear each other's counters.

Discord and other bridges

ScuffedDiag has no Discord settings and no webhook keys of its own. It exposes an API and an event feed (below). The EndlessLink chat bridge uses them for a Discord slash command and for server-log alerts, and its configuration lives with that mod. Commands arriving through a bridge are judged by the sender's own permission answer, see Permissions.

API for other mods

com.zbeve.scuffeddiag.api.DiagApi is the supported entry point (API version 3). Every parameter and return value is a plain JDK type, so a caller can reach it by reflection with no compile dependency, and it degrades to an error map when ScuffedDiag is absent. Methods are safe from any thread and never throw.

Events delivered to a listener, on a dedicated thread through a bounded queue that drops when full:

Load levels are observation only: a mod can read them to shed optional work, and ScuffedDiag itself takes no gameplay action.

Overhead

• Always-on cost is small and bounded: one-second off-thread samplers, a 5-second load poll, and the census, which uses at most about census.budgetUs of world-thread time per tick and reads a large world over several ticks. The census cost appears at the bottom of lagreport.
• With recording off, the freeze stack sampler, taps and spike records are skipped. The tick monitor still records the cheap tick interval so the watchdog and TPS guard can work.
• Recording turns on the stack sampler (only walks stacks while a tick is already over budget) and four packet taps, and runs a collision probe on each populated world every 30 seconds, time-capped at 15 ms. At the default baseline schedule that is a 10% duty cycle.
• Per-system timing in Refixes adds a small allocation per system tick while armed, so it is armed only for windows and for 2 minutes after a severe stall.
• Heavy tools are on demand only: heaphisto and heapdump pause the server, leakhunt and allocprofile add roughly 1 to 2% while recording, and the lock dump costs one safepoint and a stack walk per thread.
• Report building runs on a low-priority thread, and streaming the zip does not buffer the bundle in memory.

Troubleshooting