Now
What is happening this minute, read from files the machines write — never by asking the dev server during a run.
Dev server up, nothing running
— idle on world
Machines
Every machine FreyaCraft runs on, live: production is read once a minute (files, /proc, BlueMap, one tick query), the dev VM every 20 s, this site from itself.
Prod VM 101 · 192.168.40.25
read under a minute agoServer
up
since
·
world
Players
0
nobody
CPU · java
0.0%
· 0.0%
load 0.0 / 0.03 / 0.03 · 4 cores
Memory · disk
8,142 MB
unit · disk free 52,133 MB · world 896 MB
0 crashes in 7 days · 3 on disk
newest ·
crash-2026-08-23_02.06.24-server.txt
Watching Server
java.lang.Error: Watchdog (Watching Server)
java.base@25.0.4/jdk.internal.misc.Unsafe.park(Native
java.base@25.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:271)
knot//net.minecraft.util.thread.BlockableEventLoop.waitForTasks(BlockableEventLoop.java:168)
knot//net.minecraft.util.thread.BlockableEventLoop.managedBlock(BlockableEventLoop.java:158)
0 errors · 0 warnings, last hour · 2 from connections · 0 joins
Netty Epoll IO #7: Error sending packet clientbound/minecraft:disconnect
Netty Epoll IO #4: Error sending packet clientbound/minecraft:disconnect
0 data files rejected at boot · 392 mods
392 mods loaded in 33 s · ready 3.385 s after the server thread started · 11 errors and 286 warnings during the boot
booted
(9 d 1 h ago) · read from
2026-09-20-2.log.gz
7 world backups · newest 15.9 h ago, 545 MB
world-2026-09-29-0430.tar.zst
· 3819 MB over 7 archives in /var/backups/freyacraft
6 timers · 3 services
freyacraft-stats.timer
· next · enabled
freyacraft-ledger.timer
· next · enabled
freyacraft-world-backup.timer
· next · enabled
freyacraft-chunk-audit.timer
· next · enabled
freya-notifier-wipe.timer
· next · enabled
freyacraft-sample.timer
· next · enabled
freya-notifier.service
· active
freyacraft-stats.service
· inactive
freyacraft-world-backup.service
· inactive
0 Ledger actions in the last hour
nobody has done anything the log records
847,274 rows, 161 MB · newest
3/3 BlueMap maps updating
render threads on · world, world_the_end, world_the_nether
last full update
world
· 0 of 315 regions re-rendered in the last hour, last
last hour
last six hours
today
0 today of 315
· one cell is 1024 blocks
world_the_end
· 0 of 0 regions re-rendered in the last hour
world_the_nether
· 0 of 8 regions re-rendered in the last hour, last
last hour
last six hours
today
0 today of 8
· one cell is 1024 blocks
Dev VM 106 · 192.168.40.30
read under a minute agoServer
up
since
·
world
Players
0
nobody
CPU · java
0.6%
· 0.0%
load 0.12 / 0.08 / 0.08 · 8 cores
Memory · disk
2,725 MB
unit · disk free 63,894 MB · world 1,541 MB
22 crashes in 7 days · 140 on disk
newest ·
crash-2026-09-29_10.14.47-server.txt
Watching Server
java.lang.Error: Watchdog (Watching Server)
java.base@25.0.4.1/jdk.internal.misc.Unsafe.park(Native
java.base@25.0.4.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:271)
knot//net.minecraft.util.thread.BlockableEventLoop.waitForTasks(BlockableEventLoop.java:168)
knot//net.minecraft.util.thread.BlockableEventLoop.managedBlock(BlockableEventLoop.java:158)
0 errors · 0 warnings, last hour · 2 from connections · 0 joins
Netty Epoll IO #1: Error sending packet clientbound/minecraft:disconnect
Netty Epoll IO #5: Error sending packet clientbound/minecraft:disconnect
0 data files rejected at boot · 392 mods
392 mods loaded in 24 s · ready 3.167 s after the server thread started · 12 errors and 283 warnings during the boot
booted
(9 h 13 min ago)
8 timers · 2 services
freyacraft-sample.timer
· next · enabled
freya-alert.timer
· next · enabled
freya-gate-check.timer
· next · enabled
freya-build.timer
· next · enabled
freyacraft-dev-ledger.timer
· next · enabled
freya-cases-prune.timer
· next · enabled
freya-client-lane.timer
· next · enabled
freya-window.timer
· next · enabled
freyacraft-dev-world-refresh.service
· inactive
freya-window.service
· inactive
0 Ledger actions in the last hour
nobody has done anything the log records
863,772 rows, 164 MB · newest
3/3 BlueMap maps updating
render threads on · world, world_the_end, world_the_nether
last full update
world
· 4 of 4417 regions re-rendered in the last hour, last
last hour
last six hours
today
1166 today of 4417, newest 600 drawn
· one cell is 1024 blocks
world_the_end
· 0 of 0 regions re-rendered in the last hour
world_the_nether
· 0 of 34 regions re-rendered in the last hour, last
last hour
last six hours
today
0 today of 34
· one cell is 1024 blocks
webgate
VM 102 ·
192.168.40.26 · this site (Phoenix behind Caddy)
app up 2 h 40 min · BEAM 112 MB · 545 processes · run queue 0 on 2 schedulers
Oban refresh ×2 · stats ×1 · default ×3 — 0 running, 0 waiting, 0 scheduled, 0 retrying, 21 discarded
the schedule · 6 jobs
| Job | When | Last run | Failed |
|---|---|---|---|
IndexRefresh |
daily 04:15 UTC | 0 | |
StatsSync |
hourly at :03 | 0 | |
LedgerSync |
hourly at :07 | 0 | |
HarnessSync |
daily */6:45 UTC | — | 21 |
ParitySync |
hourly at :13 | 0 | |
WindowWatch |
daily 14:00 UTC | 0 |
prod stats read
· season Late Spring, day 3
events
pushes in the last 24 h: 8292 received · 0 caught by a net
· 0 unexplained
| kind | host | n | last |
|---|---|---|---|
sample |
dev | 4237 | |
sample |
prod | 1371 | |
stats |
prod | 685 | |
release |
dev | 6 | |
driver |
dev | 2 | |
window |
dev | 10 | |
run |
dev | 1683 | |
ledger |
dev | 258 | |
server |
dev | 32 | |
bundle |
dev | 5 | |
lane |
dev | 1 | |
backup |
prod | 2 |
Gaming PC
ZeroStarTower ·
192.168.0.79 · the rendered client, nightly 02:30 Central
last lane run ·
pass · 66 s to the world
Harness
The dev VM's test harness this minute — the rendered client, the runner, the units — with the run in progress and its live instruments.
VM CPU · java · memory
0.6%
· java 0.0%
unit 2,725 MB · load 0.12 / 0.08 / 0.08 · disk free 63,894 MB · world 1,541 MB
Rendered client
lane
ok
68.1 s to the world
last run ended
(11 h 59 min ago) · 30 fps median
· next 02:30 Central nightly
Frame
behind
the lane's newest run is
20260929T081501Z; the site holds 20260916T074050Z. Everything above it is live, the frame is not — the harvest
is behind.
Null test
20260916T073002Z · two frames 5.7 s apart · 40.4% of sampled pixels moved
The run
The runner's own status file, written after every cycle.
cluster on finished
started (11 h 25 min ago) · ended (9 h 12 min ago)
window
raw
runner.py window
Progress
cycle 19 / 60
seed 6140253
Tick
12.67 ms
median 15.27 ms
Heap
5629 MB
91% of 6144 MB · cut at 85%
four probes in one chunk -- overlapping generation radii on one queue
previous: cluster 19/60 cycles, worst 0 ms, wall budget: cut at 19/60 cycles
Live telemetry The dev server's own instruments, live. The JVM's collection log gives the heap at one-second resolution; the runner gives the tick figure per cycle; the server's log gives its stalls; /proc gives the box. Open on its own while a run is on.
Results
What the harness found: the nightly window, lane by lane; a session's own runs, one row each with their per-cycle numbers.
Nightly window
Latest window
2 h 13 min · anvil · snapshot 2026-09-27
Verdict
pass
8 lanes: 8 pass
Worst tick
60,060 ms
across every lane
| Lane | Verdict | Worst tick | Took | Summary | |
|---|---|---|---|---|---|
| wiring | pass | 0 | 1 min | 2 sides: 3 pass worst 0 ms · 1 min | |
| data | pass | 0 | 8 s | static sweep: 3 pass worst 0 ms · 8 s | |
| arena | pass | 8,040 | 9 min | 34 scenarios: 34 pass worst 8,040 ms · 9 min | |
| gui | pass | 0 | 5 min | 7 scenarios: 7 pass worst 0 ms · 5 min | |
| world | pass | 7,612 | 4 min | 8 scenarios: 8 pass worst 7,612 ms · 4 min | |
| regress | pass | 60,060 | 66 min | 8 cases: 8 pass worst 60,060 ms · 66 min | |
| load | pass | 14,946 | 22 min | 6 scenarios: 6 pass worst 14,946 ms · 22 min | |
| hunt | pass | 10,710 | 21 min | 4 runs in 20 min: 4 pass worst 10,710 ms · 21 min |
Rendered client
| Lane | Verdict | To the world | Ran | Summary | |
|---|---|---|---|---|---|
| client | pass | 66 s | reached the world in 66 s 66 s to the world · |
client
Frames — what the real client drew, one per lane run; click for the full frame where this site holds it
Last thirty nights
wiring
data
arena
gui
world
regress
load
hunt
Session runs — one row per run; click a row for its per-cycle charts, or tick two or more to overlay them
| vs | File | Scenario | Config | Cycles | ms/tick | Heap | World | Worst stall | Outcome |
|---|---|---|---|---|---|---|---|---|---|
variants-sabotage
run ·
|
arena_variants | arena | 3/3 | 5.4 | 1027 MB | +0 MB | — | expect | |
s72-structures-sabotage
run ·
|
arena_structures | arena | 48/48 | 3.3 | 1074 MB | +0 MB | — | expect | |
s71-spawns-sabotage
run ·
|
arena_spawns | arena | 1/1 | 1.9 | 1026 MB | +0 MB | — | expect | |
s71-cooking-sabotage
run ·
|
arena_cooking | arena | 1/1 | 0.5 | 1016 MB | +0 MB | — | expect | |
s70-lootroll
run ·
|
arena_lootroll | arena | 12/12 | 17.5 | 1217 MB | +0 MB | — | ran its length | |
s69-cutting-sabotage
run ·
|
arena_cutting | arena | 9/9 | 8.7 | 1044 MB | +0 MB | — | expect | |
s69-cutting
run ·
|
arena_cutting | arena | 9/9 | 8.9 | 1074 MB | +0 MB | — | ran its length | |
s69-craft-sabotage3
run ·
|
arena_craftrecipes | 9/9 | 4.1 | 1300 MB | +0 MB | — | expect | ||
s69-craft12c
run ·
|
arena_craftrecipes | arena | 133/133 | 4.4 | 1264 MB | +0 MB | — | ran its length | |
s69-craft-smoke
run ·
|
arena_craftrecipes | 9/9 | 0.4 | 1016 MB | +0 MB | — | ran its length | ||
s69-craft-sabotage2
run ·
|
arena_craftrecipes | arena | 9/9 | 0.2 | 1016 MB | +0 MB | — | expect | |
s69-craft-sabotage
run ·
|
arena_craftrecipes | 9/9 | 0.1 | 1030 MB | +0 MB | — | expect |
Pending for prod
Everything staging carries that production does not yet, and the trail each one has left: every stage it passed, who or what moved it, on what evidence. Staging writes the trail; the only thing this page does is take a card off.
36 of 39 checked values agree, and every difference is
accounted for. Read .
staged
1
file world/resources.zip
needs a restart
file world/resources.zipStaging's zip carries twelve BetterNether wood textures production's does not, written by ops/polydec-wood-textures.py at deploy step 11 from each wood's own block model. Without them PolyDecorations' cross-mod furniture names files no mod ships and 311 items draw the missing-texture checkerboard, which production still does. No window item distinguishes the two states: the wiring lane's `pack` item passes on either machine, because before the fix the same thirteen textures were excused by name in packcheck-expected.json. The reading that does distinguish them is `packcheck.py --expected /dev/null` -- zero blank claims with nothing excused at all -- and the move to proven is a session's `parity mark` citing it. Since S68 the zip also carries the merged `items/crossbow.json` from ops/crossbow-definition-merge.py (compat #19: both crossbow mods ship that file, so on every client one drew plain); the wiring lane's `pack` item reads it as one merged asset, none shadowed, and that reading is the same on production once it ships. Ships with the pack regenerate, resource-pack-sha1 rewrite and the restart that hash costs (PROD-DEPLOY steps 13 and 14), so it waits for a window with nobody on.
staged 2026-09-06
staged
proven
rehearsed
- staged · parity · — since 2026-09-06
no proof yet — nothing has proven it on a window
proven
3
datapack freyacraft-modfix
ships any time
datapack freyacraft-modfixStaging's pack sets More Mobs' `$global ts.mm.version` at every load when More Mobs left it at 0 (ops/mod-datafixes.py, more_mobs_version). More Mobs stores that score from a random player's DataVersion, so a boot or /reload with nobody online stores 0 and every gate on it stays shut: no variated mob and no custom head. Production booted that way and its world of 2026-09-24 04:30 holds no score, read from staging's copy of it. arena_modfix's creeper variant needs the score, so it passes nightly only with the fix. Production gets it by `freya-regen --only modfix` on VM 101 and a `/reload` (ops/PROD-DEPLOY.md), after which the digests agree and the card ships.
staged 2026-09-26
staged
proven
rehearsed
- staged · parity · — since 2026-09-26
- proven · window 20260927T090002Z · — arena_modfix pass
datapack freyacraft-compatfix
ships any time
datapack freyacraft-compatfixStaging's pack carries compat #20 and #21. #20: More Delight's, Rustic Delight's and Spanish Delight's knife-on-a-potato cutting recipes rewritten by ops/compat-datapack.py into one recipe with all three results at all three ids, so a knife on a raw potato drops diced potatoes, potato slices and sliced potato together and Spanish Delight's three potato dishes become makeable; production's pack still has the three colliding recipes, of which the manager keeps More Delight's. #21: the six Farmer's Delight cooking recipes that Rustic Delight and Easter's Delight's built-in pack also write (baked cod stew in three copies, noodle soup, beef stew, fried rice, mushroom rice and vegetable soup in two) written once each, every slot the copies disagree on pointed at a freyacraft tag of the copies' own ingredients, so a cooked egg, Rustic's cooking oil or Spanish Delight's sliced onion cooks whichever copy the loader would have kept; production's pack has none of the six, so the loader's pick decides there. arena_cutting's fourth board asserts the three drops and arena_cooking cooks one pot per copy nightly on staging; production gets both by `freya-regen --only compat` on VM 101 and a `/reload` (ops/PROD-DEPLOY.md), after which the digests agree and the card ships.
staged
proven
rehearsed
- staged · parity · — first seen
- proven · window 20260907T090000Z · — arena_cutting pass
hostpack mods
ships any time
hostpack modsStaging's host-modpack serves Bifrost's `bifrost-client-0.4.0.jar` (sha256 e058dc58...) with `config/bifrost-client.properties` `enabled=true`: F2 of Bifrost's crossing plan, a client mod that draws the Valheim disc's mirrors and Vikings, optional through its own `enabled` switch, which AutoModpack's `allowEditsInFiles` keeps across syncs. Filed on Bifrost's channel 2026-09-26. Production carries no Bifrost client mod, and the crossing plan keeps production out until Jose brings it back. The proof is the rendered lane's client syncing the jar from staging and the loader listing it with the switch on (2026-09-26 19:24 UTC, session 95); no window scenario reads it, so the move to proven is that `parity mark`.
staged
proven
rehearsed
- staged · parity · — first seen
- proven · session · — 20260906T073003Z babies frame re-judged by babies_check.py e61810d3: four babies at 0.47-0.54 of their adults, sibling ok (packdefaults 1.1.0 draws More Babies babies of modded mobs small (compat row 15))
- shipped · parity · — the two machines agree; prod 325f0e0ad71ee695ddf20302c7ac5804bd7a886a
- proven · S95 · — rendered client joined staging 2026-09-26 19:24 UTC: synced bifrost-client-0.4.0.jar and its config, loader listed bifrost-client 0.4.0, mod logged loaded enabled=true
History 4
file config/freyacraft-tweaks.json
shipped
file config/freyacraft-tweaks.json
shipped
- staged · parity · — first seen
- shipped · parity · — the two machines agree; prod 2c0108984d7658db0145481825dfb0c9bf0da4e4
- proven · S98 · — 1.4.0 on staging: arena_voxyair passed window 20260927T090002Z (air-section guard); guardVillageSignLocate by hand, rim_walk east with no stall at the village step (/root/s97-rvnproof.json) and an unnamed village_taiga at (792, -680) named on a fresh boot, sign at (792, 84, -682) (/root/s98-rvnvillage-on.txt) (the village-sign guard has no scenario in proven_by; its proofs are the two by-hand runs)
- shipped · parity · — the two machines agree; prod 2c0108984d7658db0145481825dfb0c9bf0da4e4
datapack freyacraft-mob-heads
shipped
datapack freyacraft-mob-heads
shipped
- staged · parity · — since 2026-09-09
- proven · window 20260909T032217Z · — arena_mobheads pass
- shipped · parity · — the two machines agree; prod 7da0f8155d3b2fe9e4c1a1a966089330d3ab3a54
datapack freyacraft-lasso-capture
shipped
datapack freyacraft-lasso-capture
shipped
- staged · parity · — first seen
- proven · window 20260902T090000Z · — arena_lasso pass
- shipped · parity · — the two machines agree; prod 72606b029354d7c1d7e607fa5c73d3cd9f33dfe6
hostpack shaderpacks
shipped
hostpack shaderpacks
shipped
- staged · parity · — first seen
- proven · window 20260903T090000Z · — arena_shaderflora pass
- shipped · parity · — the two machines agree; prod 9fb7cecc204be2df0597c154de258c34a811c129
Evidence
The bundles the runner left — why, on what, what the heap held. The bundle itself stays on the dev VM.
| When | Why | Scenario | Config | Heap | What held it | Server thread |
|---|---|---|---|---|---|---|
20260918-063008-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-061832-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-060548-arena_ice
|
crash | arena_iceburst | arena | — | no histogram | — |
20260918-060125-scatter
|
tick budget | scatterburst | — | no histogram | — | |
20260918-055830-scatter
|
crash | scatterburst | stock | — | no histogram | — |
20260918-055533-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-055201-arena_ice
|
crash | arena_iceburst | arena | — | no histogram | — |
20260918-054734-scatter
|
tick budget | scatterburst | — | no histogram | — | |
20260918-054409-scatter
|
crash | scatterburst | stock | — | no histogram | — |
20260918-054103-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-053237-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-051524-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-051135-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-050628-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-045359-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-044738-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-043919-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-042830-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-042459-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-040140-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-032752-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-015638-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-005821-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-005028-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260918-003652-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260917-234934-arena_ice
|
expect | arena_iceburst | arena | — | no histogram | — |
20260917-234542-scatter
|
tick budget | scatterburst | — | no histogram | — | |
20260917-234304-scatter
|
crash | scatterburst | stock | — | no histogram | — |
20260917-234011-arena_missed
|
expect | arena_missedburst | — | no histogram |
java.lang.Thread.State: TIMED_WAITING (parking)
jdk.internal.misc.Unsafe.park(java.base@25.0.4.1/Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(java.base@25.0.4.1/LockSupport.java:271)
|
|
20260917-233657-arena_ice
|
crash | arena_iceburst | arena | — | no histogram | — |
Both servers over time
Prod and dev side by side over the chosen range: production is read once a minute (files, /proc, BlueMap, one tick query), the dev VM every 20 s.
range
Performance — CPU, tick and memory
Activity — players, stalls and errors, Ledger, BlueMap
Server errors in the trailing hour, prod vs dev
Lines the server itself raised as errors; connection noise from the internet (anything a Netty thread logged) is counted apart and excluded, so a rise here is the server's own. The clean series starts 2026-08-30.
Next
What is scheduled, and what the session that launched the current work wrote about it.
Scheduled
What ran last finished
unit freya-s98-guichat: runner.py run gui_chat --out /root/s98-guichat.json
prove show_chat: options.txt planted chatVisibility:2, gui_chat must pass
How it is read back, and notes
read back with
ssh root@192.168.40.30 'python3 /opt/freya-harness/since.py <stamp>' — or /ops, Session runs and The run
Glossary
Every term on this page, and how the names are read. Terms elsewhere link here; hover one for its short line.
Machines
- dev VM (VM 103)
- The staging copy of the server on the hub, where every test runs.
- A virtual machine on the house hub (Proxmox) that boots a copy of production's world and mods — 4 cores, 10 GB, `192.168.40.30`. Everything the harness does runs here; production is never touched. It carries production's own heap flags and its Minecraft unit stays up, so a measurement taken here is taken on a machine in production's state.
- prod VM (VM 101)
- The live server everyone plays on.
- `freyacraft`, VM 101 on the hub, `192.168.40.25`, the Minecraft server the whitelist admits. The harness reads its nightly world archive into the dev VM and never runs against it.
- webgate
- The ingress VM that serves this site and reads the others over a restricted key.
- VM 102, `192.168.40.26`: Caddy in front of this Phoenix app and BlueMap. It reaches the game VMs over SSH with a key that may only run the harvest scripts by name (`freya-harvest live`, `results`, `client`, …) — no shell, no commands — which is how this page reads the dev VM without holding access to it.
- events
- The pushes: each machine reports a state change the moment it happens, and every reader left on a clock is a net that counts what it catches.
- `POST /api/events/<kind>` behind a per-machine bearer token. The app stores the row, tells every open page and runs the pull the kind implies; a `net` row is a worker on its clock finding something no push announced, which is a fault in a push. Plan of record: `ops/EVENTS-DESIGN.md`.
- gaming PC (ZeroStarTower)
- Jose's PC, which runs the real Minecraft client against the dev VM three times a night: one leg on the arena and two on the fixture world for the distance pair.
- The rendered-client lane: Prism Launcher starts the actual modpack client on the GPU, joins the dev server, stands in the world for two minutes, takes a screenshot and pushes its findings to the dev VM. It is the only tier that draws pixels.
- hub (host)
- The physical Proxmox machine every VM runs on.
- `192.168.0.2`. Every VM shares its CPU and RAM, which is why each VM's memory is sized deliberately rather than generously.
- Ledger
- The mod that logs every block and item a player touches on prod, into a SQLite file the site reads.
- Ledger records block breaks and placements, container traffic and kills per player in `world/ledger.sqlite` on the prod VM. The site's Hall of Fame reads its per-player totals once an hour; the health strip reads the last hour of the log once a minute, so the tile says who is doing what right now without asking the server.
- BlueMap
- The live 3D map of the world, rendered on the server and served on its own port.
- BlueMap renders the world's tiles on the game VM as chunks change and serves them at `map.*`. Its state file (`bluemap/pluginState.json`) says whether the render threads are on, which maps take updates and when the last full update ran; that is what the health strip shows. Render progress lives only in the server's memory, but each map's tile-state files under the web root (`rstate/x<n>/z<n>.tiles.dat`, one per region of 32 hires tiles, 1024 blocks a side) are touched when a tile in that region re-renders; the strip's region map draws the day's touched regions by age on the extent of everything rendered, north up.
The harness
- harness
- The test system on the dev VM: `runner.py` driving scenarios against the server and measuring it.
- `ops/harness/runner.py` boots the dev server on a config profile, arms Carpet fake players (probes), runs a scenario for a number of cycles, and reads the server's own instruments: tick time from the game, stalls from its log, the heap from the JVM's GC log, the world's size from disk. Every number on this page is the harness's own.
- runner.py
- The harness's command: `run`, `verify`, `hunt`, `window`, `repro`, `regress`, `server`, `world`.
- One Python process per run. It is the only RCON client while it runs — a second one would interleave replies and inflate the numbers — so this page reads files it writes (`live.json` per cycle) and never asks the server directly during a run.
- scenario
- One kind of load or check: idle, settle, travel, cluster, chunkrace, chop, nether, explore, the arena and world checks.
- A module in `ops/harness/scenarios/` with a SPEC (probes, tick mode, cycles, what it asserts) and a `drive` step run once per cycle. `idle` moves nothing and is the control; `travel` teleports probes across the map; `cluster` packs them together; `chunkrace` races chunk loads on a seed; `settle` churns blocks at the view-distance edge; the `arena_*` and `world_*` ones check a feature works and record their tick numbers alongside.
- cycle
- One drive step of a scenario, then time advances; the run's unit of progress.
- After each cycle the runner reads the tick figure, the log's stalls, the round-trip times of its own commands and the last collection in the GC log, and writes them to `live.json`. `cycle 12 / 60` is twelve of sixty done.
- tick mode
- How game time advances between cycles: real, burst, sprint or step.
- `real` — the server ticks at its normal 20 a second and the runner sleeps `cycle_sleep` seconds per cycle; the tick figure is the server's own 100-tick average. `burst` — drive a cycle, then `/tick sprint` some ticks as fast as the box can; the figure is ms per tick during the sprint, which measures the box's speed, not what a player feels. `sprint` — set up once, sprint the whole budget. `step` — freeze and step a fixed number of ticks, deterministic, for races.
- probe
- A Carpet fake player the runner arms as a stand-in for a real one.
- `Probe1`…`Probe4`, spawned with Carpet's `/player` command. They load chunks, hold tickets and take damage like a player, but never answer a client handshake — which is why Voxy World Gen ignores them unless the dev-only `freyacraft-harness` mod marks them Voxy-modded.
- Voxy-modded probe
- A probe marked as if its client had answered Voxy World Gen's handshake, so the mod generates terrain around it.
- Voxy World Gen generates chunks only around players whose client acknowledged its handshake; a fake player never does. `/harness voxymodded <probe>` (the dev-only harness mod) sets the same flag, so a run exercises the mod's generation. `voxy_modded` on a result records which probes were marked.
- arming
- Spawning the probes and setting the scenario up, before the first cycle — measured apart, never judged.
- The first spawn of a boot generates virgin terrain on a cold JVM and stalls the server for ten to fifteen seconds; that is the harness's cost, so its stalls and slow commands are recorded under `arming` and the cycles are measured from after it. When arming stalled, the runner waits the server's 15-second warning interval before marking where the cycles begin.
- config profile
- A named server setup a run is measured on: `stock`, `prod-roster`, `no-voxy-worldgen`, `voxy-r64-t3`, `stock-histogram`…
- A profile fully determines the files it names — a `.properties` or `.json` config, mods parked or restored, datapacks swapped, extra JVM flags — so the other side of an A/B can never inherit a leftover. `stock` is dev's whole roster at default config; `prod-roster` parks the sixteen mods production does not carry; `no-voxy-worldgen` parks that one mod; `voxy-r<radius>-t<tasks>` sets Voxy World Gen's generation radius in chunks and its concurrent tasks; `stock-histogram` adds `-XX:+PrintClassHistogram` so every thread dump carries a class histogram.
Modes
- verify
- The same scenarios on two or more configs, same seed, same order — an A/B with a per-scenario table.
- Each config restarts the server on the pristine fixture world so the second side never reads terrain the first generated. The `diff` is one row per config per scenario, never averaged across scenarios.
- hunt
- A timed wander through the load scenarios in random order on one config, looking for anything that breaks.
- `hunt --minutes 120 --config stock`: the runner picks scenarios and seeds by the date and runs them back to back until the budget is spent, saving a bundle for every failure. `hunt_diff.py` sets two hunts side by side, run by run.
- window
- The nightly run of every lane in order — arena, gui, world, regress, load, hunt — one results file per night.
- `freya-window.timer` fires it at 04:00 Central. Each lane's verdict, worst tick, duration and items are what the Nightly board shows; the thirty-night strip is one dot per lane per night.
- repro / regress
- Fire a saved case repeatedly for a rate (repro); run the whole corpus of cases (regress).
- A case is a scenario plus config plus seed that once failed, kept under `ops/harness/cases/`. `repro` reports how often it fires; `regress` runs every case and expects the fixed ones to stay quiet.
- lane
- A group of scenarios the window runs together: arena, gui, world, regress, load, hunt — and the rendered client from the PC.
- The functional lanes (arena, gui, world) check features on a flat arena world or the fixture; regress replays the corpus; load and hunt stress the server. The client lane is the PC's real client run, judged on reaching the world, textures and crashes.
- arena lane
- Twelve functional checks on a flat arena world — does each staged feature do what it says.
- `arena_*` scenarios: composters, cutting, crafters, ice, lassos, mob heads, mob tweaks, the mod fixes, the pets fix, rails, biomes, compat. Each places what it needs by command, runs a few bursts, and reads the world back. Quick and deterministic; a fail here is a feature that broke.
- gui lane
- Six checks through a real headless client on the arena — chat, smithing, anvil, enchanting, trading, the calendar.
- The headless client (tier 3) joins the dev server and the harness reads its screens, slots and tooltips. It proves the client-side of a feature: the rename the tweaks mixin makes, the GUI a mod adds.
- world lane
- Six checks on the fixture world — sleeping, graves, ponds, fishing, a walk over every base, structure generation.
- `world_*` scenarios run on the pristine copy of production's world against targets discovery read from its region files (warps, beds, ponds, river banks, structure starts). They prove the tweaks hold on real terrain.
- regress lane
- The corpus of saved cases, each fired for its expected outcome — a fixed bug stays quiet, an open one still fires.
- Each case is a scenario + config + seed that once failed. `quiet` cases (fixed) must not fire; `fires` cases (open, with a guard elsewhere) are expected to; `any` records a rate. A warn is a case that fired when it should not have or took a stall the SPEC allows.
- load lane
- The load scenarios at fixed seeds — settle, travel, cluster, chunkrace, chop — judged on their tick budgets.
- Each drives four probes for a fixed number of cycles and fails on a stall over its SPEC's `max_tick_ms`, a crash, an OOM or a dead RCON. Its numbers are the night-to-night baseline.
- hunt lane
- Twenty minutes of the load scenarios in an order seeded by the date — the part of the night that looks for what nobody wrote a case for.
- Whatever fails becomes a bundle and, once read, a case in the corpus. Its verdict is the worst run's.
- client lane
- The gaming PC's real client joining the dev server: sync, join, textures, a crash, a frame.
- Pass means the client reached the world; findings are disconnects, crash reports and real missing textures (template models' unbound variables are counted apart).
Measurements
- ms per tick
- How long the server needs for one game tick; the budget is 50 ms (20 ticks a second).
- Under 50 ms the server keeps up; over it, time falls behind and players feel it. In real tick mode the figure is the server's own 100-tick average; in burst mode it is the sprint's ms per tick, a measure of the box's speed under that load.
- stall
- A moment the server thread fell behind: its own "Can't keep up! … Running Nms behind" line.
- The server writes that line at the top of its tick loop, at most once per 15 seconds, carrying the lag it has not yet reported — so one line can stand for several short stalls and lands up to 15 s late. `worst stall` is the largest figure seen during the cycles; the threshold that fails a load scenario is in its SPEC (usually 5,000 ms).
- slow command
- A runner command whose round trip took longer than a tick should — the second stall instrument.
- Every command the runner sends runs on the server thread, so its round trip reads the thread's state directly; one over the threshold is logged with the command. A stall watch dumps the server thread's stack from inside any wait past 8 s.
- heap (live set)
- Memory the JVM's objects hold after a collection — what the server really keeps, not what it has borrowed.
- Read from the JVM's GC log: each collection's `before -> after (committed)`. `after` a young collection still includes the old generation's garbage, so it overstates; the histogram's total after a forced full collection is the honest live set (`live_full_mb`). Committed is what the JVM has taken from the OS so far, up to `-Xmx` (5,120 MB on dev).
- heap cut
- A run is stopped when its live set passes 85% of `-Xmx` — the next stop would be a crash.
- Past that line the run is measuring memory, not what it was written to measure, and its next stop is a watchdog crash inside whatever the server thread happens to be doing. The crossing is judged on the live set, never on the figure after a young pause (G1 leaves the old generation's garbage in, so that figure sits near `-Xmx` by design under a hot allocator): the GC log's last full collection when it is recent, else a histogram forced under `stock-histogram`. A crossing with the live set under the line is `held` and the run goes on; one nothing can judge is `unjudged` and the run goes on to whatever ends it. The cut takes a bundle with a thread dump — under `stock-histogram`, a class histogram naming what fills the heap.
- GC pause / full GC
- A stop-the-world moment while the JVM collects; a full one is the heap's distress signal.
- G1 collects the young generation often (tens of ms) and falls back to a full compaction only under pressure (seconds). `full_gcs` counts the latter since arming; a histogram dump forces one on purpose.
- world growth
- How many MB the world on disk grew during a run — what the run generated.
- `du` of the level before and after. Voxy World Gen's generation shows here first: chunks it generates are saved like any visited chunk.
- class histogram
- Every class in the JVM with its instance count and bytes, biggest first — what fills the heap, by name.
- Taken by SIGQUIT under `-XX:+PrintClassHistogram` (the `stock-histogram` profile or `FC_JVM_FLAGS`), which forces a full collection first. The bundle keeps the dump; this page shows the top rows and the total.
- evidence bundle (case)
- A directory the runner leaves when a run fails, is cut, or asks for a closing dump: dump, log slice, scenario, config, timings.
- `/var/lib/freya-harness/cases/<stamp>-<scenario>/`: `case.json`, `thread-dump.txt`, `server-log.txt`, any stall dumps and crash reports. The bundle stays on the dev VM; the Evidence section shows its summary.
- verdict
- What fails a run: a load scenario on its tick budget, a functional one on its checks; a crash, an OOM or a dead RCON fails either.
- Everything else the runner measures is recorded, never judged. A SPEC can list its own verdict keys; a run started with `--set verdict=[…]` records stalls without failing on them.
Names
- unit names (freya-s29-grid-smoke)
- `freya-` + the session that started it + what it is + `-smoke` when it is the plumbing run.
- A run outliving the session that launched it lives in a systemd unit on the dev VM, named `freya-<session>-<what>`: `freya-s29-idle` is session 29's idle run, `freya-s29-grid` its config grid, `-smoke` the same chain deliberately shortened to prove every step before the real one is scheduled (RULES F12). `freya-window` and `freya-long-roster` are timers; `freyacraft-dev` is the Minecraft server itself.
- result files (s29-idle-real.json)
- `s<session>-<what>.json` in /root on the dev VM: one file per runner invocation, every run inside it.
- `--out /root/s29-idle-real.json` — the session that ran it, what it was, `-smoke` for a plumbing run, `-attempt1` for one that was redone. The Session runs table is these files, flattened to one row per run.
- voxy-r64-t3
- Voxy World Gen's config as a profile: generation radius 64 chunks, 3 chunks generating at once.
- `generationRadius` is how far the mod generates around each Voxy-modded player, in chunks; `maxActiveTasks` how many chunks it has in flight. The grid `voxy-r{32,64,128}-t{3,6}` prices the setting that ships; dev's own file is radius 128, 6 tasks.
- session (S29)
- One working session in the repo's history; the number stamps what that session started.
- The project keeps a session file per sitting (`context/sessions/session-29.md`). Units, result files and bundles carry the session number so a reading can be traced to the notes that explain it.