h5i-db / Manual / Operations guide

h5i-db Operations Guide#

How to run an h5i-db database in production: backup and restore, vacuum and compaction cadence, plan hygiene, disk-usage math, filesystem caveats, and the torn-HEAD recovery runbook.

Everything below refers to the on-disk layout (see crates/h5i-db-core/src/layout.rs):

<root>/
  FORMAT                              # format + minimum reader version
  catalog/tables/<hash-of-name>.json  # table name -> table UUID
  snapshots/<hash-of-name>.json       # snapshot name -> {table uuid: version}
  tables/<table-uuid>/
    HEAD                              # the ONLY mutable object per table
    spec/<revision>.json              # schema revisions
    manifests/<seq>.json              # one immutable manifest per version
    segments/<segment-uuid>.parquet   # immutable data

Two properties make operations simple:

Backup#

Immutable objects mean a plain file copy is a correct backup if you copy in the right order and don't run destructive maintenance concurrently.

Procedure#

  1. Don't run vacuum --apply (or drop-table) during the backup window. Vacuum is the only thing that deletes objects; a copy that races it can miss files it already indexed. Plain writers are safe to leave running.
  2. Copy in this order (older references first; HEAD before the objects it references is the one order that is wrong, so copy HEAD first, per table, then its immutable objects):

bash DB=/data/market.db BK=/backup/market.db-$(date +%F) mkdir -p "$BK" cp "$DB/FORMAT" "$BK/" cp -r "$DB/catalog" "$DB/snapshots" "$DB/forks" "$BK/" 2>/dev/null || true for t in "$DB"/tables/*/; do b="$BK/tables/$(basename "$t")"; mkdir -p "$b" cp "$t/HEAD" "$b/" # 1. pin the version to back up cp -r "$t/spec" "$t/manifests" "$b/" # 2. immutable metadata cp -r "$t/segments" "$b/" # 3. immutable data done

Why this order works: the copied HEAD names some sequence S. Every manifest 0..=S and every segment they reference already existed when HEAD was copied, and (with vacuum paused) nothing deletes them, so all of them are present in the later copy steps. Commits that land during the backup produce manifests > S; they may be half-copied, which is harmless: they are unreachable from the copied HEAD and are exactly what vacuum classifies as debris. 3. Skip transient files if you meet them: HEAD.lock, HEAD.tmp.*. 4. Validate the backup before trusting it:

bash h5i-db tables "$BK" h5i-db verify "$BK" <table> --deep # re-reads every segment checksum

Filesystem/LVM/ZFS snapshots are also fine (crash-consistent is enough: the commit protocol fsyncs data before HEAD moves, so any point-in-time image is a valid database).

Restore#

A backup is a database. Point the CLI at it, or copy it back into place and run h5i-db verify per table. There is no replay/WAL step.

Note that restoring an old backup rewinds all tables to the backup time; to rewind a single table inside a live database, prefer h5i-db restore <db> <table> <version>, which is what versioning is for.

Vacuum#

h5i-db vacuum <db> [table] [--grace-seconds N] [--apply] removes unreachable objects: segments referenced by no committed manifest and no live mutation plan, manifests above HEAD (crashed-writer leftovers), and *.lock / HEAD.tmp.* debris. Without --apply it is a dry run.

Vacuum runs from the base database and treats every fork as a root: it consults every fork's pins and every fork's catalog, so a fork's tables and the base segments its shadows reference are never candidates. It refuses to run through a --fork handle, because reachability is a database-wide question.

Guidance:

Forks#

A fork is a named, writable workspace over a pinned view of the database. It costs one small JSON object and copies no data: its tables are ordinary tables, and the versions it forked from are shared by reference. Three things follow that an operator needs to know.

A fork is a GC root. Its pin holds the base's retention floor down, so set-retention refuses to expire a version a fork pins, and drop-table refuses on a table a fork pins. Both name the fork in the error. The fix is always to promote or drop that fork, never to force the floor.

Forks nest. fork create <db> <child> --fork <parent> creates a fork inside a fork, and a child pins its parent's tables the same way a top-level fork pins the database's. Dropping a parent that a child still pins is refused and names the children; fork drop one level at a time, or drop the whole subtree. Depth is capped at 32.

FORK_INDEX.json is a cache. It sits at the database root and answers "which forks pin what" in one read instead of one per fork. It is rebuilt from the fork objects whenever it disagrees with them, so it is safe to delete at any time and does not need to be backed up: a restore without it simply rebuilds it on first use. Every other object is authoritative.

Routine checks:

h5i-db fork list <db>       # age, tables owned, bytes_own, bytes_pinned
h5i-db fork diff <db> <f>   # what it changed (manifests only, no segments)

Abandoned forks are the usual cause of a database that will not shrink; see Disk-usage math below.

Compaction#

Frequent small appends produce many small segments; queries then pay per-segment open/prune cost. h5i-db compact <db> <table> rewrites them into target-sized segments as a new version (row count is verified to be preserved; the commit aborts otherwise).

Mutation-plan hygiene#

Plans (--plan on delete-range / replace-range) stage their segments at plan time and protect them from vacuum until applied, discarded, or expired (TTL: 7 days, PLAN_TTL_SECONDS).

Disk-usage math#

Nothing is ever deleted except by vacuum, and vacuum only deletes unreachable objects; every committed version pins its segments forever (version retention/GC is roadmap work). Practical consequences:

"Vacuum ran and disk did not shrink"#

The usual answer is a fork. Reclamation is deferred, not lost: a live fork pins the versions it forked from, the retention floor cannot rise past them, and so the segments they reference stay reachable, including the pre-compaction copies of segments main has since rewritten. Main pays nothing for a fork until main tries to reclaim.

h5i-db fork list <db>   # bytes_pinned per fork

bytes_pinned is everything in the pinned version, so it is an upper bound on what dropping that fork could release, not a measure of waste: main's current head usually shares most of those bytes. Two forks pinning the same version each report it, so the column does not sum.

Order of operations to actually reclaim:

  1. h5i-db fork drop <db> <name> (or promote it first, if the work is wanted). Nested forks must go from the leaves up.
  2. h5i-db set-retention …, now free to move past the released pin.
  3. h5i-db vacuum <db> --apply.

A fork's own tables are not vacuumed piecemeal: debris inside them is reclaimed wholesale when the fork is dropped, which is why an abandoned fork costs more than an active one.

Filesystem caveats#

Runbook: torn or corrupt HEAD#

Should not happen on a supported filesystem: HEAD is replaced by write-temp → fsync → rename → directory-fsync. Treat an occurrence as a signal of filesystem misbehavior (see caveats above), not as routine wear.

Symptoms#

Diagnosis#

h5i-db verify <db> <table>        # walks the checksum chain from HEAD back
cat <db>/tables/<uuid>/HEAD       # {"format":1,"table_id":"…","sequence":N,
                                  #  "manifest_checksum":"<blake3-hex>"}
ls <db>/tables/<uuid>/manifests/  # zero-padded sequence-numbered JSON

Find the table's UUID via the catalog: h5i-db tables <db> then match, or grep -l '<name>' <db>/catalog/tables/*.json.

Recovery#

  1. Stop writers for the affected table.
  2. Find the newest intact manifest. Starting from the highest file in manifests/, compute each candidate's checksum and walk its parent chain:

bash b3sum <db>/tables/<uuid>/manifests/<seq>.json # blake3 of file bytes

A manifest is a good recovery point if it parses, its parent_checksum matches the blake3 of the parent file, and every segment path it lists exists with the recorded byte size. (This is exactly the check verify runs from HEAD; you are doing it from a candidate sequence instead.) 3. Rewrite HEAD to point at that manifest. HEAD is four fields of JSON, and manifest_checksum must be the blake3 hex of the chosen manifest's exact bytes:

bash SEQ=…; UUID=…; DB=… SUM=$(b3sum --no-names "$DB/tables/$UUID/manifests/$(printf %012d $SEQ).json") printf '{"format":1,"table_id":"%s","sequence":%d,"manifest_checksum":"%s"}' \ "$UUID" "$SEQ" "$SUM" > /tmp/HEAD.new mv /tmp/HEAD.new "$DB/tables/$UUID/HEAD"

  1. Re-verify: h5i-db verify <db> <table> --deep must be clean.
  2. Clean up: manifests above the recovered sequence and any HEAD.tmp.* / HEAD.lock files are debris; a later vacuum (dry-run first) removes them. Do not delete them by hand before verify is clean.

If no manifest verifies, restore the table from backup (above). Rolling HEAD back this way discards the commits after the recovery point; check committed_at_ns / note in the recovered manifest to know exactly where the table now stands.