h5i-db / Manual / Operations guide

h5i-db Operations Guide#

How to run an h5i-db database in production: backup and restore, vacuum and compaction cadence, plan hygiene, disk-usage math, filesystem caveats, and the torn-HEAD recovery runbook.

Everything below refers to the on-disk layout (see crates/h5i-db-core/src/layout.rs):

<root>/
  FORMAT                              # format + minimum reader version
  catalog/tables/<hash-of-name>.json  # table name -> table UUID
  snapshots/<hash-of-name>.json       # snapshot name -> {table uuid: version}
  tables/<table-uuid>/
    HEAD                              # the ONLY mutable object per table
    spec/<revision>.json              # schema revisions
    manifests/<seq>.json              # one immutable manifest per version
    segments/<segment-uuid>.parquet   # immutable data

Two properties make operations simple:

Backup#

Immutable objects mean a plain file copy is a correct backup if you copy in the right order and don't run destructive maintenance concurrently.

Procedure#

  1. Don't run vacuum --apply (or drop-table) during the backup window. Vacuum is the only thing that deletes objects; a copy that races it can miss files it already indexed. Plain writers are safe to leave running.
  2. Copy in this order (older references first; HEAD before the objects it references is the one order that is wrong, so copy HEAD first, per table, then its immutable objects):

bash DB=/data/market.db BK=/backup/market.db-$(date +%F) mkdir -p "$BK" cp "$DB/FORMAT" "$BK/" cp -r "$DB/catalog" "$DB/snapshots" "$BK/" 2>/dev/null || true for t in "$DB"/tables/*/; do b="$BK/tables/$(basename "$t")"; mkdir -p "$b" cp "$t/HEAD" "$b/" # 1. pin the version to back up cp -r "$t/spec" "$t/manifests" "$b/" # 2. immutable metadata cp -r "$t/segments" "$b/" # 3. immutable data done

Why this order works: the copied HEAD names some sequence S. Every manifest 0..=S and every segment they reference already existed when HEAD was copied, and (with vacuum paused) nothing deletes them, so all of them are present in the later copy steps. Commits that land during the backup produce manifests > S; they may be half-copied, which is harmless: they are unreachable from the copied HEAD and are exactly what vacuum classifies as debris. 3. Skip transient files if you meet them: HEAD.lock, HEAD.tmp.*. 4. Validate the backup before trusting it:

bash h5i-db tables "$BK" h5i-db verify "$BK" <table> --deep # re-reads every segment checksum

Filesystem/LVM/ZFS snapshots are also fine (crash-consistent is enough: the commit protocol fsyncs data before HEAD moves, so any point-in-time image is a valid database).

Restore#

A backup is a database. Point the CLI at it, or copy it back into place and run h5i-db verify per table. There is no replay/WAL step.

Note that restoring an old backup rewinds all tables to the backup time; to rewind a single table inside a live database, prefer h5i-db restore <db> <table> <version>, which is what versioning is for.

Vacuum#

h5i-db vacuum <db> [table] [--grace-seconds N] [--apply] removes unreachable objects: segments referenced by no committed manifest and no live mutation plan, manifests above HEAD (crashed-writer leftovers), and *.lock / HEAD.tmp.* debris. Without --apply it is a dry run.

Guidance:

Compaction#

Frequent small appends produce many small segments; queries then pay per-segment open/prune cost. h5i-db compact <db> <table> rewrites them into target-sized segments as a new version (row count is verified to be preserved; the commit aborts otherwise).

Mutation-plan hygiene#

Plans (--plan on delete-range / replace-range) stage their segments at plan time and protect them from vacuum until applied, discarded, or expired (TTL: 7 days, PLAN_TTL_SECONDS).

Disk-usage math#

Nothing is ever deleted except by vacuum, and vacuum only deletes unreachable objects; every committed version pins its segments forever (version retention/GC is roadmap work). Practical consequences:

Filesystem caveats#

Runbook: torn or corrupt HEAD#

Should not happen on a supported filesystem: HEAD is replaced by write-temp → fsync → rename → directory-fsync. Treat an occurrence as a signal of filesystem misbehavior (see caveats above), not as routine wear.

Symptoms#

Diagnosis#

h5i-db verify <db> <table>        # walks the checksum chain from HEAD back
cat <db>/tables/<uuid>/HEAD       # {"format":1,"table_id":"…","sequence":N,
                                  #  "manifest_checksum":"<blake3-hex>"}
ls <db>/tables/<uuid>/manifests/  # zero-padded sequence-numbered JSON

Find the table's UUID via the catalog: h5i-db tables <db> then match, or grep -l '<name>' <db>/catalog/tables/*.json.

Recovery#

  1. Stop writers for the affected table.
  2. Find the newest intact manifest. Starting from the highest file in manifests/, compute each candidate's checksum and walk its parent chain:

bash b3sum <db>/tables/<uuid>/manifests/<seq>.json # blake3 of file bytes

A manifest is a good recovery point if it parses, its parent_checksum matches the blake3 of the parent file, and every segment path it lists exists with the recorded byte size. (This is exactly the check verify runs from HEAD; you are doing it from a candidate sequence instead.) 3. Rewrite HEAD to point at that manifest. HEAD is four fields of JSON, and manifest_checksum must be the blake3 hex of the chosen manifest's exact bytes:

bash SEQ=…; UUID=…; DB=… SUM=$(b3sum --no-names "$DB/tables/$UUID/manifests/$(printf %012d $SEQ).json") printf '{"format":1,"table_id":"%s","sequence":%d,"manifest_checksum":"%s"}' \ "$UUID" "$SEQ" "$SUM" > /tmp/HEAD.new mv /tmp/HEAD.new "$DB/tables/$UUID/HEAD"

  1. Re-verify: h5i-db verify <db> <table> --deep must be clean.
  2. Clean up: manifests above the recovered sequence and any HEAD.tmp.* / HEAD.lock files are debris; a later vacuum (dry-run first) removes them. Do not delete them by hand before verify is clean.

If no manifest verifies, restore the table from backup (above). Rolling HEAD back this way discards the commits after the recovery point; check committed_at_ns / note in the recovered manifest to know exactly where the table now stands.