Past issues affecting snapshot builds or availability, with a timeline of updates for each.
Snapshot downloads throttled to ~260 KB/s per IP
Resolved
Resolved
Snapshot file serving (the .tar.zst archives and their .sha256 checksums) has been moved to Cloudflare R2, entirely off the Tunnel path that was being throttled. make-snapshot.sh now uploads the retained archive to R2 as its final step, and the download route redirects there instead of streaming from local disk -- the allowlist, per-IP rate limit, and download-count tracking are unchanged, only where the bytes come from moved. Verified directly: R2 now serves at normal, unthrottled CDN speed, performing comparably to an unrelated reference CDN tested from the same network rather than showing the same per-IP cap as before. Older retained snapshots were re-synced to match the current 1-archive retention window.
Identified
Root cause confirmed: the throttle isn't coming from this box. A direct local test against the origin (bypassing Cloudflare entirely) served the same snapshot file at 234 MB/s -- roughly 900x the reported speed -- ruling out disk I/O, the download route, and the local network. The bottleneck is Cloudflare's edge/tunnel bandwidth shaping for large file downloads served through the proxy, applied per-IP regardless of connection count (consistent with a single stream and 16 parallel aria2 connections capping at the same aggregate speed). Moving snapshot file serving to Cloudflare R2 (object storage, no origin exposure, free egress, not subject to the same large-file shaping) while keeping the website and API on the existing Tunnel setup.
Investigating
Received a report from a node operator bootstrapping from AWS us-east-2 (Ohio): snapshot downloads are capped at ~260 KB/s per IP, reproduced with both a single stream and 16 parallel connections (aria2 -x16) -- same aggregate speed either way. Investigating whether this is origin-side (disk, app) or somewhere in the network path.
Some snapshots between Jul 13–19 failed to open
Resolved
Resolved
Verified the fix by booting an isolated test node from a fresh snapshot built with the new capture method: the database opened cleanly, including the ancient/freezer store that was failing before, and the node synced and processed live blocks with no errors. All snapshots built between 2026-07-13 and 2026-07-19 03:00 UTC have been removed. The current "latest" snapshot was built with the fixed process and is safe to use.
Monitoring
Deployed a new capture method: the archive node is now briefly paused for a consistent point-in-time checkpoint instead of being read while live. Files with a real immutability guarantee (the database's *.sst files) are hardlinked almost instantly; everything else (write-ahead logs, the freezer/ancient store) is copied for real while still paused, since those get modified in place even after the node resumes. First snapshot built with the new method completed with no errors. Verifying before removing the affected builds.
Identified
Root cause confirmed: snapshots were captured by running tar directly against the archive node's live data directory while it kept writing and compacting in the background. A long-running tar could archive a database MANIFEST that referenced files the node had since deleted through normal compaction, producing an archive that looked complete but wasn't internally consistent. All builds since 2026-07-13 are affected.
Investigating
Received a report from a node operator that recent snapshot builds fail to open, with the database reporting files referenced by its manifest as missing. Investigating.
