Shrink LLM logs losslessly by hash-referencing repeated strings: condense-json 1.0

Simon Willison·low signalXBlueskyLinkedInCopy link

condense-json 1.0 scans a JSON structure for duplicate strings and substrings drawn from a supplied replacements object and swaps each match for a compact reference, marking condensed regions with a `{"$r": [...]}` wrapper and pointing at replacements via `{"$": "ID"}`. The transformation is fully reversible through `uncondense_json()`, so it is a storage-layer optimization rather than lossy summarization. Willison built it to cut the space consumed by SQLite logs of LLM interactions, where the same prompt fragments and tool schemas recur across every row — a good fit for anyone persisting agent traces at volume.