Transcode logs into an LLM-native alphabet instead of truncating them: 812x smaller, 99.3% forensic QA accuracy versus 51.0% for a token-matched raw prefix

arXiv 2607.28100·low signalXBlueskyLinkedInCopy link

PCAP-LM treats network capture ingestion as lossy knowledge extraction rather than compression, encoding packet direction, TCP/TLS state, log-scale size, and inter-packet delay into a purpose-built ASCII alphabet, then collapsing repetitive behavior with a constrained PMI-BPE tokenizer and motif run-length encoding, with a @REFS side-index preserving lossless drill-down. On 5G/4G TLS 1.3 traffic the vocabulary saturates at 159 tokens and the representation is 812x smaller than `tshark -V`, fitting whole captures in one context window; a frontier LLM scores 99.3% on held-out forensic QA against 51.0% from a token-budget-matched raw prefix. The generalizable skill is the design move — build a domain-specific symbolic representation with a pointer back to the raw data — though the authors are explicit about the cost, including a 24% false-negative rate on TCP retransmissions.