Semantic chunking loses to simple chunking in new RAG test

A July 2 evaluation pitted cluster-based semantic chunking against fixed-size and recursive chunking on long, structured academic theses, scored with RAGAs. The fancier method didn't win.

That's a flag for anyone treating chunking strategy as the safe upgrade: findings like this are worth more than another 2% benchmark bump.

Performance moved more with document formatting, preprocessing, and query type than with chunking strategy itself. The upgrade teams treat as a solved improvement might just be noise next to the prep work nobody wants to do.

There's a second flag in the same paper. RAGAs' faithfulness metric showed limited reliability on this dataset. If faithfulness is the primary gate before a RAG feature ships, that's a problem. A metric that doesn't hold up on long structured documents isn't one you want deciding what goes to production.

Before reaching for a more sophisticated chunker, check how documents are formatted going in and what preprocessing happens before the chunker runs. Then look at whether query routing is doing its job. Chunking strategy is the part everyone tunes because it's legible and easy to swap. This evaluation says that's not where the wins are, at least not on long academic theses.