A continual-learning method loses when the baseline gets stronger

A periodic hierarchy beat replay by 5.0 points at LoRA rank 8, then lost by 11.6 points at rank 72, per a paper comparing periodic hierarchies to replay.

Anyone citing this class of result to justify a training choice is trusting a comparison that can flip with one hyperparameter. The reversal held on Llama-3.2-1B too, and on held-out paraphrases of the same eval questions.

The setup was a 24-month stream of Wikidata facts. The authors evaluated both methods across different months, replay ranks, and query phrasings.

The authors don't call the hierarchy a dead end. They frame it as a lower-update-cost operating point. It's cheaper to keep current. It doesn't win on accuracy once the comparison is fair.

A rank-8 LoRA is a common default in continual-learning papers because it's cheap to run and easy to report. It's also a weak baseline. Raise the rank and replay gains the capacity to retain old facts while absorbing new ones. That's the job replay was built for. The hierarchy's edge wasn't a smarter learning strategy. It was a starved comparison.

A continual-learning result built on one checkpoint, one rank, one eval slice looks like a method win. Ask what happens at the next rank up before you believe it. If the paper doesn't show a sweep, the reported gap is a property of the baseline it picked, not the method it's selling.