LoRA Speedrun ranks fine-tuning methods by wall-clock time

LoRA Speedrun ranks fine-tuning techniques by wall-clock training time, not the FLOP counts and loss curves most benchmarks report, per Saivineeth147's GitHub repo. For anyone fine-tuning on a single GPU, that's the number that actually predicts how long a run takes.

The repo picked up 114 points on Hacker News. The thread debated whether wall-clock is the honest metric for someone iterating alone on a single GPU.

It is. FLOPs made sense as a stand-in for speed back when training was compute-bound, with every operation costing about the same wall-clock time. LoRA fine-tuning mostly isn't compute-bound anymore. Memory bandwidth, not FLOP count, now decides how fast a run finishes on a single GPU. That's why FLOPs stopped correlating with wall-clock once training turned memory-bound. A leaderboard that times the run instead of counting FLOPs settles the argument instead of just refereeing it on paper.

Whether other fine-tuning tools start reporting wall-clock next to loss curves is worth watching. That's the number builders actually plan around when they've only got one GPU to work with.