Hacker News
GPT-5.6 Variants Benchmarked on 16 Hack The Box Challenges: Sol Solves 93.75% at $0.29 Median, While Luna Scores 0% on Hard
An August 17 write-up ran five GPT-5.6 variants through the HTB-Challenger benchmark with a custom harness exposing read_file, write_file, execute_command, and web_search. Results climb steeply with tier: Luna 50% (8/16), Luna Pro 68.75%, Terra 75%, Terra Pro 87.5%, Sol 93.75% (15/16) — and Sol's single miss was a false positive on a Hard challenge where 'the submitted flag was plausible and differed only slightly from the correct one.' The cost curve is non-monotonic and worth noting: Terra Pro cost $0.91 median per challenge while the better-scoring Sol cost $0.29.
↳ Follow the thread