Public story · 2026-08-24 · high

Benchmark ties higher safety scores to more refusals

Privacy scores barely tracked with safety or security, and one model's robustness collapsed from 56.9 to 2.6.

Why now: The paper went up on arXiv on August 20, giving builders comparing open-weight models a rubric beyond one safety number.

Issue

2026-08-24

Confidence

high

Sources

Redaction

passed

Story

A testing platform called aiXamine ran more than 120 large language models through 46 safety, security and privacy checks.

It found that tighter alignment makes models refuse more often. One model that scored 99.3 on safety alignment turned down one in three benign queries during testing.

A model's privacy score barely moved with its safety or security score in the same testing. The three risks don't rise or fall together, so a high score on one doesn't predict the others.

The paper treats safety, security and privacy as connected, not separate rankings. It ran the checks across nine services, in more than 5,000 test runs total.

One model was distilled off-policy, without a correction step back onto the teacher's own outputs. Its robustness score dropped from 56.9 to 2.6 on the same base architecture.

A related quantization study found a 4-bit compressed model beating its uncompressed teacher on most benchmarks. Compression trade-offs cut in different directions depending on the method.

Source trail

arXiv 2608.20554 More from arxiv.org

Claim evidence

aiXamine (arXiv 2608.20554, Aug 20) is a black-box platform running 46 tests across nine services to evaluate safety, security and privacy as interdependent rather than separate axes, applied to over 120 LLMs in more than 5,000 runs. It documents a safety tax where stronger alignment systematically raises over-refusal (a model scoring 99.3 on safety alignment refused one in three benign queries), finds privacy near-orthogonal to other trustworthiness dimensions, and characterizes distillation-induced robustness collapse where off-policy distillation without on-policy correction dropped robustness from 56.9 to 2.6 on the same base architecture. The practical read for builders picking an open-weight model: a single-axis leaderboard number cannot tell you what the model gave up.
Finding 21900 arXiv 2608.20554

Provenance

Canonical issue: 2026-08-24
AI generated: yes
Story unit: 21900
Labels: source-backed

Fetching from the wire…

← The Wire

Public story · 2026-08-24 · high

Benchmark ties higher safety scores to more refusals

Privacy scores barely tracked with safety or security, and one model's robustness collapsed from 56.9 to 2.6.

Why now: The paper went up on arXiv on August 20, giving builders comparing open-weight models a rubric beyond one safety number.

Issue

2026-08-24

Confidence

high

Sources

Redaction

passed

Story

A testing platform called aiXamine ran more than 120 large language models through 46 safety, security and privacy checks.

It found that tighter alignment makes models refuse more often. One model that scored 99.3 on safety alignment turned down one in three benign queries during testing.

A model's privacy score barely moved with its safety or security score in the same testing. The three risks don't rise or fall together, so a high score on one doesn't predict the others.

The paper treats safety, security and privacy as connected, not separate rankings. It ran the checks across nine services, in more than 5,000 test runs total.

One model was distilled off-policy, without a correction step back onto the teacher's own outputs. Its robustness score dropped from 56.9 to 2.6 on the same base architecture.

A related quantization study found a 4-bit compressed model beating its uncompressed teacher on most benchmarks. Compression trade-offs cut in different directions depending on the method.

Source trail

arXiv 2608.20554 More from arxiv.org

Claim evidence

aiXamine (arXiv 2608.20554, Aug 20) is a black-box platform running 46 tests across nine services to evaluate safety, security and privacy as interdependent rather than separate axes, applied to over 120 LLMs in more than 5,000 runs. It documents a safety tax where stronger alignment systematically raises over-refusal (a model scoring 99.3 on safety alignment refused one in three benign queries), finds privacy near-orthogonal to other trustworthiness dimensions, and characterizes distillation-induced robustness collapse where off-policy distillation without on-policy correction dropped robustness from 56.9 to 2.6 on the same base architecture. The practical read for builders picking an open-weight model: a single-axis leaderboard number cannot tell you what the model gave up.
Finding 21900 arXiv 2608.20554

Provenance

Canonical issue: 2026-08-24
AI generated: yes
Story unit: 21900
Labels: source-backed

Benchmark ties higher safety scores to more refusals

Story

Related stories

Minimal AI Scaffold Rediscovers 68% of Known CVEs

AgenticRepair Patches 73% of SEC-Bench Vulnerabilities

One in Seven SWE-bench Verified Tasks Have Mismatched Issues

0.000 Attack Success Score Was Mostly Azure's Filter, Not the Agent

A benchmark quantifies AI safety training's hidden tax

Llama 3.1 8B tops Wazuh, OpenSearch on login anomalies

Agents know when they'll fail, not how to fix it

Attack Burns 67% More Tokens Without Failing Tasks

Source trail

Claim evidence

Provenance

Benchmark ties higher safety scores to more refusals

Story

Related stories

Minimal AI Scaffold Rediscovers 68% of Known CVEs

AgenticRepair Patches 73% of SEC-Bench Vulnerabilities

One in Seven SWE-bench Verified Tasks Have Mismatched Issues

0.000 Attack Success Score Was Mostly Azure's Filter, Not the Agent

A benchmark quantifies AI safety training's hidden tax

Llama 3.1 8B tops Wazuh, OpenSearch on login anomalies

Agents know when they'll fail, not how to fix it

Attack Burns 67% More Tokens Without Failing Tasks

Source trail

Claim evidence

Provenance