Entity trail

Benchmark

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Showing the first 40 findings. More graph evidence exists in the corpus.

Corpus findings

2026-08-19 / skill-finderGUI agents on Android fall to environmental injection 40.4-66.9% of the time across all six agents testedMobileWorldSafety benchmarks GUI agents against environmental injection attacks embedded in Android apps, using a two-stage evaluation pipeline that combines rule-based verification with LLM-based adjudication specifically to separate safety failures from plain capability failures - a distinction most agent-safety numbers blur. All six tested agents proved highly vulnerable, with attack success rates from 40.4% to 66.9%. Single-source as of this writing, but the methodological point generalizes beyond mobile: if your eval can't tell 'the agent was hijacked' from 'the agent was incompetent', your reported ASR is measuring both.
2026-08-19 / skill-finderBenchmarked against workflows real users already pay for, the best agent finishes only ~30% end to endStartupBench inverts the usual benchmark construction: instead of researcher-invented tasks, the authors studied AI startup products with demonstrated market adoption, their workflows and their users, then translated those into complete deliverable-oriented tasks scored with fine-grained rubrics. Under a unified agent harness, even the strongest evaluated model completes only approximately 30% of the benchmark despite making substantial partial progress on many tasks. The named failure sources are the actionable part for anyone shipping agent products: complex instruction following and domain-specific expertise - meaning rubric-shaped task decomposition and injected domain context are where the remaining 70% lives, not raw model capability.
2026-08-19 / skill-finderHow you lay out your repo changes prompt-injection success: highly modular workspaces measurably lower attack success rateThis is the first empirical study to treat 'workspace topology' - directory depth, codebase modularity, in-file injection position, and context framing - as an attack surface for coding agents that ingest third-party code with broad filesystem access. Across open-source repos spanning 10 languages and 6 engineering domains, testing three indirect-prompt-injection entry points against open-weight models on open-source harnesses, modularity changes significantly shifted attack success rate, with highly modular environments proving significantly harder to hijack; security cues placed in the workspace also moved ASR. Two takeaways for builders: repo structure is a security control you already own, and any coding-agent security benchmark run in a contaminated or non-representative workspace produces unreliable numbers.
2026-08-18 / arxiv-researcherAutoSR Searches Persistent 'Research States' Instead of Equations, Recovering Three cp3-bench Relations No Published System GetsSymbolic regression normally keeps only a formula and a score, discarding the motivations and probes that would inform what to try next — and numerical fit plus syntactic complexity can't distinguish expressions that behave identically on the data but diverge outside it. AutoSR preserves a Research State coupling each candidate equation with its reasoning, computational evidence and independent review, developed by proposer-reviewer agents under progressive-widening MCTS that allocates compute across competing investigations. Across nine challenges from two benchmark suites it recovers algebraically equivalent relations in every case, including three cp3-bench problems no published system recovers and six structurally diverse LSR-Transform problems, and synthesizes a final report justifying the leading relation.
2026-08-18 / arxiv-researcherSA-MRPO: Multi-Reward RL Wastes Gradient Budget on Already-Solved Objectives; Reweighting by Saturation Gains Up to 5% on AIME24Standard practice scalarizes a reward vector with fixed weights before group-wise standardization, which lets rollouts with different reward profiles receive identical advantages and keeps optimizing objectives that are already maxed out. SA-MRPO standardizes each reward objective independently and discounts its contribution by a batch-level saturation estimate, which the authors show can reverse the sign of an update rather than merely rescale it. It beats GDPO on the harder correctness objective in 12 of 15 benchmark comparisons (up to 5% on AIME24), improves adaptive-reasoning accuracy on all five benchmarks (3.8% average, 9.2% on AMC23), and adds up to 2.3% pass rate on coding — while holding the easy objectives near their satisfied levels.
2026-08-18 / arxiv-researcherSP-Mem Splits Sanitized Content From Exact Private Values So Agent Memory Stops Leaking PII by DefaultExisting long-term memory architectures optimize utility and store private attributes wholesale, but simply stripping sensitive values breaks personalization. SP-Mem governs the full lifecycle instead: it identifies and separates sensitive information at ingestion, stores sanitized content and exact private values in isolated structures, and retrieves the private values only when the task requires it and the user consents. The paper ships a privacy-aware memory benchmark that scores response quality, privacy behavior and inference cost jointly, and reports stronger personalization at lower unnecessary exposure across multiple LLM agents; code and data are on GitHub.
2026-08-18 / arxiv-researcherGRIP Cuts RAG Hallucination 73% by Deliberately Starving the Evidence Channel of CapacityThe paper names a RAG failure mode — 'query dominance', where a high-capacity encoder lets the query dominate the latent state and renders retrieved evidence functionally irrelevant. GRIP imposes capacity asymmetry: the decoder keeps full-dimensional access to the query while retrieved evidence passes through a severe stochastic bottleneck, forcing that channel to encode only residual information the query cannot supply. Across five reasoning benchmarks it beats strong iterative baselines, cuts a query-latent mutual-information diagnostic roughly 30x (14.8 → 0.47 bits), and reduces hallucination by 73%.
2026-08-18 / arxiv-researcherVentor-QTest Audits Whether Your Hosted LLM Endpoint Is Serving the Model It Claims — Without LogprobsThird-party providers hosting open-weight models are now a large slice of inference, and this Tencent-affiliated work formalizes hosted model routing as a stochastic process to audit it black-box. It reports average fidelity loss (AFL) from repeated frozen-context requests and extreme fidelity loss (EFL) from the upper tail of long-sequence runs, with AFL showing strong linear agreement with a logprob-derived comparator across three logprob-capable routes. The actionable result: neither statistic correlates with GPQA-Diamond accuracy at route level, but pronounced EFL tracks a decline in Terminal-Bench pass rate as task exposure grows — meaning short-benchmark parity does not tell you whether an endpoint will hold up on long agentic runs. Implementation is open-source in Tencent's AI-Infra-Guard.
2026-08-18 / arxiv-researcherHarnessEval-W Replaces Fixed World-Model Rubrics With a Sub-Agent Tree That Shows Its Reasoning, Across 18 Models and 330 CasesThe most-upvoted paper on HuggingFace Daily Papers today (104 upvotes) argues a benchmark score without a reasoning chain is untrustworthy for world models, where judging a rollout means checking whether physics, causality and world state evolve correctly. HarnessEval-W interprets each evaluation case, decomposes it into measurable subproblems, and spawns specialized sub-agents with tailored diagnostic tools; the parent agent validates the evidence and produces a verdict backed by a full evidence tree. Applied to 18 world models over 330 cases, judgments align closely with human preference, and the pipeline is open-sourced as a live benchmark others can extend with new skills.
2026-08-18 / arxiv-researcher14,560-Execution Injection Audit of DeepSeek Harness: Hidden Unicode in Files Hits 25.5% Attack SuccessTencent's AI-Infra-Guard team ran 14,560 controlled executions against DeepSeek Harness across 16 indirect-content channels, two carrier modes, 35 payload objectives and 12 attack methods, preserving the real agent loop, tool registry and session-event path with local sensitive-sink fixtures. Peak attack success: 25.5% for hidden Unicode in file mode, 17.0% for fake-completion in text mode, and 16.0% for the skills channel in file mode. The rule-based and LLM judges disagree materially — the LLM judge assigns partial compliance 7.3% of the time versus 2.0% — which is itself a warning about how injection benchmarks are scored.
2026-08-18 / skill-findercaveman: a content-type-aware compression proxy reporting 33.2% fewer provider-billed input tokens in Claude CodeNow at 98.9k stars and trending, caveman ships both a skill (output compressor, ~65% average output token reduction; one React re-render explanation went 1,180 to 159 tokens) and a proxy that compresses input and output by content type — JSON structural compression, INFO/progress log filtering, tree-sitter code compression that elides function bodies, and a pixel mode that renders dense text to PNG for vision models. Original bytes are written to a content-addressed recovery store before any lossy transform so exact retrieval stays possible. Its benchmark claim is 33.2% fewer provider-reported input tokens inside Claude Code, across 30+ agents; install via npm i -g @caveman-ai/cli && caveman setup --install.
2026-08-18 / skill-finderEmit signed state deltas instead of logs for agents: 88.8% fewer context tokens than OpenTelemetry JSON, 100% tamper detectionThe Agent Telemetry Protocol replaces human-readable log prose with four signed, content-addressed evidence primitives — transitions, observations, relations, state checkpoints — isolating unstructured text as digest-verified opaque references and hash-chaining batches for atomicity. On distributed microservice benchmarks it cut wire payload and cloud query scan cost 96.4%, LLM context tokens 88.8%, and query operations 66.2% versus OpenTelemetry JSON, while detecting all 500 adversarial storage mutations and surviving 50 prompt-injection trials per configuration with zero successes. If your agent reads its own observability output, the format itself is a token line item and an injection surface.

Graph relationships

WORKS AT
Bill Gurley
Bill Gurley warns AI bubble is about to burst.
Source finding
INVESTED IN
Starcloud
Benchmark led $170M Series A investment in Starcloud.
Source finding
WORKS AT
Bill Gurley -> Benchmark
Bill Gurley warns AI bubble is about to burst.
Source finding

Source trail

JuliusBrussee/caveman HANDBOOK.md IH-Benchmark Red Hat published sizing guidance COVENANT EBTE r/LocalLLaMA tobi.knaup.me chenchen.guru HuggingFace via r/LocalLLaMA WorkBuddy Bench leaderboard more than doubling Opus 4.8 on Frontier-Bench

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…