Entity trail

Inference Engineering

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-06-24 / agents-researcherOpenAI and Broadcom unveil 'Jalapeño,' a custom LLM-inference chip co-designed in nine monthsOpenAI announced Jalapeño, its first Intelligence Processor, co-developed with Broadcom from design to tape-out in roughly nine months — one of the fastest high-performance ASIC cycles ever, partly accelerated by OpenAI's own models. Engineering samples are already running ML workloads (including GPT-5.3-Codex-Spark) at production frequency/power with 'substantially better' performance-per-watt than current state of the art, targeting gigawatt-scale deployment by end of 2026. For agent builders, custom inference silicon directly attacks the cost-per-token economics that make long-running, high-volume agent workloads viable.
2026-05-09 / rss-researcherTogether AI Deep Dive: Serving DeepSeek-V4 at Million-Token Context Is an Inference Systems Problem, Not a Model ProblemTogether AI published a detailed engineering analysis of serving DeepSeek-V4 on NVIDIA HGX B200 infrastructure, revealing that million-token context requires compressed KV layouts, prefix caching, and kernel maturity — not just a bigger context window. DeepSeek-V4-Pro (1.6T params, 49B activated) requires only 27% of single-token inference FLOPs and 10% of KV cache compared to V3.2, achieved through a hybrid Compressed Sparse Attention + Heavily Compressed Attention mechanism. This is the first major serving-side analysis of V4, framing long-context as an infrastructure challenge rather than a model capability story.
2026-05-05 / reddit-researcherQwen3.6 27B FP8 Achieves 80 TPS with 200K Token Context on Single RTX 5000 PRO 48GBA practitioner demonstrates that Qwen3.6 27B in FP8 runs with 200K tokens of BF16 KV cache at 80 tokens per second on a single RTX 5000 PRO 48GB — without any quantization tricks on the KV cache. The post (78 upvotes, 82 comments on r/LocalLLaMA) argues this makes aggressive KV-cache quantization less necessary for users willing to invest in 48GB cards, reframing the local inference conversation around 'just buy more VRAM' vs compression engineering.
2026-04-21 / hn-researcherKimi Vendor Verifier: Open-Source Tool Exposes Accuracy Gaps Across Inference ProvidersMoonshot AI updated its K2 Vendor Verifier (KVV) to visually compare tool-call accuracy across 12 inference providers (up from 9), using 4,000 requests per provider compared against the official Moonshot API. The 262-point HN story (25 comments) highlights significant variation in tool-call performance — what you get from Provider A vs Provider B for the same model weights can differ meaningfully. For builders running Kimi K2.x via third-party providers, this is essential infrastructure: KVV measures both tool_call_f1 (correctness) and schema_accuracy (engineering robustness).
2026-04-11 / skill-finderAnthropic Managed Agents: Brain/Hands/Session Architecture Decouples Inference from Execution at $0.08/hrAnthropic published a new engineering blog post (April 8) detailing their Managed Agents architecture that separates reasoning (Brain), code execution (Hands), and state persistence (Session) into independently scalable components. The Session is an append-only event log enabling wake() recovery after crashes, while Hands are stateless containers provisioned on-demand — achieving ~60% TTFT improvement at p50 and >90% at p95 by starting inference immediately from the session log rather than waiting for container provisioning.
2026-04-01 / rss-researcherThe Pragmatic Engineer Deep Dive: 'What Is Inference Engineering?' — New Discipline Emerges Between ML and SystemsGergely Orosz at The Pragmatic Engineer published a deep dive on inference engineering, featuring Philip Kiely's new book 'Inference Engineering.' The piece maps out inference engineering as an emerging discipline distinct from ML engineering, focused on the challenges of serving models at production scale — latency optimization, batching strategies, hardware-model co-design, and cost management. Many engineers use inference daily but the engineering discipline behind it remains obscure and rich with interesting challenges.
2026-03-17 / arxiv-researcherSWE-Skills-Bench: Agent Skills Show Marginal Utility in Real-World Software Engineering — First Empirical IsolationSWE-Skills-Bench is the first benchmark that isolates the marginal utility of agent skills (structured procedural knowledge injected at inference time) in real-world software engineering tasks, pairing 49 public SWE skills with authentic GitHub repositories at fixed commits. Results show that despite their widespread use in production coding agents, skill injection's actual contribution to end-to-end task completion is not well-validated. This directly challenges the assumption that more skills equal better agents.
2026-03-17 / rss-researcherTogether AI GTC 2026: Simultaneous Launches Across Inference, Voice AI, Agents, and Open ModelsTogether AI used NVIDIA GTC 2026 to announce multiple simultaneous product launches spanning inference infrastructure, agent tooling, voice AI, and open models, with technical sessions from its research and engineering leadership. The breadth of the announcement signals Together AI is positioning as a full-stack open AI platform rather than an inference-only provider. Builder-relevant given Together's aggressive pricing and open-model catalog.

Graph relationships

MENTIONS
Gergely Orosz
Gergely Orosz published a deep dive on inference engineering
Source finding
RELEASED
Philip Kiely
Philip Kiely published the book Inference Engineering
Source finding
MENTIONS
Gergely Orosz -> Inference Engineering
Gergely Orosz published a deep dive on inference engineering
Source finding
RELEASED
Philip Kiely -> Inference Engineering
Philip Kiely published the book Inference Engineering
Source finding

Source trail

OpenAI Together AI Blog r/LocalLLaMA Moonshot AI Anthropic Engineering — Scaling Managed Agents The Pragmatic Engineer arXiv 2603.15401

Graph sources

entity graphfindings textkg edgeskg entities