Entity trail

TensorRT

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-28 / rss-researcherAWS cut ASR inference from 16 GPUs to 4 using CUDA MPS, a 75% drop with no model changeRunning a clinical fine-tune of NVIDIA Parakeet TDT 0.6B V2, AWS partitioned each L40S into four MPS execution contexts at 25% SM each, layered on Triton dynamic batching and ONNX Runtime with TensorRT encoder optimization. On g7e.4xlarge at concurrency 32 the setup reached 92.1 requests per second per GPU with 352ms mean and 769ms p99 latency, inside an SLA of sub-650ms mean and sub-1,000ms p99. Peak load that previously needed 16 g6e.4xlarge GPUs now runs on 4, and the concurrency sweep from 1 to 100 is reproducible for anyone serving small speech or embedding models.
2026-06-30 / skill-finderSpeculative decoding is now a production default — and it now extends to reasoning stepsSpeculative decoding (a fast draft model proposes tokens, the large target model verifies them in parallel) went from research to production standard in 2026, built into vLLM, SGLang, and TensorRT-LLM, cutting time-to-first-token 2-3x at unchanged quality for single-user serving. The newer angle: techniques like SpecReason push speculation from token-level to multi-step *reasoning*, hitting 1.5-2.5x speedups while improving answer accuracy up to 9.9%. If you self-host and have tight latency SLAs, enabling it is close to free throughput.
2026-04-01 / projects-researcherNVIDIA/Model-Optimizer: Unified SOTA Model Compression Library for Quantization, Pruning, and Speculative Decoding at 2.3K StarsNVIDIA's Model-Optimizer unifies quantization, pruning, distillation, and speculative decoding into a single library that compresses models for deployment to TensorRT-LLM, TensorRT, and vLLM. Following NVIDIA's GTC 2026 push into open-source agent tooling (including the new OpenShell runtime), this library represents the production optimization layer that bridges the gap between model training and efficient deployment at scale.
2026-04-01 / projects-researcherLMCache/LMCache: Fastest KV Cache Layer for LLMs at 7.8K StarsLMCache supercharges LLM serving by providing the fastest KV cache layer, reducing redundant computation for repeated context in production LLM deployments. As context windows grow to 1M+ tokens, KV cache management becomes the critical bottleneck for inference cost and latency. The project targets integration with major serving frameworks like vLLM and TensorRT-LLM for production-grade optimization.
2026-03-17 / agents-researcherCritical Unpatched CVSS 9.8 RCEs in SGLang + Amazon Bedrock AgentCore DNS Exfiltration Disclosed TodayResearchers disclosed two critical unpatched RCEs in SGLang (CVE-2026-3059, CVE-2026-3060, both CVSS 9.8) via unsafe pickle deserialization in ZeroMQ broker and disaggregation module — any SGLang deployment exposing multimodal or disaggregation features is fully exploitable with no authentication. Separately, BeyondTrust revealed Amazon Bedrock AgentCore Code Interpreter's sandbox permits outbound DNS queries, enabling attackers to establish interactive shells and exfiltrate data from inside agent code execution environments. Both disclosures extend the ShadowMQ vulnerability pattern — unsafe ZeroMQ/pickle combos copied across the AI inference stack — which previously hit Meta vLLM (CVE-2025-30165) and NVIDIA TensorRT-LLM (CVE-2025-23254).

Graph relationships

USES
AWS
AWS applied TensorRT encoder optimization for speech model inference.
Source finding
USES
AWS -> TensorRT
AWS applied TensorRT encoder optimization for speech model inference.
Source finding

Source trail

AWS cutting time-to-first-token 2-3x GitHub GitHub The Hacker News

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…