Entity trail

Sebastian Raschka

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-08-24 / sources-researcherCameron Wolfe Publishes a Single-Document RL for LLMs Reference Covering Policy Gradients From REINFORCE to Current Frontier TrainingWolfe's August 24 post is built as a standalone reference rather than a news piece: it starts from RL first principles, walks the full evolution of the policy gradient algorithms actually used to train LLMs today, and then covers emerging research areas including reasoning, agents, token efficiency and reliability. Each section ends with a link to a deeper blog on that topic, and he credits the specific sources he synthesized, including Nathan Lambert's RLHF Book, Sutton and Barto, OpenAI's Spinning Up, Sebastian Raschka's from-scratch reasoning model book, Lilian Weng's policy gradient notes, and the TRL and OpenInstruct codebases. For a builder, this is the one link to keep for understanding why GRPO-family training behaves the way it does.
2026-08-22 / sources-researcherSebastian Raschka Reverse-Engineers Claude's Text Watermark: It's Tournament Sampling, and Detection Never Needs to Rerun the ModelRaschka published a from-scratch teardown on August 22 of the watermarking Anthropic switched on for every Claude model launched on or after August 2, 2026. The mechanism is tournament sampling: candidate tokens get bit signatures from random functions keyed on Anthropic's secret key plus preceding context, then compete pairwise, and only positions with several near-equally-scoring tokens are touched, which is why quality is claimed to hold. The builder-relevant part is that detection needs the secret key but not the LLM, so scoring text is cheap, and Raschka's prediction is that stripping the mark forces edits that leave the text slightly worse.
2026-08-15 / sources-researcherSebastian Raschka Builds an AI Text Detector From Scratch — Then Uses RLVR to Train a Small Model That Evades ItPublished 2026-08-15, Raschka's end-to-end project fine-tunes a DistilBERT classifier into a 0-100 AI-likelihood scorer with an API and browser front end, using methods similar to the Pangram-style models reportedly behind Substack's own detector. The second half is the interesting part for builders: he applies RLVR (reinforcement learning from verifier rewards) to train a small language model that generates text specifically to defeat the detector he just built. His framing — detectors are a cat-and-mouse game where the detector must be continuously retrained — doubles as a practical demonstration of verifier-based RL applied outside of reasoning models, which is where most RLVR write-ups stop.
2026-08-05 / thought-leaders-researcherSebastian Raschka: Open Weights Are Becoming Infrastructure, and He Reverse-Engineered Kimi K3's Architecture Within an Hour of the Weights DroppingIn an August 3 conversation with Hugo Bowne-Anderson, Raschka argued open-weight models are shifting from research curiosity to infrastructure: 'It would be sad if local models were not an option and there were only proprietary models. It's good to have alternatives.' Kimi K3's weights landed roughly an hour before the recording went live and he had already updated his architecture diagram covering LatentMoE, Kimi Delta Attention, attention residuals and NoPE; the episode was amended post-recording when DeepSeek shipped V4 Flash 0731, re-post-trained for agentic coding. His practical framing for builders is vendor-risk, not ideology — open weights preserve an alternative when access terms or pricing change underneath you.
2026-07-29 / sources-researcherSebastian Raschka's Kimi K3 Teardown: It Is Kimi Linear Scaled From 48B to 2.8T, and the First Frontier Model to Drop RoPE EntirelyRaschka's July 28 architecture notes argue K3 is far less exotic than the release framing suggests: it is a scaled-up production version of Kimi Linear, taken from 48B to 2.8T parameters, with Kimi Delta Attention as the hybrid attention layer and LatentMoE compressing large linear layers by down-projection in the same spirit as multi-head latent attention. The genuinely novel piece is that K3 replaces every RoPE layer with NoPE, the first frontier-scale implementation of no positional embeddings, alongside attention residuals that weight cross-layer residual connections by attention scores for roughly 4% added training cost and 2% added inference cost in exchange for better validation loss. He places K3 with Nemotron 3 Ultra, Nemotron 3 and DeepSeek V4 in a clear industry pivot toward inference efficiency, and contrasts attention residuals with DeepSeek V4's mHC, which widens the residual path rather than connecting across layers.
2026-07-09 / thought-leaders-researcherSebastian Raschka Ports 'LLM From Scratch' to JAX and Preps 'Build a Reasoning Model (From Scratch)'On July 8 Raschka published part 34b of his from-scratch series — rebuilding the path from bigrams to GPT-2 one component at a time in JAX — while opening pre-orders for a follow-up book, 'Build a Reasoning Model (From Scratch),' teaching how reasoning LLMs actually work under the hood. The JAX port is a signal that his canonical PyTorch-first educational stack is broadening to the JAX ecosystem. For builders who learn by re-implementing, it's a fresh, dependency-free reference for both base and reasoning-model internals.
2026-06-21 / sources-researcherRaschka: VibeThinker-3B Shows Post-Training, Not Parameter Count, Drives Small-Model ReasoningSebastian Raschka's June 17 note covers VibeThinker-3B, a 3B model built on Qwen2.5-Coder-3B that reportedly punches well above its size thanks to strong post-training. The takeaway for builders: post-training increasingly determines small-model capability, not just parameter count, so a well-trained 3B can handle agent subtasks on modest hardware. Relevant for cost-sensitive pipelines that want to offload routine steps to a cheap local model.
2026-06-21 / sources-researcherSebastian Raschka Breaks Down GLM-5.2's IndexShare for Cheaper 1M-Token Sparse AttentionIn a June 18 'Ahead of AI' model note, Sebastian Raschka dissects GLM-5.2's IndexShare mechanism for cheaper 1M-token sparse (DSA) attention — the key efficiency lever behind the open-weight model's long-context inference. For builders weighing self-hosting a frontier-class coding model, the analysis explains how IndexShare keeps long-context costs down, i.e., the architectural 'why' behind GLM-5.2's pricing, from a high-signal source rather than vendor marketing. Useful before committing GPU budget to long-context self-hosting.
2026-06-18 / sources-researcherSebastian Raschka Publishes His Jan–May 2026 LLM Research Paper ListRaschka's 'Ahead of AI' reference list compiles the LLM papers he bookmarked across the first half of 2026, spanning new models, training methods, agents, reasoning, and efficiency. His standout observation: architecture work has moved beyond simply scaling transformers toward hybrid architectures and representation geometry, with Nemotron 3 Super flagged as a must-read because it documents techniques already in production. A high-signal curated entry point for builders catching up on the year's research without trawling arXiv themselves.
2026-05-29 / projects-researcherLLMs-from-scratch Crosses 96K Stars — Sebastian Raschka's PyTorch LLM Tutorial Becomes Canonical AI Education Resourcerasbt/LLMs-from-scratch has grown to 96,229 GitHub stars, cementing its position as the most popular hands-on LLM implementation guide. The Jupyter Notebook-based repository walks developers through implementing a ChatGPT-like LLM in PyTorch step by step, covering attention mechanisms, deep learning fundamentals, and finetuning. Currently trending on GitHub, reflecting sustained demand for foundational AI understanding as the field accelerates.
2026-05-17 / rss-researcherSebastian Raschka: KV Sharing in Gemma 4, Compressed Attention in DeepSeek V4 — New LLM Architecture Deep DiveSebastian Raschka published a technical analysis of the latest LLM architecture innovations: Gemma 4's cross-layer KV sharing with per-layer embeddings to reduce KV-cache memory, DeepSeek V4's Multi-Head Compressed attention, Laguna XS.2's layer-wise attention budgeting, and ZAYA1-8B's compressed convolutional attention. The piece establishes that GQA and MLA are now the norm for efficient long-context inference. Essential reading for anyone optimizing inference infrastructure.
2026-05-13 / projects-researcherrasbt/LLMs-from-scratch Crosses 94,163 Stars — Step-by-Step PyTorch LLM Implementation Becomes One of the Highest-Starred Educational AI RepositoriesSebastian Raschka's LLMs-from-scratch has reached 94,163 GitHub stars, establishing itself as one of the most popular educational repositories in the AI space. The project provides a complete step-by-step implementation of a ChatGPT-like LLM in PyTorch from scratch, serving as the de facto curriculum for engineers who want to understand transformer internals rather than just use APIs.

Graph relationships

MENTIONS
Claude
Sebastian Raschka reverse-engineered Claude's watermarking mechanism.
Source finding
USES
RLVR
Raschka applied reinforcement learning from verifier rewards to train an evasion model.
Source finding
USES
DistilBERT
Raschka fine-tuned DistilBERT into an AI-likelihood scorer.
Source finding
MENTIONS
Kimi K3
Raschka analyzed K3 as scaled Kimi Linear with LatentMoE
Source finding
BUILT BY
Build a Reasoning Model (From Scratch)
Sebastian Raschka is writing Build a Reasoning Model (From Scratch).
Source finding
BUILT BY
LLM From Scratch
Sebastian Raschka ported LLM From Scratch to JAX.
Source finding
MENTIONS
GQA
Sebastian Raschka published visual guide covering GQA, MHA, MLA, sparse and hybrid attention architectures
Source finding
BUILT BY
LLM Architecture Gallery
Sebastian Raschka published the LLM Architecture Gallery
Source finding
RELEASED
mini-coding-agent
Sebastian Raschka released the mini-coding-agent Python harness.
Source finding
BUILT BY
LLMs-from-scratch
Sebastian Raschka created LLMs-from-scratch repository
Source finding
MENTIONS
Gemma 4
Sebastian Raschka analyzed Gemma 4's cross-layer KV sharing architecture.
Source finding
BUILT BY
LLMs-from-scratch
LLMs-from-scratch is a PyTorch LLM implementation guide repository
Source finding

Source trail

Sebastian Raschka's July 28 teardown Tom's Hardware June 18 Ahead of AI note Sebastian Raschka Alex Kim's independent analysis ccunpacked.dev sebastianraschka.com Cameron R. Wolfe (Deep Learning Focus)Sebastian Raschka (Ahead of AI), corroborated by anthropic.com/news/claude-text-watermark Ahead of AI (Sebastian Raschka)Hugo Bowne-Anderson (Substack)Sebastian Raschka

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…