Information-Theoretic Review Frames LLM Agent Architectures as Feedback-Driven Stochastic Processes With Hard Structural Ceilings

arXiv 2608.13510·low signalXBlueskyLinkedInCopy link

This survey argues that achievable ML performance is bounded by structural properties of the data-generating process rather than algorithmic sophistication, working through Fano-type bounds on minimal classification error and Cramér-Rao limits on parametric estimation precision, and examining how implicit assumptions of independence, ergodicity, and distributional stability undermine inferential validity. It reviews Markov Random Fields and potential-based representations for encoding dependence, then describes decision systems including LLM-integrated agent architectures as feedback-driven stochastic processes whose state-dependent dynamics can induce emergent macroscopic behavior. It is a framing paper with no new experiments — useful as vocabulary for why a better harness sometimes cannot beat a badly specified data model.