Laya gets a native Apple Silicon runtime

A developer built a native Apple Silicon runtime for Laya the day after its release. laya-mlx skips PyTorch and Transformers entirely, running directly on Apple's MLX framework. For anyone weighing whether a small model needs a network round trip at all, that speed is the pitch. The repo launched September 19 and passed 549 stars within a day.

On an M3 Max with 40 GPU cores, the runtime reports 13.4ms median end-to-end latency for short English questions and 7.4ms on the multilingual checkpoint. Batched throughput reaches 146.8 questions per second on the English model and 395.0 on the multilingual one. Peak memory stays between 687.6 and 943.6 MiB per question. A Core ML version from the same author claims about 5ms.

Accuracy didn't get traded away for speed. All three checkpoints matched the upstream model's selected answer on all 63 validation questions, in both FP32 and FP16 precision.

What the repo doesn't say is how these numbers hold up outside a 40-core M3 Max. It's also silent on whether the approach ports to earlier Apple Silicon chips with fewer GPU cores. And it doesn't compare against a PyTorch baseline on the same hardware, so the speedup over the original implementation isn't quantified, only the absolute latency.