Research Notes

Notes on papers I read, systems I build, and ideas I am working through: multimodal generation, 3D vision, world models, and coding agents.

6 posts
TutorialMoELLM Systems

Marin End-to-End Tutorial: From LatentMoE and Datakit to 535B Pre-Training and Agentic RL

End-to-end guide to Open Athena's Marin stack: Normalized LatentMoE, Histogram QB, Datakit 25.25T curation, DP11×EP64 GB200 training with MuonH/AdamH, 262K context extension, Relay & Plateau Dropping SFT, TITO GRPO, and the 26-benchmark Eval Policy.

November 22, 202660 min readEN · 中文
Deep DiveLooped TransformersScaling Laws

Looped Transformers: Compute-Matched Scaling, Training Gradients, and Internal Circuits

Unifying SMELT, Truncate or Not?, Inkling 2 Mini, ELT, and Looped-DiT across LLM scaling, (T, B) BPTT tradeoffs, attention residuals, and elastic/deeply supervised diffusion looping.

November 21, 202626 min readEN · 中文
ToolBrowser ExtensionLLM

Duet-Translator: Side-by-Side Web Translation with Paragraph-Aligned Scrolling

Read any web page and its translation side by side in same-origin iframes, aligned paragraph by paragraph with LIS denoising, streaming previews, and a 6-round recursive self-improvement (RSI) loop for speed and cost.

November 21, 202611 min readEN
ToolCoding AgentsWorkflow

PocketRig: A Phone-First Workflow for Coding Agents with Paseo

Run Claude Code and Codex on an always-on workstation through your own Paseo instance, switch seamlessly between phone and Mac over Tailscale or a company VPN, send heavy jobs to GPU nodes, and review small results anywhere.

November 20, 202611 min readEN
TutorialDistributed TrainingLLM Systems

Distributed LLM Training: Sharding, Collectives, and Parallelism

Notes connecting 16Φ HBM budgets, mixed precision, named device meshes, and the four collectives across DDP/ZeRO/FSDP, Megatron-LM tensor parallelism, GPipe/1F1B pipelines, and DeepSpeed MoE.

October 12, 202622 min readEN · 中文
Deep DiveFlow MatchingRL

Reinforcement Learning in Flow Matching Models

From MDP, PPO/DPO, and GRPO foundations to marginal-preserving ODE-to-SDE conversion, Flow-GRPO/DanceGRPO/MixGRPO/TempFlow-GRPO, forward-process DiffusionNFT and RAM, and multi-turn Agentic RL.

September 2, 202627 min readEN · 中文