Papers

Filtered to mixture-of-experts · clear filter

Browse by term

continual learning 33reinforcement learning 21large language models 9vision-language models 8language models 6benchmarking 5autoregressive models 4diffusion transformers 4generative models 4multimodal learning 4robotics 4video generation 4benchmarks 3computer vision 3policy optimization 3self-distillation 3vision-language-action models 3world modeling 3agent-based systems 2autonomous agents 2calibration 2coding agents 2diffusion models 2foundation models 2image synthesis 2in-context learning 2knowledge graphs 2LLMs 2multimodal evaluation 2multimodal large language models 2

Matching papers

Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

6 upvotes · 21 JUL 2026 · Nuemaan Malik

This paper investigates where the optimizer state should be allocated in the context of mixture-of-experts training to reduce memory usage without sacrificing accuracy. Practitioners might care because optimizing memory usage can be crucial for large-scale language models.

mixture-of-expertsoptimizersmemory-efficient trainingAdamWSkewAdamfactored estimators