Papers

Filtered to Mixture-of-Experts · clear filter

Browse by term

continual learning 86reinforcement learning 48benchmarking 13large language models 12benchmarks 11vision-language models 10language models 9robotics 7world models 7natural language processing 6recursive self-improvement 6generative models 5on-policy distillation 5video generation 5attention mechanisms 4coding agents 4LLMs 4multi-agent systems 4multimodal learning 4multimodal models 4self-distillation 4self-supervised learning 4transformers 4vision-language-action models 4world modeling 4agent-based systems 3agentic models 3agentic search 3agents 3autonomous systems 3

Matching papers

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

90 upvotes · 18 SEP 2026 · Ran Cheng, Longfei Xu, Zheng Liu et al.

This paper proposes a new architecture for Mixture-of-Experts (MoE) models that balances participation, execution, and materialization costs. Practitioners might care because it can lead to significant performance gains in applications where memory and computational resources are limited.

block-conditioningmixture-of-expertssparse routingdense output-mixinghypernetworksresidual gating

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

86 upvotes · 1 SEP 2026 · Shaowen Wang, Ge Zhang, Kairong Luo et al.

This paper investigates how to make Looped Transformers more efficient by scaling their depth while keeping the same number of parameters and compute usage, and it finds that this approach can lead to significant performance gains. Practitioners might care about this because it could lead to faster and more accurate models in resource-constrained environments.

Looped TransformersMixture-of-ExpertsTransformersscaling lawsefficient neural networksneural architecture search