Transformers has implemented expert streaming to offload MoE model weights, achieving speeds of ~14 tok/s, 6% behind Strata's 13.9 tok/s, by streaming only the routed experts and using a bounded cache that groups_mm runs on as is. AI summary
Firehose
Filtered to Companies, tagged “large-scale inference” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives