This paper proposes a new decoding strategy for looped language models, called late entry, which allows the model to skip initial passes and directly use a deep context cache, resulting in faster decoding times. Practitioners might care about this approach because it could lead to significant speedups in applications where decoding speed is crucial.
Firehose
Filtered to Papers, tagged “sequential decoding” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives