This paper proposes a new way to use n-gram embeddings in language models, allowing multiple layers to share the same embedding table but adapt it to each layer's needs. Practitioners might care because this approach can significantly reduce the memory required for n-gram tables, enabling larger and more complex models.
Firehose
Filtered to Papers, tagged “language models” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives
Browse by tag
This paper explores whether the Transformer architecture is still suitable for weight-space programming, a technique where a large language model writes target weights by hand without gradient-based training. Practitioners might care because it challenges the conventional choice of Transformers and could lead to more efficient and accurate models.