Training LLMs
As the overwhelming majority of EleutherAI’s resources have gone towards training LLMs. EleutherAI has trained and released several LLMs, and the codebases used to train them. Several of these LLMs were the largest or most capable available at the time and have been widely used since in open-source research applications.
Libraries we currently recommend people use out-of-the-box include:
Mesh Transformer Jax, a lightweight TPU training framework developed by Ben Wang
GPT-NeoX, a PyTorch library built off of Megatron-DeepSpeed which supports training models as large as GPT-3 scale on multiple hosts within a single computing cluster
trlX, a PyTorch library for finetuning large language models with Reinforcement Learning via Human Feedback (RLHF)
RWKV, a PyTorch library for training RNN with transformer-level LLM performance.
Libraries
A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)
RWKV is an RNN with transformer-level performance at some language modeling tasks. Unlike other RNNs, it can be scaled to tens of billions of parameters efficiently.
A library for efficiently training large language models with tens of billions of parameters in a multimachine distributed context. This library is currently maintained by EleutherAI.
A JAX and TPU-based library developed by Ben Wang. The library has been used to train GPT-J.
A library for training language models written in Mesh TensorFlow. This library was used to train the GPT-Neo models, but has since been retired and is no longer maintained. We currently recommend the GPT-NeoX library for LLM training.
Models
A suite of models designed to enable controlled scientific research on transparently trained LLMs
A series of Korean autoregressive language models made by the EleutherAI polyglot team. We currently have trained and released 1.3B, 3.8B, and 5.8B parameter models.
RWKV is an RNN with transformer-level performance at some language modeling tasks. Unlike other RNNs, it can be scaled to tens of billions of parameters efficiently.
An open source English autoregressive language model trained on the Pile. At the time of its release, it was the largest publicly available language model in the world.
A CLIP-like model trained on (text, critique) pairs with the goal of learning the relationships between passages of text and natural language feedback on those passages.
Papers
Other Links
