Introducing gigaGPT: GPT-3 sized models in 565 lines of code

gigaGPT is a codebase by Cerebras that allows for training GPT models with over 100 billion parameters. It utilizes the memory and compute capacity of Cerebras hardware to enable large scale training on vanilla torch.nn code. The models trained with gigaGPT show promising results and can scale to models in excess of 1 trillion parameters.