Improving Sub-billion Scale LLM Design Experiments

Some of the techniques used in the LLM pretraining design include:

• Embedding Sharing
• Grouped Query Design
• SwiGLU Activations for the Multi Perceptron Layer
• Intermidate blockwise weight sharing