AI Infra

Concepts intro

Hugging Face articles:

• Model parallelism basis: https://huggingface.co/docs/transformers/v4.15.0/en/parallelism#zero-data-parallel
• FSDP: https://huggingface.co/docs/accelerate/en/usage_guides/fsdp
• Deepspeed: https://huggingface.co/docs/accelerate/en/usage_guides/deepspeed
• Megatron-LM: https://huggingface.co/docs/accelerate/en/usage_guides/megatron_lm

Some Best Practices

• https://github.com/microsoft/DeepSpeed
• https://github.com/NVIDIA/Megatron-LM

LLM

Optimization

Training Optimization: https://developer.nvidia.com/blog/mastering-llm-techniques-training Inference Optimization: https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization

Application

RAG: https://www.youtube.com/watch?v=YuRFba27_1w Agent: https://www.youtube.com/watch?v=q1XFm21I-VQ

MLOps

Intro

https://madewithml.com/

Application

https://www.youtube.com/watch?v=45Zs12Xlg2g

Papers

Communication

• Two-Tree Algorithms for Full Bandwidth Broadcast, Reduction and Scan
• TPU v4 An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

Megatron-LM

• Megatron-LM Training Multi-Billion Parameter Language Models Using Model Parallelism
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
• Reducing Activation Recomputation in Large Transformer Models

Zero

• ZeRO Memory Optimizations Toward Training Trillion Parameter Models

Megascale

• MegaScale Scaling Large Language Model Training to More Than 10,000 GPUs

Cautious

Something critical:

• Do not try to use LLM to enhance the learning process, such generating questions and answers. You will get nothing.
• Consult the expert to save most of the time.

Acknowledgement

Thanks to Liyue Zhang and Guangnan Feng