This project involves implementing a lightweight 1-BIT LLM (Large Language Model) using a custom configuration of Llama architecture, employing quantization techniques to optimize performance and reduce memory usage.
- Model Configuration: Nous-Hermes-Llama-2-7b (customized to a smaller scale).
- Quantization: Includes activation and weight quantization for improved efficiency.
- Backend Framework: Hugging Face Transformers.
- Training Dataset: Cosmopedia-100k-pretrain.
- Implementation of 1-bit quantization for activations and weights.
- Modified Llama RMSNorm and decoder layers.
- Training and monitoring with WandB.
To run this project, ensure you have the following installed:
transformersdatasetswandbacceleratetorch
Install them via:
pip install datasets wandb accelerate transformers torch- Model Config:
NousResearch/Nous-Hermes-llama-2-7b - Layers: 6
- Dimensions: 768
- Heads: 6
- Intermediate Size: 1024
- Context Length: 256
- Hugging Face Token:
hf_************ - WandB Token:
************
Ensure you replace any sensitive credentials in the code or environment variables.
-
Login to Hugging Face:
from huggingface_hub import login login(token="hf_************")
-
Login to WandB:
import wandb wandb.login(key="************")
-
Dataset Setup: Load the dataset from Hugging Face Hub:
data = load_dataset("abideen/Cosmopedia-100k-pretrain")
-
Tokenization: Tokenize the dataset using a customized function to handle truncation and chunking.
-
Model Creation: Create and configure the Llama model:
from transformers import LlamaForCausalLM model = LlamaForCausalLM(config)
-
Quantization: Convert the model to a 1-BIT LLM using custom quantization functions:
convert_to_bitnet(model, copy_weights=False)
-
Training: Set up the training configuration using
TrainingArgumentsand train the model withTrainer:from transformers import Trainer trainer = Trainer( model=model, tokenizer=tokenizer, args=args, data_collator=data_collator, train_dataset=tokenized_data["train"], ) trainer.train()
- Quantization: The activation and weight quantization significantly reduce model size and computational requirements.
- Model Output: The final model is stored in the
./Llama2-70Mdirectory. - Sensitive Information: Ensure API keys and tokens are managed securely and not shared publicly.
- Experiment with larger context lengths and layer counts.
- Improve quantization efficiency and accuracy.
- Add validation and testing pipelines.
Feel free to contribute, open issues, or suggest enhancements for this project!