wehub-resource-sync
caf324b09d
Build documentation / build (push) Failing after 0s
Deploy "method_comparison" Gradio to Spaces / deploy (push) Has been cancelled
Deploy "PEFT shop" Gradio app to Spaces / deploy (push) Has been cancelled
tests on transformers main / tests (push) Has been cancelled
tests / check_code_quality (push) Has been cancelled
tests / tests (ubuntu-latest, 3.10) (push) Has been cancelled
tests / tests (ubuntu-latest, 3.11) (push) Has been cancelled
tests / tests (ubuntu-latest, 3.12) (push) Has been cancelled
tests / tests (ubuntu-latest, 3.13) (push) Has been cancelled
tests / tests (windows-latest, 3.10) (push) Has been cancelled
tests / tests (windows-latest, 3.11) (push) Has been cancelled
tests / tests (windows-latest, 3.12) (push) Has been cancelled
tests / tests (windows-latest, 3.13) (push) Has been cancelled
Secret Leaks / trufflehog (push) Has been cancelled
CI security linting / zizmor latest via Cargo (push) Has been cancelled
42 lines
1.1 KiB
Bash
42 lines
1.1 KiB
Bash
python train.py \
|
|
--seed 100 \
|
|
--model_name_or_path "mistralai/Mistral-7B-v0.1" \
|
|
--dataset_name "smangrul/ultrachat-10k-chatml" \
|
|
--chat_template_format "chatml" \
|
|
--add_special_tokens False \
|
|
--append_concat_token False \
|
|
--splits "train,test" \
|
|
--max_length 2048 \
|
|
--num_train_epochs 1 \
|
|
--logging_steps 5 \
|
|
--log_level "info" \
|
|
--logging_strategy "steps" \
|
|
--eval_strategy "epoch" \
|
|
--save_strategy "epoch" \
|
|
--push_to_hub \
|
|
--hub_private_repo True \
|
|
--hub_strategy "every_save" \
|
|
--bf16 True \
|
|
--packing True \
|
|
--learning_rate 1e-4 \
|
|
--lr_scheduler_type "cosine" \
|
|
--weight_decay 1e-4 \
|
|
--warmup_steps 0 \
|
|
--max_grad_norm 1.0 \
|
|
--output_dir "mistral-sft-lora" \
|
|
--per_device_train_batch_size 8 \
|
|
--per_device_eval_batch_size 8 \
|
|
--gradient_accumulation_steps 8 \
|
|
--gradient_checkpointing True \
|
|
--use_reentrant True \
|
|
--dataset_text_field "content" \
|
|
--use_peft_lora True \
|
|
--lora_r 8 \
|
|
--lora_alpha 16 \
|
|
--lora_dropout 0.1 \
|
|
--lora_target_modules "all-linear" \
|
|
--use_4bit_quantization True \
|
|
--use_nested_quant True \
|
|
--bnb_4bit_compute_dtype "bfloat16" \
|
|
--use_flash_attn True
|