wehub-resource-sync
80 lines
2.5 KiB
Markdown
80 lines
2.5 KiB
Markdown
# DiT for Image Classification
|
|
|
|
This folder contains the image classification running instructions on DiT for RVL-CDIP.
|
|
|
|
## Usage
|
|
### Data Preparation
|
|
|
|
**RVL-CDIP**
|
|
|
|
Download the "rvl-cdip.tar.gz" from this [link](https://www.cs.ryerson.ca/~aharley/rvl-cdip/) (~37GB). Then extract it to `PATH-to-rvlcdip`.
|
|
|
|
### Evaluation
|
|
Following commands provide example to evaluate the fine-tuned checkpoints.
|
|
```bash
|
|
python -m torch.distributed.launch --nproc_per_node=8 --master_port=47770 run_class_finetuning.py \
|
|
--model beit_base_patch16_224 #beit_base_patch16_224 / beit_large_patch16_224
|
|
--data_path "/path/to/rvlcdip"
|
|
--eval_data_path "/path/to/rvlcdip"
|
|
--enable_deepspeed
|
|
--nb_classes 16
|
|
--eval
|
|
--data_set rvlcdip
|
|
--finetune /path/to/model.pth
|
|
--output_dir output_dir
|
|
--log_dir output_dir/tf
|
|
--batch_size 256
|
|
--abs_pos_emb
|
|
--disable_rel_pos_bias
|
|
```
|
|
|
|
### Training
|
|
Fine-tune DiT on RVL-CDIP:
|
|
```bash
|
|
exp_name=dit-base-exp
|
|
|
|
mkdir -p output/${exp_name}
|
|
python -m torch.distributed.launch --nproc_per_node=8 run_class_finetuning.py
|
|
--model beit_base_patch16_224 #beit_base_patch16_224 / beit_large_patch16_224
|
|
--data_path "/path/to/rvlcdip"
|
|
--eval_data_path "/path/to/rvlcdip"
|
|
--nb_classes 16
|
|
--data_set rvlcdip
|
|
--finetune /path/to/model.pth
|
|
--output_dir output/${exp_name}/
|
|
--log_dir output/${exp_name}/tf
|
|
--batch_size 64
|
|
--lr 5e-4
|
|
--update_freq 2
|
|
--eval_freq 10
|
|
--save_ckpt_freq 10
|
|
--warmup_epochs 20
|
|
--epochs 180
|
|
--layer_scale_init_value 1e-5
|
|
--layer_decay 0.75
|
|
--drop_path 0.2
|
|
--weight_decay 0.05
|
|
--clip_grad 1.0
|
|
--abs_pos_emb
|
|
--disable_rel_pos_bias
|
|
```
|
|
|
|
|
|
## Citation
|
|
|
|
If you find this repository useful, please consider citing our work:
|
|
```
|
|
@misc{li2022dit,
|
|
title={DiT: Self-supervised Pre-training for Document Image Transformer},
|
|
author={Junlong Li and Yiheng Xu and Tengchao Lv and Lei Cui and Cha Zhang and Furu Wei},
|
|
year={2022},
|
|
eprint={2203.02378},
|
|
archivePrefix={arXiv},
|
|
primaryClass={cs.CV}
|
|
}
|
|
```
|
|
|
|
|
|
## Acknowledgment
|
|
This part is built using the [timm](https://github.com/rwightman/pytorch-image-models) library, the [Beit](https://github.com/microsoft/unilm/tree/master/beit) repository, the [DeiT](https://github.com/facebookresearch/deit) repository and the [Dino](https://github.com/facebookresearch/dino) repository.
|