wehub-resource-sync
25 lines
626 B
Markdown
25 lines
626 B
Markdown
|
|
# Install dependency
|
|
```bash
|
|
pip install -r requirement.txt
|
|
```
|
|
|
|
# Download the data set
|
|
```bash
|
|
export WORKDIR_ROOT=<a directory which will hold all working files>
|
|
|
|
```
|
|
The downloaded data will be at $WORKDIR_ROOT/ML50
|
|
|
|
# preprocess the data
|
|
Install SPM [here](https://github.com/google/sentencepiece)
|
|
```bash
|
|
export WORKDIR_ROOT=<a directory which will hold all working files>
|
|
export SPM_PATH=<a path pointing to sentencepice spm_encode.py>
|
|
```
|
|
* $WORKDIR_ROOT/ML50/raw: extracted raw data
|
|
* $WORKDIR_ROOT/ML50/dedup: dedup data
|
|
* $WORKDIR_ROOT/ML50/clean: data with valid and test sentences removed from the dedup data
|
|
|
|
|