chore: import upstream snapshot with attribution

2026-07-13 13:24:13 +08:00
commit 1037506f2e
6050 changed files with 1731598 additions and 0 deletions
@@ -0,0 +1,178 @@
+# **Baseline Simultaneous Translation**
+---
+
+This is an instruction of training and evaluating a *wait-k* simultanoes LSTM model on MUST-C English-Gernam Dataset.
+
+[STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework](https://https://www.aclweb.org/anthology/P19-1289/)
+
+
+## **Requirements**
+Install fairseq (make sure to use the correct branch):
+```
+git clone --branch simulastsharedtask git@github.com:pytorch/fairseq.git
+cd fairseq
+pip install -e .
+```
+
+Assuming that fairseq is installed in a directory called `FAIRSEQ`.
+
+Install SentencePiece. One easy way is to use anaconda:
+
+```
+conda install -c powerai sentencepiece
+```
+
+Download the MuST-C data for English-German available at https://ict.fbk.eu/must-c/.
+We will assume that the data is downloaded in a directory called `DATA_ROOT`.
+
+
+## **Text-to-text Model**
+---
+### Data Preparation
+Train a SentencePiece model:
+```shell
+for lang in en de; do
+    python $FAIRSEQ/examples/simultaneous_translation/data/train_spm.py \
+        --data-path $DATA_ROOT/data \
+        --vocab-size 10000 \
+        --max-frame 3000 \
+        --model-type unigram \
+        --lang $lang \
+        --out-path .
+```
+
+Process the data with the SentencePiece model:
+```shell
+proc_dir=proc
+mkdir -p $proc_dir
+for split in train dev tst-COMMON tst-HE; do
+    for lang in en de; do
+        spm_encode \
+            --model unigram-$lang-10000-3000/spm.model \
+            < $DATA_ROOT/data/$split/txt/$split.$lang \
+            > $proc_dir/$split.spm.$lang
+    done
+done
+```
+
+Binarize the data:
+
+```shell
+proc_dir=proc
+fairseq-preprocess \
+    --source-lang en --target-lang de \
+    --trainpref $proc_dir/train.spm \
+    --validpref $proc_dir/dev.spm \
+    --testpref $proc_dir/tst-COMMON.spm \
+    --thresholdtgt 0 \
+    --thresholdsrc 0 \
+    --workers 20 \
+    --destdir ./data-bin/mustc_en_de \
+```
+
+### Training
+
+
+```shell
+mkdir -p checkpoints
+CUDA_VISIBLE_DEVICES=1 python $FAIRSEQ/train.py data-bin/mustc_en_de \
+    --save-dir checkpoints \
+    --arch berard_simul_text_iwslt \
+    --simul-type waitk \
+    --waitk-lagging 2 \
+    --optimizer adam \
+    --max-epoch 100 \
+    --lr 0.001 \
+    --clip-norm 5.0  \
+    --batch-size 128  \
+    --log-format json \
+    --log-interval 10 \
+    --criterion cross_entropy_acc \
+    --user-dir $FAIRSEQ/examples/simultaneous_translation
+```
+
+## **Speech-to-text Model**
+---
+### Data Preparation
+First, segment wav files.
+```shell 
+python $FAIRSEQ/examples/simultaneous_translation/data/segment_wav.py \
+    --datapath $DATA_ROOT
+```
+Similar to text-to-text model, train a Sentencepiecemodel, but only train on German
+```Shell
+python $FAIRSEQ/examples/simultaneous_translation/data/train_spm.py \
+    --data-path $DATA_ROOT/data \
+    --vocab-size 10000 \
+    --max-frame 3000 \
+    --model-type unigram \
+    --lang $lang \
+    --out-path .
+```
+## Training
+```shell
+mkdir -p checkpoints
+CUDA_VISIBLE_DEVICES=1 python $FAIRSEQ/train.py data-bin/mustc_en_de \
+    --save-dir checkpoints \
+    --arch berard_simul_text_iwslt \
+    --waitk-lagging 2 \
+    --waitk-stride 10 \
+    --input-feat-per-channel 40 \
+    --encoder-hidden-size 512 \
+    --output-layer-dim 128 \
+    --decoder-num-layers 3 \
+    --task speech_translation \
+    --user-dir $FAIRSEQ/examples/simultaneous_translation
+    --optimizer adam \
+    --max-epoch 100 \
+    --lr 0.001 \
+    --clip-norm 5.0  \
+    --batch-size 128  \
+    --log-format json \
+    --log-interval 10 \
+    --criterion cross_entropy_acc \
+    --user-dir $FAIRSEQ/examples/simultaneous_translation
+```
+
+## Evaluation
+---
+### Evaluation Server
+For text translation models, the server is set up as follow give input file and reference file. 
+
+``` shell
+python ./eval/server.py \
+    --hostname localhost \
+    --port 12321 \
+    --src-file $DATA_ROOT/data/dev/txt/dev.en \
+    --ref-file $DATA_ROOT/data/dev/txt/dev.de
+```
+For speech translation models, the input is the data direcrory.
+``` shell
+python ./eval/server.py \
+    --hostname localhost \
+    --port 12321 \
+    --ref-file $DATA_ROOT \
+    --data-type speech
+```
+
+### Decode and Evaluate with Client
+Once the server is set up, run client to evaluate translation quality and latency.
+```shell
+# TEXT
+python $fairseq_dir/examples/simultaneous_translation/evaluate.py \
+    data-bin/mustc_en_de \
+    --user-dir $FAIRSEQ/examples/simultaneous_translation \
+    --src-spm unigram-en-10000-3000/spm.model\
+    --tgt-spm unigram-de-10000-3000/spm.model\
+    -s en -t de \
+    --path checkpoints/checkpoint_best.pt
+
+# SPEECH
+python $fairseq_dir/examples/simultaneous_translation/evaluate.py \
+    data-bin/mustc_en_de \
+    --user-dir $FAIRSEQ/examples/simultaneous_translation \
+    --data-type speech \
+    --tgt-spm unigram-de-10000-3000/spm.model\
+    -s en -t de \
+    --path checkpoints/checkpoint_best.pt
+```
@@ -0,0 +1,115 @@
+# Introduction to evaluation interface
+The simultaneous translation models from sharedtask participents are evaluated under a server-client protocol. The participents are requisted to plug in their own model API in the protocol, and submit a docker file.
+
+## Server-Client Protocol
+An server-client protocol that will be used in evaluation. For example, when a *wait-k* model (k=3) translate the English sentence "Alice and Bob are good friends" to Genman sentence "Alice und Bob sind gute Freunde." , the evaluation process is shown as following figure. 
+
+While every time client needs to read a new state (word or speech utterence), a "GET" request is supposed to sent over to server. Whenever a new token is generated, a "SEND" request with the word predicted (untokenized word) will be sent to server immediately. The server can hence calculate both latency and BLEU score of the sentence.
+
+### Server
+The server code is provided and can be set up directly locally for development purpose. For example, to evaluate a text simultaneous test set,
+
+```shell
+
+  python fairseq/examples/simultaneous_translation/eval/server.py \
+    --hostname local_host  \
+    --port 1234 \
+    --src-file SRC_FILE \  
+    --ref-file REF_FILE  \  
+    --data-type text \
+```
+The state that server sent to client is has the following format
+```json
+{
+  'sent_id': Int,
+  'segment_id': Int,
+  'segment': String
+}
+```
+
+### Client
+The client will handle the evaluation process mentioned above. It should be out-of-box as well. The client's protocol is as following table
+
+|Action|Content|
+|:---:|:---:|
+|Request new word / utterence| ```{key: "Get", value: None}```|
+|Predict word "W"| ```{key: "SEND", value: "W"}```|
+
+
+
+The core of the client module is the agent, which needs to be modified to different models accordingly. The abstract class of agent is as follow, the evaluation process happens in the `decode()` function. 
+```python
+class Agent(object):
+    "an agent needs to follow this pattern"
+    def __init__(self, *args, **kwargs):
+        ...
+
+    def init_states(self):
+        # Initializing states
+        ...
+
+    def update_states(self, states, new_state):
+        # Update states with given new state from server
+        # TODO (describe the states)
+        ...
+
+    def finish_eval(self, states, new_state):
+        # Check if evaluation is finished
+        ...
+    
+    def policy(self, state: list) -> dict:
+        # Provide a action given current states
+        # The action can only be either
+        # {key: "GET", value: NONE} 
+        # or
+        # {key: "SEND", value: W}
+        ...
+
+    def reset(self):
+        # Reset agent
+        ...
+        
+    def decode(self, session):
+        
+        states = self.init_states()
+        self.reset()      
+
+        # Evaluataion protocol happens here
+        while True:
+            # Get action from the current states according to self.policy()
+            action = self.policy(states)
+
+            if action['key'] == GET:
+                # Read a new state from server
+                new_state = session.get_src()
+                states = self.update_states(states, new_state)
+
+                if self.finish_eval(states, new_state):
+                    # End of document
+                    break 
+                
+            elif action['key'] == SEND:
+                # Send a new prediction to server
+                session.send_hypo(action['value'])
+                
+                # Clean the history, wait for next sentence
+                if action['value'] == DEFAULT_EOS:
+                    states = self.init_states() 
+                    self.reset()
+            else:
+                raise NotImplementedError
+
+ 
+```
+Here an implementation of agent of text [*wait-k* model](somelink). Notice that the tokenization is not considered.
+
+## Quality
+The quality is measured by detokenized BLEU. So make sure that the predicted words sent to server are detokenized. An implementation is can be find [here](some link)
+
+## Latency
+The latency metrics are 
+* Average Proportion
+* Average Lagging
+* Differentiable Average Lagging
+Again Thery will also be evaluated on detokenized text.
+