chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,138 @@
|
||||
# Chinese Text Normalization
|
||||
|
||||
## 1. How To Use
|
||||
```
|
||||
python normalize.py --language "zh" --text "text to be normalized"
|
||||
```
|
||||
|
||||
## 2. TN Pipeline
|
||||
There are 3 components in TN pipeline:
|
||||
* pre-processing (before tagger)
|
||||
* non-standard word normalization
|
||||
* post-processing (after verbalizer)
|
||||
|
||||
### 2.1 Pre-Processing
|
||||
#### Char Width Conversion (全角 -> 半角)
|
||||
```
|
||||
苹果ＣＥＯ宣布发布新ＩＰＨＯＮＥ -> 苹果CEO宣布发布新IPHONE
|
||||
他说：“我们已经吃过了！”。 -> 他说:"我们已经吃过了!".
|
||||
```
|
||||
* covers English letters, digits, punctuations and some symbols
|
||||
* the complete mapping table `data/char/fullwidth_to_halfwidth.tsv`
|
||||
|
||||
#### Denylist (Removal)
|
||||
Sometime you may want to remove certain things like interjections/fillers "啊", "呃" etc
|
||||
```
|
||||
呃这个呃啊我不知道 -> 这个我不知道
|
||||
```
|
||||
* customizable via `data/denylist/denylist.tsv`
|
||||
|
||||
|
||||
### 2.2 Non-Standard-Words(NSW) normalization
|
||||
#### Numbers
|
||||
```
|
||||
共465篇，约315万字 -> 共四百六十五篇，约三百一十五万字
|
||||
共计6.42万人 -> 共计六点四二万人
|
||||
同比升高0.6个百分点 -> 同比升高零点六个百分点
|
||||
```
|
||||
|
||||
#### Fraction
|
||||
```
|
||||
总量的1/5以上 -> 总量的五分之一以上
|
||||
相当于头发丝的1/16 -> 相当于头发丝的十六分之一
|
||||
3/2是一个假分数 -> 二分之三是一个假分数
|
||||
```
|
||||
|
||||
#### Percentage
|
||||
```
|
||||
同比增长6.3% -> 同比增长百分之六点三
|
||||
增幅0.4% -> 增幅百分之零点四
|
||||
```
|
||||
|
||||
#### Date
|
||||
```
|
||||
2002/01/28 -> 二零零二年一月二十八日
|
||||
2002-01-28 -> 二零零二年一月二十八日
|
||||
2002.01.28 -> 二零零二年一月二十八日
|
||||
2002/01 -> 二零零二年一月
|
||||
```
|
||||
|
||||
#### Time
|
||||
```
|
||||
8月16号12:00之前 -> 八月十六号十二点之前
|
||||
我是5:02开始的 -> 我是五点零二分开始的
|
||||
于5:35:36发射 -> 于五点三十五分三十六秒发射
|
||||
8:00am准时开会 -> 上午八点准时开会
|
||||
```
|
||||
|
||||
#### Math
|
||||
```
|
||||
比分定格在78:96 -> 比分定格在七十八比九十六
|
||||
计算-2的绝对值是2 -> 计算负二的绝对值是二
|
||||
±2的平方都是4 -> 正负二的平方都是四
|
||||
```
|
||||
|
||||
#### Money
|
||||
```
|
||||
价格是￥13.5 -> 价格是十三点五元
|
||||
价格是$13.5 -> 价格是十三点五美元
|
||||
价格是A$13.5 -> 价格是十三点五澳元
|
||||
价格是HKD13.5 -> 价格是十三点五港元
|
||||
```
|
||||
|
||||
#### Measure
|
||||
```
|
||||
重达25kg -> 二十五千克
|
||||
最高气温38°C -> 最高气温三十八摄氏度
|
||||
实际面积120m² -> 实际面积一百二十平方米
|
||||
渲染速度10ms一帧 -> 渲染速度十毫秒一帧
|
||||
```
|
||||
|
||||
#### Number series (phone, mobile numbers)
|
||||
```
|
||||
可以打我手机13501234567 -> 可以打我手机一三五零一二三四五六七
|
||||
可以拨打12306来咨询 -> 可以拨打一二三零六来咨询
|
||||
```
|
||||
|
||||
#### Erhua(儿化音) Removal
|
||||
```
|
||||
这儿有只鸟儿 -> 这有只鸟
|
||||
这事儿好办 -> 这事好办
|
||||
我儿子喜欢这地儿 -> 我儿子喜欢这地
|
||||
```
|
||||
* erhua whitelist is customizable via `data/erhua/whitelist.tsv`
|
||||
|
||||
#### Whitelist(Replacement)
|
||||
a set of user-defined hard mapping, i.e. exact-string matching & replacement
|
||||
```
|
||||
C E O -> CEO
|
||||
G P U -> GPU
|
||||
O2O -> O to O
|
||||
B2B -> B to B
|
||||
```
|
||||
* customizable via `data/whitelist/default.tsv`
|
||||
|
||||
### 2.3 Post-Processing
|
||||
#### Punctuation Removal
|
||||
If enabled, punctuations are removed.
|
||||
|
||||
#### Uppercase or Lowercase Conversion
|
||||
If enabled, English letters are converted to uppercases / lowercases
|
||||
|
||||
#### Out-Of-Vocabulary(OOV) Tagger
|
||||
If enabled, OOV chars are tagged with `<oov>` and `</oov>`, e.g.:
|
||||
```
|
||||
我们안녕 -> 我们<oov>안</oov><oov>녕</oov>
|
||||
雪の花 -> 雪<oov>の</oov>花
|
||||
```
|
||||
* default charset (national standard) [通用规范汉字表](https://zh.wikipedia.org/wiki/通用规范汉字表)
|
||||
* you can extend charset via `data/char/charset_extension.tsv`
|
||||
|
||||
## 3. Credits
|
||||
Author: Zhenxiang MA @ Tsinghua University
|
||||
|
||||
Advisors: [SpeechColab](https://github.com/SpeechColab) organization
|
||||
|
||||
The authors of this work would like to thank:
|
||||
* The authors of foundational libraries like OpenFst & Pynini
|
||||
* NeMo team and NeMo open-source community
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
吶
|
||||
诶
|
||||
屌
|
||||
囧
|
||||
飚
|
||||
屄
|
||||
|
+8105
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,92 @@
|
||||
！ !
|
||||
＂ "
|
||||
＃ #
|
||||
＄ $
|
||||
％ %
|
||||
＆ &
|
||||
＇ '
|
||||
（ (
|
||||
） )
|
||||
＊ *
|
||||
＋ +
|
||||
， ,
|
||||
－ -
|
||||
． .
|
||||
／ /
|
||||
０ 0
|
||||
１ 1
|
||||
２ 2
|
||||
３ 3
|
||||
４ 4
|
||||
５ 5
|
||||
６ 6
|
||||
７ 7
|
||||
８ 8
|
||||
９ 9
|
||||
； ;
|
||||
＜ <
|
||||
＝ =
|
||||
＞ >
|
||||
？ ?
|
||||
＠ @
|
||||
Ａ A
|
||||
Ｂ B
|
||||
Ｃ C
|
||||
Ｄ D
|
||||
Ｅ E
|
||||
Ｆ F
|
||||
Ｇ G
|
||||
Ｈ H
|
||||
Ｉ I
|
||||
Ｊ J
|
||||
Ｋ K
|
||||
Ｌ L
|
||||
Ｍ M
|
||||
Ｎ N
|
||||
Ｏ O
|
||||
Ｐ P
|
||||
Ｑ Q
|
||||
Ｒ R
|
||||
Ｓ S
|
||||
Ｔ T
|
||||
Ｕ U
|
||||
Ｖ V
|
||||
Ｗ W
|
||||
Ｘ X
|
||||
Ｙ Y
|
||||
Ｚ Z
|
||||
＼ \
|
||||
＾ ^
|
||||
＿ _
|
||||
｀ `
|
||||
ａ a
|
||||
ｂ b
|
||||
ｃ c
|
||||
ｄ d
|
||||
ｅ e
|
||||
ｆ f
|
||||
ｇ g
|
||||
ｈ h
|
||||
ｉ i
|
||||
ｊ j
|
||||
ｋ k
|
||||
ｌ l
|
||||
ｍ m
|
||||
ｎ n
|
||||
ｏ o
|
||||
ｐ p
|
||||
ｑ q
|
||||
ｒ r
|
||||
ｓ s
|
||||
ｔ t
|
||||
ｕ u
|
||||
ｖ v
|
||||
ｗ w
|
||||
ｘ x
|
||||
ｙ y
|
||||
ｚ z
|
||||
｛ {
|
||||
｜ |
|
||||
： :
|
||||
｝ }
|
||||
～ ~
|
||||
|
Can't render this file because it contains an unexpected character in line 92 and column 7.
|
@@ -0,0 +1 @@
|
||||
<oov> </oov>
|
||||
|
@@ -0,0 +1,72 @@
|
||||
！
|
||||
？
|
||||
｡
|
||||
。
|
||||
＂
|
||||
＃
|
||||
＄
|
||||
％
|
||||
＆
|
||||
＇
|
||||
（
|
||||
）
|
||||
＊
|
||||
＋
|
||||
，
|
||||
－
|
||||
／
|
||||
：
|
||||
；
|
||||
＜
|
||||
＝
|
||||
＞
|
||||
＠
|
||||
［
|
||||
＼
|
||||
］
|
||||
＾
|
||||
＿
|
||||
｀
|
||||
｛
|
||||
｜
|
||||
｝
|
||||
～
|
||||
｟
|
||||
｠
|
||||
｢
|
||||
｣
|
||||
､
|
||||
、
|
||||
〃
|
||||
》
|
||||
「
|
||||
」
|
||||
『
|
||||
』
|
||||
【
|
||||
】
|
||||
〔
|
||||
〕
|
||||
〖
|
||||
〗
|
||||
〘
|
||||
〙
|
||||
〚
|
||||
〛
|
||||
〜
|
||||
〝
|
||||
〞
|
||||
〟
|
||||
〰
|
||||
–
|
||||
—
|
||||
‘
|
||||
’
|
||||
‛
|
||||
“
|
||||
”
|
||||
„
|
||||
‟
|
||||
…
|
||||
‧
|
||||
﹏
|
||||
|
@@ -0,0 +1,9 @@
|
||||
壹 一
|
||||
贰 二
|
||||
叁 三
|
||||
肆 四
|
||||
伍 五
|
||||
陆 六
|
||||
柒 七
|
||||
捌 八
|
||||
玖 九
|
||||
|
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
来
|
||||
前
|
||||
后
|
||||
内
|
||||
之后
|
||||
|
||||
|
@@ -0,0 +1,2 @@
|
||||
啊
|
||||
呃
|
||||
|
@@ -0,0 +1,35 @@
|
||||
儿女
|
||||
儿子
|
||||
儿孙
|
||||
女儿
|
||||
儿媳
|
||||
妻儿
|
||||
胎儿
|
||||
婴儿
|
||||
新生儿
|
||||
婴幼儿
|
||||
幼儿
|
||||
少儿
|
||||
小儿
|
||||
儿歌
|
||||
儿童
|
||||
儿科
|
||||
托儿所
|
||||
孤儿
|
||||
儿戏
|
||||
儿化
|
||||
台儿庄
|
||||
鹿儿岛
|
||||
正儿八经
|
||||
吊儿郎当
|
||||
生儿育女
|
||||
托儿带女
|
||||
养儿防老
|
||||
痴儿呆女
|
||||
佳儿佳妇
|
||||
儿怜兽扰
|
||||
儿无常父
|
||||
儿不嫌母丑
|
||||
儿行千里母担忧
|
||||
儿大不由爷
|
||||
苏乞儿
|
||||
|
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
: 比
|
||||
： 比
|
||||
|
@@ -0,0 +1,7 @@
|
||||
+ 加
|
||||
~ 到
|
||||
± 正负
|
||||
= 等于
|
||||
× 乘
|
||||
÷ 除
|
||||
° 度
|
||||
|
@@ -0,0 +1,90 @@
|
||||
amu 原子质量
|
||||
bar 巴
|
||||
° 度
|
||||
º 度
|
||||
°c 摄氏度
|
||||
°C 摄氏度
|
||||
ºc 摄氏度
|
||||
ºC 摄氏度
|
||||
℃ 摄氏度
|
||||
cm2 平方厘米
|
||||
cm² 平方厘米
|
||||
cm3 立方厘米
|
||||
cm³ 立方厘米
|
||||
cm 厘米
|
||||
cwt 美担
|
||||
db 分贝
|
||||
dm3 立方分米
|
||||
dm³ 立方分米
|
||||
dm 分米
|
||||
ds 毫秒
|
||||
°f 华氏度
|
||||
°F 华氏度
|
||||
℉ 华氏度
|
||||
ft 英尺
|
||||
ghz 吉赫兹
|
||||
gw 吉瓦
|
||||
gwh 吉瓦时
|
||||
hz 赫兹
|
||||
kbps 千比特每秒
|
||||
kcal 千卡
|
||||
kgf 千克力
|
||||
kg 千克
|
||||
khz 千赫兹
|
||||
km2 平方千米
|
||||
km² 平方千米
|
||||
km 千米
|
||||
kpa 千帕
|
||||
kwh 千瓦时
|
||||
kw 千瓦
|
||||
kW 千瓦
|
||||
lb 磅
|
||||
lbs 磅
|
||||
m2 平方米
|
||||
m² 平方米
|
||||
m3 立方米
|
||||
m³ 立方米
|
||||
mbps 兆比特每秒
|
||||
mg 毫克
|
||||
mhz 兆赫兹
|
||||
mi2 平方英里
|
||||
mi² 平方英里
|
||||
mi 英里
|
||||
min 分钟哦
|
||||
ml 毫升
|
||||
mm2 平方毫米
|
||||
mm² 平方毫米
|
||||
mol 摩尔
|
||||
mpa 兆帕
|
||||
mph 英里每小时
|
||||
ng 纳克
|
||||
nm 纳米
|
||||
ns 纳秒
|
||||
oz 盎司
|
||||
pa 帕斯卡
|
||||
rad 弧度
|
||||
rpm 转每分
|
||||
sq ft 平方英尺
|
||||
sq mi 平方英里
|
||||
sv 系沃特
|
||||
tb 太字节
|
||||
tj 万亿焦耳
|
||||
tl 台两
|
||||
v 伏特
|
||||
yd 码
|
||||
μg 微克
|
||||
μm 微米
|
||||
μs 微秒
|
||||
ω 欧米茄
|
||||
gb 吉字节
|
||||
gpa 吉帕斯卡
|
||||
gy 戈瑞
|
||||
ha 公顷
|
||||
m 米
|
||||
mm 毫米
|
||||
ms 毫秒
|
||||
mv 毫伏
|
||||
mw 毫瓦
|
||||
pg 皮克
|
||||
ps 皮秒
|
||||
s 秒
|
||||
|
@@ -0,0 +1,211 @@
|
||||
匹
|
||||
张
|
||||
座
|
||||
回
|
||||
场
|
||||
尾
|
||||
条
|
||||
个
|
||||
首
|
||||
阙
|
||||
阵
|
||||
网
|
||||
炮
|
||||
顶
|
||||
丘
|
||||
棵
|
||||
只
|
||||
支
|
||||
袭
|
||||
辆
|
||||
挑
|
||||
担
|
||||
颗
|
||||
壳
|
||||
窠
|
||||
曲
|
||||
墙
|
||||
群
|
||||
腔
|
||||
砣
|
||||
座
|
||||
客
|
||||
贯
|
||||
扎
|
||||
捆
|
||||
刀
|
||||
令
|
||||
打
|
||||
手
|
||||
罗
|
||||
坡
|
||||
山
|
||||
岭
|
||||
江
|
||||
溪
|
||||
钟
|
||||
队
|
||||
单
|
||||
双
|
||||
对
|
||||
口
|
||||
头
|
||||
脚
|
||||
板
|
||||
跳
|
||||
枝
|
||||
件
|
||||
贴
|
||||
针
|
||||
线
|
||||
管
|
||||
名
|
||||
位
|
||||
身
|
||||
堂
|
||||
课
|
||||
本
|
||||
页
|
||||
家
|
||||
户
|
||||
层
|
||||
丝
|
||||
毫
|
||||
厘
|
||||
分
|
||||
钱
|
||||
两
|
||||
斤
|
||||
担
|
||||
铢
|
||||
石
|
||||
钧
|
||||
锱
|
||||
忽
|
||||
克
|
||||
毫
|
||||
厘
|
||||
分
|
||||
寸
|
||||
尺
|
||||
丈
|
||||
里
|
||||
寻
|
||||
常
|
||||
铺
|
||||
程
|
||||
米
|
||||
撮
|
||||
勺
|
||||
合
|
||||
升
|
||||
斗
|
||||
石
|
||||
盘
|
||||
碗
|
||||
碟
|
||||
叠
|
||||
桶
|
||||
笼
|
||||
盆
|
||||
盒
|
||||
杯
|
||||
钟
|
||||
斛
|
||||
锅
|
||||
簋
|
||||
篮
|
||||
盘
|
||||
桶
|
||||
罐
|
||||
瓶
|
||||
壶
|
||||
卮
|
||||
盏
|
||||
箩
|
||||
箱
|
||||
煲
|
||||
啖
|
||||
袋
|
||||
钵
|
||||
年
|
||||
月
|
||||
日
|
||||
季
|
||||
刻
|
||||
时
|
||||
周
|
||||
天
|
||||
秒
|
||||
分
|
||||
旬
|
||||
纪
|
||||
岁
|
||||
世
|
||||
更
|
||||
夜
|
||||
春
|
||||
夏
|
||||
秋
|
||||
冬
|
||||
代
|
||||
伏
|
||||
辈
|
||||
丸
|
||||
泡
|
||||
粒
|
||||
颗
|
||||
幢
|
||||
堆
|
||||
条
|
||||
根
|
||||
支
|
||||
道
|
||||
面
|
||||
片
|
||||
张
|
||||
颗
|
||||
块
|
||||
架
|
||||
千米
|
||||
分米
|
||||
厘米
|
||||
毫米
|
||||
微米
|
||||
纳米
|
||||
亿
|
||||
千万
|
||||
百万
|
||||
万
|
||||
千
|
||||
百
|
||||
亿块
|
||||
千万块
|
||||
百万块
|
||||
万块
|
||||
千块
|
||||
百块
|
||||
亿角
|
||||
千万角
|
||||
百万角
|
||||
万角
|
||||
千角
|
||||
百角
|
||||
亿毛
|
||||
千万毛
|
||||
百万毛
|
||||
万毛
|
||||
千毛
|
||||
百毛
|
||||
亿分
|
||||
千万分
|
||||
百万分
|
||||
万分
|
||||
千分
|
||||
百分
|
||||
亿元
|
||||
千万元
|
||||
百万元
|
||||
万元
|
||||
千元
|
||||
百元
|
||||
|
@@ -0,0 +1,117 @@
|
||||
ALL 阿尔巴尼亚列克
|
||||
AFN 阿富汗 阿富汗尼
|
||||
ARS 阿根廷比索
|
||||
AWG 阿鲁巴盾
|
||||
AUD 澳元
|
||||
AZN 阿塞拜疆马纳特
|
||||
BSD 巴哈马元
|
||||
BBD 巴巴多斯元
|
||||
BYN 白俄罗斯卢布
|
||||
BZD 伯利兹元
|
||||
BMD 百慕大元
|
||||
BOB 玻利维亚玻利维亚诺
|
||||
BAM 波斯尼亚和黑塞哥维那可兑换马克
|
||||
BWP 博茨瓦纳普拉
|
||||
BGN 保加利亚列弗
|
||||
BRL 巴西雷亚尔
|
||||
BND 文莱达鲁萨兰国元
|
||||
KHR 柬埔寨瑞尔
|
||||
CAD 加元
|
||||
KYD 开曼群岛元
|
||||
CLP 智利比索
|
||||
CNY 人民币
|
||||
COP 哥伦比亚比索
|
||||
CRC 哥斯达黎加科隆
|
||||
HRK 克罗地亚库纳
|
||||
CUP 古巴比索
|
||||
CZK 捷克克朗
|
||||
DKK 丹麦克朗
|
||||
DOP 多米尼加共和国比索
|
||||
XCD 东加勒比元
|
||||
EGP 埃及镑
|
||||
SVC 萨尔瓦多科隆
|
||||
EUR 欧元成员国
|
||||
FKP 福克兰群岛（马尔维纳斯）镑
|
||||
FJD 斐济元
|
||||
GHS 加纳塞地
|
||||
GIP 直布罗陀镑
|
||||
GTQ 危地马拉格查尔
|
||||
GGP 根西岛镑
|
||||
GYD 圭亚那元
|
||||
HNL 洪都拉斯伦皮拉
|
||||
HKD 港元
|
||||
HUF 匈牙利福林
|
||||
ISK 冰岛克朗
|
||||
INR 印度卢比
|
||||
IDR 印尼盾
|
||||
IRR 伊朗里亚尔
|
||||
IMP 马恩岛英镑
|
||||
ILS 以色列谢克尔
|
||||
JMD 牙买加元
|
||||
JPY 日元
|
||||
JEP 泽西镑
|
||||
KZT 哈萨克斯坦腾格
|
||||
KPW 朝鲜园
|
||||
KRW 韩元
|
||||
KGS 吉尔吉斯斯坦索姆
|
||||
LAK 老挝基普
|
||||
LBP 黎巴嫩镑
|
||||
LRD 利比里亚元
|
||||
MKD 马其顿代纳尔
|
||||
MYR 马来西亚令吉
|
||||
MUR 毛里求斯卢比
|
||||
MXN 墨西哥比索
|
||||
MNT 蒙古图格里克
|
||||
MNT 摩洛哥迪拉姆
|
||||
MZN 莫桑比克梅蒂卡尔
|
||||
NAD 纳米比亚元
|
||||
NPR 尼泊尔卢比
|
||||
ANG 荷属安的列斯盾
|
||||
NZD 新西兰元
|
||||
NIO 尼加拉瓜科尔多瓦
|
||||
NGN 尼日利亚奈拉
|
||||
NOK 挪威克朗
|
||||
OMR 阿曼里亚尔
|
||||
PKR 巴基斯坦卢比
|
||||
PAB 巴拿马巴尔博亚
|
||||
PYG 巴拉圭瓜拉尼
|
||||
PEN 秘鲁索尔
|
||||
PHP 菲律宾比索
|
||||
PLN 波兰兹罗提
|
||||
QAR 卡塔尔里亚尔
|
||||
RON 罗马尼亚列伊
|
||||
RUB 俄罗斯卢布
|
||||
SHP 圣赫勒拿镑
|
||||
SAR 沙特阿拉伯里亚尔
|
||||
RSD 塞尔维亚第纳尔
|
||||
SCR 塞舌尔卢比
|
||||
SGD 新加坡元
|
||||
SBD 所罗门群岛元
|
||||
SOS 索马里先令
|
||||
KRW 韩元
|
||||
ZAR 南非兰特
|
||||
LKR 斯里兰卡卢比
|
||||
SEK 瑞典克朗
|
||||
CHF 瑞士法郎
|
||||
SRD 苏里南元
|
||||
SYP 叙利亚镑
|
||||
TWD 新台币
|
||||
THB 泰铢
|
||||
TTD 特立尼达和多巴哥元
|
||||
TRY 土耳其里拉
|
||||
TVD 图瓦卢元
|
||||
UAH 乌克兰格里夫纳
|
||||
AED 阿联酋迪拉姆
|
||||
GBP 英镑
|
||||
USD 美元
|
||||
UYU 乌拉圭比索
|
||||
UZS 乌兹别克斯坦索姆
|
||||
VEF 委内瑞拉玻利瓦尔
|
||||
VND 越南东
|
||||
YER 也门里亚尔
|
||||
ZWD 津巴布韦元
|
||||
J￥ 日元
|
||||
JPY￥ 日元
|
||||
HK＄ 港元
|
||||
A$ 澳元
|
||||
CAD$ 加元
|
||||
|
@@ -0,0 +1,63 @@
|
||||
Lek 阿尔巴尼亚列克
|
||||
ƒ 阿鲁巴盾
|
||||
Br 白俄罗斯卢布
|
||||
BZ$ 伯利兹元
|
||||
$b 玻利维亚玻利维亚诺
|
||||
KM 波斯尼亚和黑塞哥维那可兑换马克
|
||||
P 博茨瓦纳普拉
|
||||
лв 保加利亚列弗
|
||||
R$ 巴西雷亚尔
|
||||
៛ 柬埔寨瑞尔
|
||||
¥ 人民币
|
||||
₡ 哥斯达黎加科隆
|
||||
kn 克罗地亚库纳
|
||||
₱ 古巴比索
|
||||
Kč 捷克克朗
|
||||
kr 丹麦克朗
|
||||
RD$ 多米尼加共和国比索
|
||||
€ 欧元
|
||||
¢ 加纳塞地
|
||||
Q 危地马拉格查尔
|
||||
L 洪都拉斯伦皮拉
|
||||
Ft 匈牙利福林
|
||||
₹ 印度卢比
|
||||
Rp 印尼盾
|
||||
£ 英镑
|
||||
￡ 英镑
|
||||
₪ 以色列谢克尔
|
||||
J$ 牙买加元
|
||||
лв 哈萨克斯坦腾格
|
||||
₩ 朝鲜园
|
||||
лв 吉尔吉斯斯坦索姆
|
||||
₭ 老挝基普
|
||||
ден 马其顿代纳尔
|
||||
RM 马来西亚令吉
|
||||
₨ 毛里求斯卢比
|
||||
₮ 蒙古图格里克
|
||||
MT 莫桑比克梅蒂卡尔
|
||||
C$ 尼加拉瓜科尔多瓦
|
||||
₦ 尼日利亚奈拉
|
||||
₨ 巴基斯坦卢比
|
||||
B/. 巴拿马巴尔博亚
|
||||
Gs 巴拉圭瓜拉尼
|
||||
S/. 秘鲁索尔
|
||||
₱ 菲律宾比索
|
||||
zł 波兰兹罗提
|
||||
lei 罗马尼亚列伊
|
||||
₽ 卢布
|
||||
Дин. 塞尔维亚第纳尔
|
||||
S 索马里先令
|
||||
R 南非兰特
|
||||
CHF 瑞士法郎
|
||||
NT$ 新台币
|
||||
฿ 泰铢
|
||||
TT$ 特立尼达和多巴哥元
|
||||
₺ 土耳其里拉
|
||||
₴ 乌克兰格里夫纳
|
||||
$ 美元
|
||||
$U 乌拉圭比索
|
||||
лв 乌兹别克斯坦索姆
|
||||
Bs 委内瑞拉玻利瓦尔
|
||||
₫ 越南东
|
||||
Z$ 津巴布韦元
|
||||
￥ 人民币
|
||||
|
@@ -0,0 +1,9 @@
|
||||
1 一
|
||||
2 二
|
||||
3 三
|
||||
4 四
|
||||
5 五
|
||||
6 六
|
||||
7 七
|
||||
8 八
|
||||
9 九
|
||||
|
@@ -0,0 +1,9 @@
|
||||
1
|
||||
2 二
|
||||
3 三
|
||||
4 四
|
||||
5 五
|
||||
6 六
|
||||
7 七
|
||||
8 八
|
||||
9 九
|
||||
|
@@ -0,0 +1,2 @@
|
||||
- 负
|
||||
+ 正
|
||||
|
@@ -0,0 +1 @@
|
||||
0 零
|
||||
|
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
|
||||
|
@@ -0,0 +1,24 @@
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
11
|
||||
12
|
||||
13
|
||||
14
|
||||
15
|
||||
16
|
||||
17
|
||||
18
|
||||
19
|
||||
20
|
||||
21
|
||||
22
|
||||
23
|
||||
24
|
||||
|
@@ -0,0 +1,10 @@
|
||||
am 上午
|
||||
a.m. 上午
|
||||
a m 上午
|
||||
AM 上午
|
||||
A M 上午
|
||||
pm 下午
|
||||
p.m. 下午
|
||||
p m 下午
|
||||
P M 下午
|
||||
PM 下午
|
||||
|
@@ -0,0 +1,7 @@
|
||||
0
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
|
@@ -0,0 +1 @@
|
||||
0
|
||||
|
@@ -0,0 +1,79 @@
|
||||
O 2 O O to O
|
||||
O2O O to O
|
||||
P 2 P P to P
|
||||
P2P P to P
|
||||
CD C D
|
||||
a.m. AM
|
||||
am. AM
|
||||
A M AM
|
||||
Am AM
|
||||
am AM
|
||||
p.m. PM
|
||||
pm. PM
|
||||
Pm PM
|
||||
pm PM
|
||||
atm ATM
|
||||
A T M ATM
|
||||
Atm ATM
|
||||
ufo UFO
|
||||
Ufo UFO
|
||||
U F O UFO
|
||||
u f o UFO
|
||||
m v MV
|
||||
mv MV
|
||||
MV MV
|
||||
F I F A fifa
|
||||
f i f a fifa
|
||||
FIFA fifa
|
||||
nba NBA
|
||||
Nba NBA
|
||||
NBA NBA
|
||||
n b a NBA
|
||||
x L XL
|
||||
X L XL
|
||||
x x L XXL
|
||||
X X L XXL
|
||||
x x x L XXXL
|
||||
X X X L XXXL
|
||||
Q Q QQ
|
||||
qq QQ
|
||||
Qq QQ
|
||||
gpu GPU
|
||||
GPU GPU
|
||||
cpu CPU
|
||||
C P U CPU
|
||||
WIFI wifi
|
||||
ms MS
|
||||
ceo CEO
|
||||
c e o CEO
|
||||
C E O CEO
|
||||
dv DV
|
||||
d v DV
|
||||
D V DV
|
||||
wto WTO
|
||||
w t o WTO
|
||||
W T O WTO
|
||||
pc PC
|
||||
p c PC
|
||||
P C PC
|
||||
gps GPS
|
||||
G P S GPS
|
||||
M.V.P MVP
|
||||
mvp MVP
|
||||
M V P MVP
|
||||
vip VIP
|
||||
V I P VIP
|
||||
cba CBA
|
||||
c b a CBA
|
||||
CBA CBA
|
||||
mba MBA
|
||||
m b a MBA
|
||||
M B A MBA
|
||||
iq IQ
|
||||
IQ IQ
|
||||
EQ EQ
|
||||
cctv CCTV
|
||||
C C t v CCTV
|
||||
kfc KFC
|
||||
K F C KFC
|
||||
Steam steam
|
||||
|
@@ -0,0 +1,117 @@
|
||||
import os
|
||||
import string
|
||||
from pathlib import Path
|
||||
from typing import Dict
|
||||
|
||||
import pynini
|
||||
from pynini import Far
|
||||
from pynini.export import export
|
||||
from pynini.lib import byte, pynutil, utf8
|
||||
|
||||
FUN_CHAR = utf8.VALID_UTF8_CHAR
|
||||
|
||||
FUN_DIGIT = byte.DIGIT
|
||||
FUN_LOWER = pynini.union(*string.ascii_lowercase).optimize()
|
||||
FUN_UPPER = pynini.union(*string.ascii_uppercase).optimize()
|
||||
FUN_ALPHA = pynini.union(FUN_LOWER, FUN_UPPER).optimize()
|
||||
|
||||
FUN_SPACE = " "
|
||||
FUN_WHITE_SPACE = pynini.union(" ", "\t", "\n", "\r", "\u00A0").optimize()
|
||||
FUN_NOT_SPACE = pynini.difference(FUN_CHAR, FUN_WHITE_SPACE).optimize()
|
||||
FUN_NOT_QUOTE = pynini.difference(FUN_CHAR, r'"').optimize()
|
||||
|
||||
FUN_PUNCT = pynini.union(*map(pynini.escape, string.punctuation)).optimize()
|
||||
|
||||
|
||||
FUN_SIGMA = pynini.closure(FUN_CHAR)
|
||||
|
||||
delete_space = pynutil.delete(pynini.closure(FUN_WHITE_SPACE))
|
||||
delete_zero_or_one_space = pynutil.delete(pynini.closure(FUN_WHITE_SPACE, 0, 1))
|
||||
insert_space = pynutil.insert(" ")
|
||||
delete_extra_space = pynini.cross(pynini.closure(FUN_WHITE_SPACE, 1), " ")
|
||||
|
||||
|
||||
def generator_main(file_name: str, graphs: Dict[str, "pynini.FstLike"]):
|
||||
"""
|
||||
Exports graph as OpenFst finite state archive (FAR) file with given file name and rule name.
|
||||
|
||||
Args:
|
||||
file_name: exported file name
|
||||
graphs: Mapping of a rule name and Pynini WFST graph to be exported
|
||||
"""
|
||||
exporter = export.Exporter(file_name)
|
||||
for rule, graph in graphs.items():
|
||||
exporter[rule] = graph.optimize()
|
||||
exporter.close()
|
||||
print(f"Created {file_name}")
|
||||
|
||||
|
||||
class GraphFst:
|
||||
"""
|
||||
Base class for all grammar fsts.
|
||||
|
||||
Args:
|
||||
name: name of grammar class
|
||||
kind: either 'classify' or 'verbalize'
|
||||
deterministic: if True will provide a single transduction option,
|
||||
for False multiple transduction are generated (used for audio-based normalization)
|
||||
"""
|
||||
|
||||
def __init__(self, name: str, kind: str, deterministic: bool = True):
|
||||
self.name = name
|
||||
self.kind = kind
|
||||
self._fst = None
|
||||
self.deterministic = deterministic
|
||||
|
||||
self.far_path = Path(os.path.dirname(__file__) + "/grammars/" + kind + "/" + name + ".far")
|
||||
if self.far_exist():
|
||||
self._fst = Far(
|
||||
self.far_path, mode="r", arc_type="standard", far_type="default"
|
||||
).get_fst()
|
||||
|
||||
def far_exist(self) -> bool:
|
||||
"""
|
||||
Returns true if FAR can be loaded
|
||||
"""
|
||||
return self.far_path.exists()
|
||||
|
||||
@property
|
||||
def fst(self) -> "pynini.FstLike":
|
||||
return self._fst
|
||||
|
||||
@fst.setter
|
||||
def fst(self, fst):
|
||||
self._fst = fst
|
||||
|
||||
def add_tokens(self, fst) -> "pynini.FstLike":
|
||||
"""
|
||||
Wraps class name around to given fst
|
||||
|
||||
Args:
|
||||
fst: input fst
|
||||
|
||||
Returns:
|
||||
Fst: fst
|
||||
"""
|
||||
return pynutil.insert(f"{self.name} {{ ") + fst + pynutil.insert(" }")
|
||||
|
||||
def delete_tokens(self, fst) -> "pynini.FstLike":
|
||||
"""
|
||||
Deletes class name wrap around output of given fst
|
||||
|
||||
Args:
|
||||
fst: input fst
|
||||
|
||||
Returns:
|
||||
Fst: fst
|
||||
"""
|
||||
res = (
|
||||
pynutil.delete(f"{self.name}")
|
||||
+ delete_space
|
||||
+ pynutil.delete("{")
|
||||
+ delete_space
|
||||
+ fst
|
||||
+ delete_space
|
||||
+ pynutil.delete("}")
|
||||
)
|
||||
return res @ pynini.cdrewrite(pynini.cross("\u00A0", " "), "", "", FUN_SIGMA)
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import GraphFst
|
||||
from fun_text_processing.text_normalization.zh.utils import (
|
||||
UNIT_1e01,
|
||||
UNIT_1e02,
|
||||
UNIT_1e03,
|
||||
UNIT_1e04,
|
||||
get_abs_path,
|
||||
)
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Cardinal(GraphFst):
|
||||
"""
|
||||
self.graph_cardinal:
|
||||
5 -> 五
|
||||
12 -> 十二
|
||||
213 -> 二百一十三
|
||||
3123 -> 三千一百二十三
|
||||
51234 -> 五万一千二百三十四
|
||||
51,234 -> 五万一千二百三十四
|
||||
0.125 -> 零点一二五
|
||||
self.fst:
|
||||
123 -> cardinal{ integer: "一二三" }
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="cardinal", kind="classify", deterministic=deterministic)
|
||||
# base cardinal from digit to chinese
|
||||
graph_digit = pynini.string_file(get_abs_path("data/number/digit.tsv"))
|
||||
graph_zero = pynini.string_file(get_abs_path("data/number/zero.tsv"))
|
||||
graph_teen = pynini.string_file(get_abs_path("data/number/digit_teen.tsv"))
|
||||
graph_sign = pynini.string_file(get_abs_path("data/number/sign.tsv"))
|
||||
|
||||
graph_digit_with_zero = graph_digit | graph_zero # from 0(read out) to 9
|
||||
graph_no_zero = pynini.cross("0", "") # cardinalzero,in some case we don't read it out
|
||||
graph_digit_no_zero = graph_digit | graph_no_zero # from 0(not read out) to 9
|
||||
insert_zero = pynutil.insert("零")
|
||||
delete_punct = pynutil.delete(",") # to deal with 5,000,cardinalin english form
|
||||
|
||||
# 15 in 215
|
||||
graph_ten_u = (graph_digit + pynutil.insert(UNIT_1e01) + graph_digit_no_zero) | (
|
||||
graph_zero + graph_digit
|
||||
)
|
||||
# 15 only
|
||||
graph_ten = (graph_teen + pynutil.insert(UNIT_1e01) + graph_digit_no_zero) | (
|
||||
graph_zero + graph_digit
|
||||
)
|
||||
# 215 or 200
|
||||
graph_hundred = (graph_digit + pynutil.insert(UNIT_1e02) + graph_ten_u) | (
|
||||
graph_digit + pynutil.insert(UNIT_1e02) + graph_no_zero**2
|
||||
)
|
||||
# 3000 or 3002 or 3012 or 3123
|
||||
# both with 3,000 or 3,002 or 3,012 or 3,123
|
||||
graph_thousand_sign = (
|
||||
(graph_digit + pynutil.insert(UNIT_1e03) + delete_punct + graph_hundred)
|
||||
| (
|
||||
graph_digit
|
||||
+ pynutil.insert(UNIT_1e03)
|
||||
+ delete_punct.ques
|
||||
+ graph_zero
|
||||
+ graph_digit
|
||||
+ pynutil.insert(UNIT_1e01)
|
||||
+ graph_digit_no_zero
|
||||
)
|
||||
| (
|
||||
graph_digit
|
||||
+ pynutil.insert(UNIT_1e03)
|
||||
+ delete_punct.ques
|
||||
+ graph_zero
|
||||
+ graph_no_zero
|
||||
+ graph_digit
|
||||
)
|
||||
| (graph_digit + pynutil.insert(UNIT_1e03) + delete_punct.ques + graph_no_zero**3)
|
||||
)
|
||||
# 20001234 or 2001234 or 201234 or 21234 or 20234 or 20023 or 20002 or 20000
|
||||
# 8 digits max supported,for 9 digits cardinal,often write with unit instead of read it in only cardinalform
|
||||
graph_ten_thousand_sign = (
|
||||
(graph_thousand_sign | graph_hundred | graph_ten | graph_digit_no_zero)
|
||||
+ pynutil.insert(UNIT_1e04)
|
||||
+ (
|
||||
graph_thousand_sign
|
||||
| (graph_no_zero + delete_punct.ques + insert_zero + graph_hundred)
|
||||
| (
|
||||
graph_no_zero
|
||||
+ delete_punct.ques
|
||||
+ graph_no_zero
|
||||
+ insert_zero
|
||||
+ (graph_digit + pynutil.insert(UNIT_1e01) + graph_digit_no_zero)
|
||||
)
|
||||
| (graph_no_zero + delete_punct.ques + graph_no_zero**2 + insert_zero + graph_digit)
|
||||
| (graph_no_zero**4)
|
||||
)
|
||||
)
|
||||
|
||||
# cardinalstring like 123 or 123.456,used in phone cardinal,ID cardinal,etc.
|
||||
graph_numstring = pynini.closure(graph_digit_with_zero, 1) | (
|
||||
pynini.closure(graph_digit_with_zero, 1)
|
||||
+ pynini.cross(".", "点")
|
||||
+ pynini.closure(graph_digit_with_zero, 1)
|
||||
)
|
||||
|
||||
graph = (
|
||||
graph_hundred
|
||||
| graph_ten
|
||||
| graph_digit_with_zero
|
||||
| graph_thousand_sign
|
||||
| graph_ten_thousand_sign
|
||||
)
|
||||
# 456.123
|
||||
graph_decimal = graph + pynini.cross(".", "点") + pynini.closure(graph_digit_with_zero, 1)
|
||||
graph_cardinal = graph | graph_decimal
|
||||
signed_cardinal = graph_sign + graph_cardinal
|
||||
self.graph_cardinal = (graph_cardinal | signed_cardinal).optimize()
|
||||
|
||||
graph_numstring = self.add_tokens(
|
||||
pynutil.insert('integer: "') + graph_numstring + pynutil.insert('"')
|
||||
)
|
||||
self.fst = graph_numstring.optimize()
|
||||
@@ -0,0 +1,14 @@
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import FUN_CHAR, GraphFst
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Char(GraphFst):
|
||||
"""
|
||||
你 -> char { name: "你" }
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="char", kind="classify", deterministic=deterministic)
|
||||
|
||||
graph = pynutil.insert('name: "') + FUN_CHAR + pynutil.insert('"')
|
||||
self.fst = graph.optimize()
|
||||
@@ -0,0 +1,97 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import (
|
||||
FUN_CHAR,
|
||||
FUN_DIGIT,
|
||||
GraphFst,
|
||||
insert_space,
|
||||
)
|
||||
from fun_text_processing.text_normalization.zh.utils import get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Date(GraphFst):
|
||||
"""
|
||||
2002年 -> tokens { date { year: "2002" } }
|
||||
2002-01-28 -> tokens { date { year: "2002" month: "01" day: "28"} }
|
||||
2002/01/28 -> tokens { date { year: "2002" month: "01" day: "28"} }
|
||||
2002.01.28 -> tokens { date { year: "2002" month: "01" day: "28"} }
|
||||
2002/02 -> tokens { date { year: "2002" month "02"} }
|
||||
02/11 -> tokens { date { year: "02" month "11"} } different with case "fraction 2/11"
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="date", kind="classify", deterministic=deterministic)
|
||||
graph_digit = pynini.string_file(get_abs_path("data/number/digit.tsv"))
|
||||
graph_zero = pynini.string_file(get_abs_path("data/number/zero.tsv"))
|
||||
year_whitelist = pynini.string_file(get_abs_path("data/date/year_suffix.tsv"))
|
||||
|
||||
delete_date_sign = pynutil.delete("/") | pynutil.delete("-") | pynutil.delete(".")
|
||||
|
||||
# 2012年
|
||||
date_type0 = (
|
||||
pynutil.insert('year: "')
|
||||
+ pynini.closure(graph_digit | graph_zero, 2, 4)
|
||||
+ "年"
|
||||
+ pynini.difference(FUN_CHAR, year_whitelist)
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
|
||||
year_2_4_digit = pynini.closure(FUN_DIGIT, 2, 4) + delete_date_sign
|
||||
year_4_digit = pynini.closure(FUN_DIGIT, 4, 4) + delete_date_sign
|
||||
year_2_digit_with_zero = "0" + FUN_DIGIT + delete_date_sign
|
||||
month_no_day_with_zero = "0" + FUN_DIGIT
|
||||
month_no_day = pynini.closure(FUN_DIGIT, 2, 2)
|
||||
month = pynini.closure(FUN_DIGIT, 1, 2) + delete_date_sign
|
||||
day = pynini.closure(FUN_DIGIT, 1, 2)
|
||||
|
||||
# 2012/01/28
|
||||
date_type1 = (
|
||||
pynutil.insert('year: "')
|
||||
+ year_2_4_digit
|
||||
+ pynutil.insert('"')
|
||||
+ insert_space
|
||||
+ pynutil.insert('month: "')
|
||||
+ month
|
||||
+ pynutil.insert('"')
|
||||
+ insert_space
|
||||
+ pynutil.insert('day: "')
|
||||
+ day
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
|
||||
# 12/01
|
||||
date_type2 = (
|
||||
pynutil.insert('year: "')
|
||||
+ year_2_4_digit
|
||||
+ pynutil.insert('"')
|
||||
+ insert_space
|
||||
+ pynutil.insert('month: "')
|
||||
+ month_no_day_with_zero
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
|
||||
# 2012/11
|
||||
date_type3 = (
|
||||
pynutil.insert('year: "')
|
||||
+ year_4_digit
|
||||
+ pynutil.insert('"')
|
||||
+ insert_space
|
||||
+ pynutil.insert('month: "')
|
||||
+ month_no_day
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
|
||||
# 02/05
|
||||
date_type4 = (
|
||||
pynutil.insert('year: "')
|
||||
+ year_2_digit_with_zero
|
||||
+ pynutil.insert('"')
|
||||
+ insert_space
|
||||
+ pynutil.insert('month: "')
|
||||
+ month_no_day
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
# add your date type as date_typex here.
|
||||
graph = date_type0 | date_type1 | date_type2 | date_type3 | date_type4
|
||||
|
||||
self.fst = self.add_tokens(graph).optimize()
|
||||
@@ -0,0 +1,26 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import FUN_DIGIT, GraphFst, insert_space
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Fraction(GraphFst):
|
||||
"""
|
||||
1/5 -> tokens { fraction { numerator: "1" denominator: "5" } }
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="fraction", kind="classify", deterministic=deterministic)
|
||||
|
||||
numerator = pynini.closure(FUN_DIGIT, 1) + pynutil.delete("/")
|
||||
denominator = pynini.closure(FUN_DIGIT, 1)
|
||||
graph = (
|
||||
pynutil.insert('numerator: "')
|
||||
+ numerator
|
||||
+ pynutil.insert('"')
|
||||
+ insert_space
|
||||
+ pynutil.insert('denominator: "')
|
||||
+ denominator
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
|
||||
self.fst = self.add_tokens(graph).optimize()
|
||||
@@ -0,0 +1,28 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import GraphFst
|
||||
from fun_text_processing.text_normalization.zh.taggers.cardinal import Cardinal
|
||||
from fun_text_processing.text_normalization.zh.utils import get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class MathSymbol(GraphFst):
|
||||
"""
|
||||
+ -> tokens { sign: "加" }
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="sign", kind="classify", deterministic=deterministic)
|
||||
"""
|
||||
add your sign in data/math/symbol.tsv,this graph just convert sigh to character,you can add more
|
||||
cases with detailed cases
|
||||
"""
|
||||
score_sign = pynini.string_file(get_abs_path("data/math/score.tsv"))
|
||||
score = (
|
||||
pynutil.insert('score: "')
|
||||
+ Cardinal().graph_cardinal
|
||||
+ score_sign
|
||||
+ Cardinal().graph_cardinal
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
graph = score
|
||||
self.fst = graph.optimize()
|
||||
@@ -0,0 +1,39 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import GraphFst, insert_space
|
||||
from fun_text_processing.text_normalization.zh.taggers.cardinal import Cardinal
|
||||
from fun_text_processing.text_normalization.zh.utils import get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Measure(GraphFst):
|
||||
"""
|
||||
1kg -> tokens { measure { cardinal { integer: "一" } units: "千克" } }
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="measure", kind="classify", deterministic=deterministic)
|
||||
|
||||
units_en = pynini.string_file(get_abs_path("data/measure/units_en.tsv"))
|
||||
units_zh = pynini.string_file(get_abs_path("data/measure/units_zh.tsv"))
|
||||
graph = (
|
||||
pynutil.insert("cardinal { ")
|
||||
+ pynutil.insert('integer: "')
|
||||
+ Cardinal().graph_cardinal
|
||||
+ pynutil.insert('"')
|
||||
+ pynutil.insert(" }")
|
||||
+ insert_space
|
||||
+ pynutil.insert('units: "')
|
||||
+ (units_en | units_zh)
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
percent_graph = (
|
||||
pynutil.insert("decimal { ")
|
||||
+ pynutil.insert('integer_part: "')
|
||||
+ Cardinal().graph_cardinal
|
||||
+ pynutil.delete("%")
|
||||
+ pynutil.insert('"')
|
||||
+ pynutil.insert(" }")
|
||||
)
|
||||
graph |= percent_graph
|
||||
|
||||
self.fst = self.add_tokens(graph).optimize()
|
||||
@@ -0,0 +1,28 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import GraphFst, insert_space
|
||||
from fun_text_processing.text_normalization.zh.taggers.cardinal import Cardinal
|
||||
from fun_text_processing.text_normalization.zh.utils import get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Money(GraphFst):
|
||||
"""
|
||||
￥1.25 -> tokens { money { fractional_part: "元" integer_part: "一点五" } }
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="money", kind="classify", deterministic=deterministic)
|
||||
|
||||
currency_code = pynini.string_file(get_abs_path("data/money/currency_code.tsv"))
|
||||
currency_symbol = pynini.string_file(get_abs_path("data/money/currency_symbol.tsv"))
|
||||
graph = (
|
||||
pynutil.insert('fractional_part: "')
|
||||
+ (currency_code | currency_symbol)
|
||||
+ pynutil.insert('"')
|
||||
+ insert_space
|
||||
+ pynutil.insert('integer_part: "')
|
||||
+ Cardinal().graph_cardinal
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
|
||||
self.fst = self.add_tokens(graph).optimize()
|
||||
@@ -0,0 +1,39 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import FUN_SIGMA, GraphFst
|
||||
from fun_text_processing.text_normalization.zh.utils import get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class PreProcessor(GraphFst):
|
||||
"""
|
||||
Preprocessing of TN:
|
||||
1. interjections removal such as '啊, 呃'
|
||||
2. fullwidth -> halfwidth char conversion
|
||||
好啊 -> 好
|
||||
呃对 -> 对
|
||||
： -> :
|
||||
； -> ;
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
remove_interjections: bool = True,
|
||||
fullwidth_to_halfwidth: bool = True,
|
||||
):
|
||||
super().__init__(name="PreProcessor", kind="processor")
|
||||
|
||||
graph = pynini.cdrewrite("", "", "", FUN_SIGMA)
|
||||
|
||||
if remove_interjections:
|
||||
remove_interjections_graph = pynutil.delete(
|
||||
pynini.string_file(get_abs_path("data/denylist/denylist.tsv"))
|
||||
)
|
||||
graph @= pynini.cdrewrite(remove_interjections_graph, "", "", FUN_SIGMA)
|
||||
|
||||
if fullwidth_to_halfwidth:
|
||||
fullwidth_to_halfwidth_graph = pynini.string_file(
|
||||
get_abs_path("data/char/fullwidth_to_halfwidth.tsv")
|
||||
)
|
||||
graph @= pynini.cdrewrite(fullwidth_to_halfwidth_graph, "", "", FUN_SIGMA)
|
||||
|
||||
self.fst = graph.optimize()
|
||||
@@ -0,0 +1,60 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import GraphFst, insert_space
|
||||
from fun_text_processing.text_normalization.zh.utils import get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Time(GraphFst):
|
||||
"""
|
||||
1:02 -> tokens { time { hours: "1" minitus: "02" } }
|
||||
1:02:36 -> tokens { time { hours: "1" minutes: "02" seconds: "36" } }
|
||||
1:02 am -> tokens { time { hours: "1" minutes: "02" seconds: "36" suffix "am" } }
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="time", kind="classify", deterministic=deterministic)
|
||||
|
||||
h = pynini.string_file(get_abs_path("data/time/hour.tsv"))
|
||||
time_tens = pynini.string_file(get_abs_path("data/time/tens.tsv"))
|
||||
time_digit = pynini.string_file(get_abs_path("data/time/digit.tsv"))
|
||||
time_zero = pynini.string_file(get_abs_path("data/time/zero.tsv"))
|
||||
time_digit = time_digit | time_zero
|
||||
time_suffix = pynini.string_file(get_abs_path("data/time/suffix.tsv"))
|
||||
|
||||
m = time_tens + time_digit
|
||||
s = (time_tens + time_digit) | time_digit
|
||||
|
||||
delete_colon = pynini.cross(":", " ")
|
||||
|
||||
# 5:05, 14:30
|
||||
h_m = (
|
||||
pynutil.insert('hours: "')
|
||||
+ h
|
||||
+ pynutil.insert('"')
|
||||
+ delete_colon
|
||||
+ pynutil.insert('minutes: "')
|
||||
+ m
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
|
||||
# 1:30:15
|
||||
h_m_s = (
|
||||
pynutil.insert('hours: "')
|
||||
+ h
|
||||
+ pynutil.insert('"')
|
||||
+ delete_colon
|
||||
+ pynutil.insert('minutes: "')
|
||||
+ m
|
||||
+ pynutil.insert('"')
|
||||
+ delete_colon
|
||||
+ pynutil.insert('seconds: "')
|
||||
+ s
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
|
||||
graph = h_m | h_m_s
|
||||
graph_suffix = (
|
||||
graph + insert_space + pynutil.insert('suffix: "') + time_suffix + pynutil.insert('"')
|
||||
)
|
||||
graph |= graph_suffix
|
||||
self.fst = self.add_tokens(graph).optimize()
|
||||
@@ -0,0 +1,83 @@
|
||||
import os
|
||||
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import FUN_SIGMA, GraphFst
|
||||
from fun_text_processing.text_normalization.zh.taggers.cardinal import Cardinal
|
||||
from fun_text_processing.text_normalization.zh.taggers.char import Char
|
||||
from fun_text_processing.text_normalization.zh.taggers.date import Date
|
||||
from fun_text_processing.text_normalization.zh.taggers.fraction import Fraction
|
||||
from fun_text_processing.text_normalization.zh.taggers.math_symbol import MathSymbol
|
||||
from fun_text_processing.text_normalization.zh.taggers.measure import Measure
|
||||
from fun_text_processing.text_normalization.zh.taggers.money import Money
|
||||
from fun_text_processing.text_normalization.zh.taggers.preprocessor import PreProcessor
|
||||
from fun_text_processing.text_normalization.zh.taggers.time import Time
|
||||
from fun_text_processing.text_normalization.zh.taggers.whitelist import Whitelist
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class ClassifyFst(GraphFst):
|
||||
"""
|
||||
Final class that composes all other classification grammars. This class can process an entire sentence including punctuation.
|
||||
For deployment, this grammar will be compiled and exported to OpenFst Finate State Archiv (FAR) File.
|
||||
More details to deployment at NeMo/tools/text_processing_deployment.
|
||||
|
||||
Args:
|
||||
input_case: accepting either "lower_cased" or "cased" input.
|
||||
deterministic: if True will provide a single transduction option,
|
||||
for False multiple options (used for audio-based normalization)
|
||||
cache_dir: path to a dir with .far grammar file. Set to None to avoid using cache.
|
||||
overwrite_cache: set to True to overwrite .far files
|
||||
whitelist: path to a file with whitelist replacements
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
input_case: str,
|
||||
deterministic: bool = True,
|
||||
cache_dir: str = None,
|
||||
overwrite_cache: bool = False,
|
||||
whitelist: str = None,
|
||||
):
|
||||
super().__init__(name="tokenize_and_classify", kind="classify", deterministic=deterministic)
|
||||
|
||||
far_file = None
|
||||
if cache_dir is not None and cache_dir != "None":
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
whitelist_file = os.path.basename(whitelist) if whitelist else ""
|
||||
far_file = os.path.join(
|
||||
cache_dir,
|
||||
f"zh_tn_{deterministic}_deterministic_{input_case}_{whitelist_file}_tokenize.far",
|
||||
)
|
||||
if not overwrite_cache and far_file and os.path.exists(far_file):
|
||||
self.fst = pynini.Far(far_file, mode="r")["tokenize_and_classify"]
|
||||
else:
|
||||
date = Date(deterministic=deterministic)
|
||||
cardinal = Cardinal(deterministic=deterministic)
|
||||
char = Char(deterministic=deterministic)
|
||||
fraction = Fraction(deterministic=deterministic)
|
||||
math_symbol = MathSymbol(deterministic=deterministic)
|
||||
money = Money(deterministic=deterministic)
|
||||
measure = Measure(deterministic=deterministic)
|
||||
time = Time(deterministic=deterministic)
|
||||
whitelist = Whitelist(deterministic=deterministic)
|
||||
|
||||
classify = pynini.union(
|
||||
pynutil.add_weight(date.fst, 1.02),
|
||||
pynutil.add_weight(fraction.fst, 1.05),
|
||||
pynutil.add_weight(money.fst, 1.05),
|
||||
pynutil.add_weight(measure.fst, 1.05),
|
||||
pynutil.add_weight(time.fst, 1.05),
|
||||
pynutil.add_weight(whitelist.fst, 1.03),
|
||||
pynutil.add_weight(cardinal.fst, 1.06),
|
||||
pynutil.add_weight(math_symbol.fst, 1.08),
|
||||
pynutil.add_weight(char.fst, 100),
|
||||
)
|
||||
token = pynutil.insert("tokens { ") + classify + pynutil.insert(" } ")
|
||||
|
||||
tagger = pynini.cdrewrite(token.optimize(), "", "", FUN_SIGMA).optimize()
|
||||
|
||||
preprocessor = PreProcessor(
|
||||
remove_interjections=True,
|
||||
fullwidth_to_halfwidth=True,
|
||||
)
|
||||
self.fst = preprocessor.fst @ tagger
|
||||
@@ -0,0 +1,25 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import GraphFst
|
||||
from fun_text_processing.text_normalization.zh.utils import get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Whitelist(GraphFst):
|
||||
"""
|
||||
ATM -> tokens { whitelist: "ATM" }
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="whitelist", kind="classify", deterministic=deterministic)
|
||||
|
||||
whitelist = pynini.string_file(get_abs_path("data/whitelist/default.tsv"))
|
||||
erhua = pynutil.insert('erhua: "') + pynini.accep("儿") + pynutil.insert('"')
|
||||
sign = pynini.string_file(get_abs_path("data/math/symbol.tsv"))
|
||||
whitelist = (
|
||||
pynutil.insert('name: "')
|
||||
+ (pynini.string_file(get_abs_path("data/erhua/whitelist.tsv")) | whitelist | sign)
|
||||
+ pynutil.insert('"')
|
||||
)
|
||||
graph = pynutil.add_weight(erhua, 0.1) | whitelist
|
||||
|
||||
self.fst = graph.optimize()
|
||||
@@ -0,0 +1,51 @@
|
||||
import csv
|
||||
import os
|
||||
|
||||
UNIT_1e01 = "十"
|
||||
UNIT_1e02 = "百"
|
||||
UNIT_1e03 = "千"
|
||||
UNIT_1e04 = "万"
|
||||
|
||||
|
||||
def get_abs_path(rel_path):
|
||||
"""
|
||||
Get absolute path
|
||||
|
||||
Args:
|
||||
rel_path: relative path to this file
|
||||
|
||||
Returns absolute path
|
||||
"""
|
||||
return os.path.dirname(os.path.abspath(__file__)) + "/" + rel_path
|
||||
|
||||
|
||||
def load_labels(abs_path):
|
||||
"""
|
||||
loads relative path file as dictionary
|
||||
|
||||
Args:
|
||||
abs_path: absolute path
|
||||
|
||||
Returns dictionary of mappings
|
||||
"""
|
||||
label_tsv = open(abs_path, encoding="utf-8")
|
||||
labels = list(csv.reader(label_tsv, delimiter="\t"))
|
||||
return labels
|
||||
|
||||
|
||||
def augment_labels_with_punct_at_end(labels):
|
||||
"""
|
||||
augments labels: if key ends on a punctuation that value does not have, add a new label
|
||||
where the value maintains the punctuation
|
||||
|
||||
Args:
|
||||
labels : input labels
|
||||
Returns:
|
||||
additional labels
|
||||
"""
|
||||
res = []
|
||||
for label in labels:
|
||||
if len(label) > 1:
|
||||
if label[0][-1] == "." and label[1][-1] != ".":
|
||||
res.append([label[0], label[1] + "."] + label[2:])
|
||||
return res
|
||||
@@ -0,0 +1,16 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import FUN_NOT_QUOTE, GraphFst
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Cardinal(GraphFst):
|
||||
"""
|
||||
tokens { cardinal { integer: "一二三" } } -> 一二三
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="cardinal", kind="verbalize", deterministic=deterministic)
|
||||
|
||||
graph = pynutil.delete('integer: "') + pynini.closure(FUN_NOT_QUOTE) + pynutil.delete('"')
|
||||
|
||||
self.fst = self.delete_tokens(graph).optimize()
|
||||
@@ -0,0 +1,14 @@
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import FUN_NOT_QUOTE, GraphFst
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Char(GraphFst):
|
||||
"""
|
||||
tokens { char: "你" } -> 你
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="char", kind="verbalize", deterministic=deterministic)
|
||||
|
||||
graph = pynutil.delete('name: "') + FUN_NOT_QUOTE + pynutil.delete('"')
|
||||
self.fst = graph.optimize()
|
||||
@@ -0,0 +1,61 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import (
|
||||
FUN_NOT_QUOTE,
|
||||
GraphFst,
|
||||
delete_space,
|
||||
)
|
||||
from fun_text_processing.text_normalization.zh.utils import UNIT_1e01, get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Date(GraphFst):
|
||||
"""
|
||||
tokens { date { year: "2002" month: "01" day: "28"} } -> 二零零二年一月二十八日
|
||||
tokens { date { year: "2002" } } -> 二零零八年
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="date", kind="verbalize", deterministic=deterministic)
|
||||
date_type0 = pynutil.delete('year: "') + pynini.closure(FUN_NOT_QUOTE) + pynutil.delete('"')
|
||||
graph_digit = pynini.string_file(get_abs_path("data/number/digit.tsv"))
|
||||
graph_teen = pynini.string_file(get_abs_path("data/number/digit_teen.tsv"))
|
||||
graph_zero = pynini.string_file(get_abs_path("data/number/zero.tsv"))
|
||||
graph_no_zero = pynini.cross("0", "")
|
||||
graph_year = pynini.closure(graph_digit | graph_zero, 2, 4)
|
||||
graph_digit_no_zero = graph_digit | graph_no_zero
|
||||
graph_2_digit_date = (graph_teen + pynutil.insert(UNIT_1e01) + graph_digit_no_zero) | (
|
||||
graph_no_zero + graph_digit
|
||||
)
|
||||
|
||||
date_type1 = (
|
||||
pynutil.delete('year: "')
|
||||
+ graph_year
|
||||
+ pynutil.insert("年")
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete('month: "')
|
||||
+ graph_2_digit_date
|
||||
+ pynutil.insert("月")
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete('day: "')
|
||||
+ graph_2_digit_date
|
||||
+ pynutil.insert("日")
|
||||
+ pynutil.delete('"')
|
||||
)
|
||||
|
||||
date_type2 = (
|
||||
pynutil.delete('year: "')
|
||||
+ graph_year
|
||||
+ pynutil.insert("年")
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete('month: "')
|
||||
+ graph_2_digit_date
|
||||
+ pynutil.insert("月")
|
||||
+ pynutil.delete('"')
|
||||
)
|
||||
|
||||
graph = date_type0 | date_type1 | date_type2
|
||||
|
||||
self.fst = self.delete_tokens(graph).optimize()
|
||||
@@ -0,0 +1,20 @@
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import GraphFst
|
||||
from fun_text_processing.text_normalization.zh.taggers.cardinal import Cardinal
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Fraction(GraphFst):
|
||||
"""
|
||||
tokens { fraction { denominator: "5" numerator: "1" } } -> 五分之一
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="fraction", kind="verbalize", deterministic=deterministic)
|
||||
|
||||
denominator = (
|
||||
pynutil.delete('denominator: "') + Cardinal().graph_cardinal + pynutil.delete('"')
|
||||
)
|
||||
numerator = pynutil.delete('numerator: "') + Cardinal().graph_cardinal + pynutil.delete('"')
|
||||
graph = denominator + pynutil.delete(" ") + pynutil.insert("分之") + numerator
|
||||
|
||||
self.fst = self.delete_tokens(graph).optimize()
|
||||
@@ -0,0 +1,16 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import FUN_NOT_QUOTE, GraphFst
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class MathSymbol(GraphFst):
|
||||
"""
|
||||
tokens { sign: "加" } -> 加
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="sign", kind="verbalize", deterministic=deterministic)
|
||||
|
||||
graph = pynutil.delete('score: "') + pynini.closure(FUN_NOT_QUOTE) + pynutil.delete('"')
|
||||
|
||||
self.fst = graph.optimize()
|
||||
@@ -0,0 +1,41 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import (
|
||||
FUN_NOT_QUOTE,
|
||||
GraphFst,
|
||||
delete_space,
|
||||
)
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Measure(GraphFst):
|
||||
"""
|
||||
tokens { measure { cardinal: "一" } units: "千克" } } -> 一千克
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="measure", kind="verbalize", deterministic=deterministic)
|
||||
|
||||
graph = (
|
||||
pynutil.delete("cardinal {")
|
||||
+ delete_space
|
||||
+ pynutil.delete('integer: "')
|
||||
+ pynini.closure(FUN_NOT_QUOTE)
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete("}")
|
||||
+ delete_space
|
||||
+ pynutil.delete('units: "')
|
||||
+ pynini.closure(FUN_NOT_QUOTE)
|
||||
+ pynutil.delete('"')
|
||||
)
|
||||
percent_graph = (
|
||||
pynutil.delete("decimal { ")
|
||||
+ pynutil.delete('integer_part: "')
|
||||
+ pynutil.insert("百分之")
|
||||
+ pynini.closure(FUN_NOT_QUOTE, 1)
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete("}")
|
||||
)
|
||||
graph |= percent_graph
|
||||
self.fst = self.delete_tokens(graph).optimize()
|
||||
@@ -0,0 +1,31 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import (
|
||||
FUN_NOT_QUOTE,
|
||||
GraphFst,
|
||||
delete_space,
|
||||
)
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Money(GraphFst):
|
||||
"""
|
||||
tokens { money { integer_part: "一点五" fractional_part: "元" } } -> 一点五元
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="money", kind="verbalize", deterministic=deterministic)
|
||||
|
||||
cur = (
|
||||
pynutil.delete('fractional_part: "')
|
||||
+ pynini.closure(FUN_NOT_QUOTE)
|
||||
+ pynutil.delete('"')
|
||||
)
|
||||
num = (
|
||||
pynutil.delete('integer_part: "')
|
||||
+ pynini.closure(FUN_NOT_QUOTE)
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
)
|
||||
graph = num + cur
|
||||
|
||||
self.fst = self.delete_tokens(graph).optimize()
|
||||
@@ -0,0 +1,73 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import (
|
||||
FUN_ALPHA,
|
||||
FUN_DIGIT,
|
||||
FUN_PUNCT,
|
||||
FUN_SIGMA,
|
||||
FUN_WHITE_SPACE,
|
||||
GraphFst,
|
||||
)
|
||||
from fun_text_processing.text_normalization.zh.utils import get_abs_path
|
||||
from pynini.lib import pynutil, utf8
|
||||
|
||||
|
||||
class PostProcessor(GraphFst):
|
||||
"""
|
||||
Postprocessing of TN, now contains:
|
||||
1. punctuation removal
|
||||
2. letter case conversion
|
||||
3. oov tagger
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
remove_puncts: bool = False,
|
||||
to_upper: bool = False,
|
||||
to_lower: bool = False,
|
||||
tag_oov: bool = False,
|
||||
):
|
||||
super().__init__(name="PostProcessor", kind="processor")
|
||||
|
||||
graph = pynini.cdrewrite("", "", "", FUN_SIGMA)
|
||||
if remove_puncts:
|
||||
remove_puncts_graph = pynutil.delete(
|
||||
pynini.union(
|
||||
FUN_PUNCT, pynini.string_file(get_abs_path("data/char/punctuations_zh.tsv"))
|
||||
)
|
||||
)
|
||||
graph @= pynini.cdrewrite(remove_puncts_graph, "", "", FUN_SIGMA).optimize()
|
||||
|
||||
if to_upper or to_lower:
|
||||
if to_upper:
|
||||
conv_cases_graph = pynini.inverse(
|
||||
pynini.string_file(get_abs_path("data/char/upper_to_lower.tsv"))
|
||||
)
|
||||
else:
|
||||
conv_cases_graph = pynini.string_file(get_abs_path("data/char/upper_to_lower.tsv"))
|
||||
|
||||
graph @= pynini.cdrewrite(conv_cases_graph, "", "", FUN_SIGMA).optimize()
|
||||
|
||||
if tag_oov:
|
||||
zh_charset_std = pynini.string_file(
|
||||
get_abs_path("data/char/charset_national_standard_2013_8105.tsv")
|
||||
)
|
||||
zh_charset_ext = pynini.string_file(get_abs_path("data/char/charset_extension.tsv"))
|
||||
|
||||
zh_charset = (
|
||||
zh_charset_std
|
||||
| zh_charset_ext
|
||||
| pynini.string_file(get_abs_path("data/char/punctuations_zh.tsv"))
|
||||
)
|
||||
en_charset = FUN_DIGIT | FUN_ALPHA | FUN_PUNCT | FUN_WHITE_SPACE
|
||||
charset = zh_charset | en_charset
|
||||
|
||||
with open(get_abs_path("data/char/oov_tags.tsv"), "r") as f:
|
||||
tags = f.readline().strip().split("\t")
|
||||
assert len(tags) == 2
|
||||
ltag, rtag = tags
|
||||
|
||||
oov_charset = pynini.difference(utf8.VALID_UTF8_CHAR, charset)
|
||||
tag_oov_graph = pynutil.insert(ltag) + oov_charset + pynutil.insert(rtag)
|
||||
graph @= pynini.cdrewrite(tag_oov_graph, "", "", FUN_SIGMA).optimize()
|
||||
|
||||
self.fst = graph.optimize()
|
||||
@@ -0,0 +1,88 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import (
|
||||
FUN_NOT_QUOTE,
|
||||
GraphFst,
|
||||
delete_space,
|
||||
)
|
||||
from fun_text_processing.text_normalization.zh.utils import UNIT_1e01, get_abs_path
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Time(GraphFst):
|
||||
"""
|
||||
tokens { time { h: "1" m: "02" s: "36" } } -> 一点零二分三十六秒
|
||||
tokens { time { suffix "am" hours: "1" minutes: "02" seconds: "36" } } -> 上午一点零二分三十六秒
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="time", kind="verbalize", deterministic=deterministic)
|
||||
graph_digit = pynini.string_file(get_abs_path("data/number/digit.tsv"))
|
||||
graph_teen = pynini.string_file(get_abs_path("data/number/digit_teen.tsv"))
|
||||
graph_zero = pynini.string_file(get_abs_path("data/number/zero.tsv"))
|
||||
graph_no_zero = pynini.cross("0", "")
|
||||
|
||||
graph_digit_no_zero = graph_digit | graph_no_zero
|
||||
|
||||
graph_2_digit_zero_none = pynini.cross("0", "") + pynini.cross("0", "")
|
||||
graph_2_digit_zero = pynini.cross("00", "零")
|
||||
|
||||
graph_2_digit_time = (graph_teen + pynutil.insert(UNIT_1e01) + graph_digit_no_zero) | (
|
||||
graph_zero + graph_digit
|
||||
)
|
||||
h = graph_2_digit_time | graph_2_digit_zero | graph_digit
|
||||
m = graph_2_digit_time | graph_2_digit_zero
|
||||
s = graph_2_digit_time | graph_2_digit_zero
|
||||
|
||||
# 6:25
|
||||
h_m = (
|
||||
pynutil.delete('hours: "')
|
||||
+ h
|
||||
+ pynutil.insert("点")
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete('minutes: "')
|
||||
+ (graph_2_digit_time)
|
||||
+ pynutil.insert("分")
|
||||
+ pynutil.delete('"')
|
||||
)
|
||||
|
||||
# 23:00
|
||||
h_00 = (
|
||||
pynutil.delete('hours: "')
|
||||
+ h
|
||||
+ pynutil.insert("点")
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete('minutes: "')
|
||||
+ (graph_2_digit_zero_none)
|
||||
+ pynutil.delete('"')
|
||||
)
|
||||
|
||||
# 9:12:52
|
||||
h_m_s = (
|
||||
pynutil.delete('hours: "')
|
||||
+ h
|
||||
+ pynutil.insert("点")
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete('minutes: "')
|
||||
+ m
|
||||
+ pynutil.insert("分")
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ pynutil.delete('seconds: "')
|
||||
+ s
|
||||
+ pynutil.insert("秒")
|
||||
+ pynutil.delete('"')
|
||||
)
|
||||
|
||||
graph = h_m | h_m_s | h_00
|
||||
graph_suffix = (
|
||||
pynutil.delete('suffix: "')
|
||||
+ pynini.closure(FUN_NOT_QUOTE)
|
||||
+ pynutil.delete('"')
|
||||
+ delete_space
|
||||
+ graph
|
||||
)
|
||||
graph |= graph_suffix
|
||||
self.fst = self.delete_tokens(graph).optimize()
|
||||
@@ -0,0 +1,49 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import GraphFst
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.cardinal import Cardinal
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.char import Char
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.date import Date
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.fraction import Fraction
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.math_symbol import MathSymbol
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.measure import Measure
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.money import Money
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.time import Time
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.whitelist import Whitelist
|
||||
|
||||
|
||||
class VerbalizeFst(GraphFst):
|
||||
"""
|
||||
Composes other verbalizer grammars.
|
||||
For deployment, this grammar will be compiled and exported to OpenFst Finate State Archiv (FAR) File.
|
||||
More details to deployment at NeMo/tools/text_processing_deployment.
|
||||
Args:
|
||||
deterministic: if True will provide a single transduction option,
|
||||
for False multiple options (used for audio-based normalization)
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True):
|
||||
super().__init__(name="verbalize", kind="verbalize", deterministic=deterministic)
|
||||
|
||||
date = Date(deterministic=deterministic)
|
||||
cardinal = Cardinal(deterministic=deterministic)
|
||||
char = Char(deterministic=deterministic)
|
||||
fraction = Fraction(deterministic=deterministic)
|
||||
math_symbol = MathSymbol(deterministic=deterministic)
|
||||
money = Money(deterministic=deterministic)
|
||||
measure = Measure(deterministic=deterministic)
|
||||
time = Time(deterministic=deterministic)
|
||||
whitelist = Whitelist(deterministic=deterministic)
|
||||
|
||||
graph = pynini.union(
|
||||
date.fst,
|
||||
cardinal.fst,
|
||||
fraction.fst,
|
||||
char.fst,
|
||||
math_symbol.fst,
|
||||
money.fst,
|
||||
measure.fst,
|
||||
time.fst,
|
||||
whitelist.fst,
|
||||
)
|
||||
|
||||
self.fst = graph.optimize()
|
||||
@@ -0,0 +1,51 @@
|
||||
import os
|
||||
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import (
|
||||
GraphFst,
|
||||
delete_space,
|
||||
generator_main,
|
||||
)
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.postprocessor import PostProcessor
|
||||
from fun_text_processing.text_normalization.zh.verbalizers.verbalize import VerbalizeFst
|
||||
from pynini.lib import pynutil
|
||||
|
||||
# import logging
|
||||
|
||||
|
||||
class VerbalizeFinalFst(GraphFst):
|
||||
""" """
|
||||
|
||||
def __init__(
|
||||
self, deterministic: bool = True, cache_dir: str = None, overwrite_cache: bool = False
|
||||
):
|
||||
super().__init__(name="verbalize_final", kind="verbalize", deterministic=deterministic)
|
||||
far_file = None
|
||||
if cache_dir is not None and cache_dir != "None":
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
far_file = os.path.join(
|
||||
cache_dir, f"zh_tn_{deterministic}_deterministic_verbalizer.far"
|
||||
)
|
||||
if not overwrite_cache and far_file and os.path.exists(far_file):
|
||||
self.fst = pynini.Far(far_file, mode="r")["verbalize"]
|
||||
else:
|
||||
token_graph = VerbalizeFst(deterministic=deterministic)
|
||||
token_verbalizer = (
|
||||
pynutil.delete("tokens {")
|
||||
+ delete_space
|
||||
+ token_graph.fst
|
||||
+ delete_space
|
||||
+ pynutil.delete(" }")
|
||||
)
|
||||
verbalizer = pynini.closure(delete_space + token_verbalizer + delete_space)
|
||||
|
||||
postprocessor = PostProcessor(
|
||||
remove_puncts=False,
|
||||
to_upper=False,
|
||||
to_lower=False,
|
||||
tag_oov=False,
|
||||
)
|
||||
|
||||
self.fst = (verbalizer @ postprocessor.fst).optimize()
|
||||
if far_file:
|
||||
generator_main(far_file, {"verbalize": self.fst})
|
||||
@@ -0,0 +1,16 @@
|
||||
import pynini
|
||||
from fun_text_processing.text_normalization.zh.graph_utils import FUN_NOT_QUOTE, GraphFst
|
||||
from pynini.lib import pynutil
|
||||
|
||||
|
||||
class Whitelist(GraphFst):
|
||||
"""
|
||||
tokens { whitelist: "ATM" } -> A T M
|
||||
"""
|
||||
|
||||
def __init__(self, deterministic: bool = True, lm: bool = False):
|
||||
super().__init__(name="whitelist", kind="verbalize", deterministic=deterministic)
|
||||
remove_erhua = pynutil.delete('erhua: "') + pynutil.delete("儿") + pynutil.delete('"')
|
||||
whitelist = pynutil.delete('name: "') + pynini.closure(FUN_NOT_QUOTE) + pynutil.delete('"')
|
||||
graph = remove_erhua | whitelist
|
||||
self.fst = graph.optimize()
|
||||
Reference in New Issue
Block a user