wehub-resource-sync
78 lines
5.5 KiB
Markdown
78 lines
5.5 KiB
Markdown
# DiffSinger (OpenVPI maintained version)
|
|
|
|
[](https://arxiv.org/abs/2105.02446)
|
|
[](https://github.com/openvpi/DiffSinger/releases)
|
|
[](https://www.bilibili.com/video/BV1be411N7JA/)
|
|
[](https://github.com/openvpi/DiffSinger/blob/main/LICENSE)
|
|
|
|
This is a refactored and enhanced version of _DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism_ based on the original [paper](https://arxiv.org/abs/2105.02446) and [implementation](https://github.com/MoonInTheRiver/DiffSinger), which provides:
|
|
|
|
- Cleaner code structure: useless and redundant files are removed and the others are re-organized.
|
|
- Better sound quality: the sampling rate of synthesized audio are adapted to 44.1 kHz instead of the original 24 kHz.
|
|
- Higher fidelity: improved acoustic models and diffusion sampling acceleration algorithms are integrated.
|
|
- More controllability: introduced variance models and parameters for prediction and control of pitch, energy, breathiness, etc.
|
|
- Production compatibility: functionalities are designed to match the requirements of production deployment and the SVS communities.
|
|
|
|
| Overview | Variance Model | Acoustic Model |
|
|
|:-------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------:|:-------------------------------------------------------------------------------------:|
|
|
| <img src="docs/resources/arch-overview.jpg" alt="arch-overview" style="zoom: 60%;" /> | <img src="docs/resources/arch-variance.jpg" alt="arch-variance" style="zoom: 50%;" /> | <img src="docs/resources/arch-acoustic.jpg" alt="arch-acoustic" style="zoom: 60%;" /> |
|
|
|
|
## User Guidance
|
|
|
|
> Navigation page including all tutorials and resources (Chinese): https://diffsinger.com
|
|
|
|
- **Installation & basic usages**: See [Getting Started](docs/GettingStarted.md)
|
|
- **Dataset creation pipelines & tools**: See [MakeDiffSinger](https://github.com/openvpi/MakeDiffSinger)
|
|
- **Best practices & tutorials**: See [Best Practices](docs/BestPractices.md)
|
|
- **Editing configurations**: See [Configuration Schemas](docs/ConfigurationSchemas.md)
|
|
- **Deployment & production**: [OpenUTAU](https://github.com/stakira/OpenUtau), [DiffScope (under development)](https://github.com/diffscope/diffscope-project)
|
|
- **Communication groups**: [QQ Group](http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=fibG_dxuPW5maUJwe9_ya5-zFcIwaoOR&authKey=ZgLCG5EqQVUGCID1nfKei8tCnlQHAmD9koxebFXv5WfUchhLwWxb52o1pimNai5A&noverify=0&group_code=907879266) (907879266), [Discord server](https://discord.gg/wwbu2JUMjj)
|
|
|
|
## Progress & Roadmap
|
|
|
|
- **Progress since we forked into this repository**: See [Releases](https://github.com/openvpi/DiffSinger/releases)
|
|
- **Roadmap for future releases**: See [Project Board](https://github.com/orgs/openvpi/projects/1)
|
|
- **Thoughts, proposals & ideas**: See [Discussions](https://github.com/openvpi/DiffSinger/discussions)
|
|
|
|
## Architecture & Algorithms
|
|
|
|
TBD
|
|
|
|
## Development Resources
|
|
|
|
TBD
|
|
|
|
## References
|
|
|
|
### Original Paper & Implementation
|
|
|
|
- Paper: [DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism](https://arxiv.org/abs/2105.02446)
|
|
- Implementation: [MoonInTheRiver/DiffSinger](https://github.com/MoonInTheRiver/DiffSinger)
|
|
|
|
### Generative Models & Algorithms
|
|
|
|
- Denoising Diffusion Probabilistic Models (DDPM): [paper](https://arxiv.org/abs/2006.11239), [implementation](https://github.com/hojonathanho/diffusion)
|
|
- [DDIM](https://arxiv.org/abs/2010.02502) for diffusion sampling acceleration
|
|
- [PNDM](https://arxiv.org/abs/2202.09778) for diffusion sampling acceleration
|
|
- [DPM-Solver++](https://github.com/LuChengTHU/dpm-solver) for diffusion sampling acceleration
|
|
- [UniPC](https://github.com/wl-zhao/UniPC) for diffusion sampling acceleration
|
|
- Rectified Flow (RF): [paper](https://arxiv.org/abs/2209.03003), [implementation](https://github.com/gnobitab/RectifiedFlow)
|
|
|
|
### Dependencies & Submodules
|
|
|
|
- [RoPE](https://github.com/lucidrains/rotary-embedding-torch) for transformer encoder
|
|
- [Mix-LN](https://www.isca-archive.org/interspeech_2024/hwang24_interspeech.pdf) for cross-speaker/lingual learning
|
|
- [HiFi-GAN](https://github.com/jik876/hifi-gan) and [NSF](https://github.com/nii-yamagishilab/project-NN-Pytorch-scripts/tree/master/project/01-nsf) for waveform reconstruction
|
|
- [pc-ddsp](https://github.com/yxlllc/pc-ddsp) for waveform reconstruction
|
|
- [RMVPE](https://github.com/Dream-High/RMVPE) and yxlllc's [fork](https://github.com/yxlllc/RMVPE) for pitch extraction
|
|
- [Vocal Remover](https://github.com/tsurumeso/vocal-remover) and yxlllc's [fork](https://github.com/yxlllc/vocal-remover) for harmonic-noise separation
|
|
|
|
## Disclaimer
|
|
|
|
Any organization or individual is prohibited from using any functionalities included in this repository to generate someone's speech without his/her consent, including but not limited to government leaders, political figures, and celebrities. If you do not comply with this item, you could be in violation of copyright laws.
|
|
|
|
## License
|
|
|
|
This forked DiffSinger repository is licensed under the [Apache 2.0 License](LICENSE).
|
|
|