chore: import upstream snapshot with attribution
Build documentation / build (push) Failing after 0s
CI / check_code_quality (push) Has been cancelled
CI / test (deps-latest, ubuntu-latest, integration) (push) Has been cancelled
CI / test (deps-latest, ubuntu-latest, unit) (push) Has been cancelled
CI / test (deps-latest, windows-latest, integration) (push) Has been cancelled
CI / test (deps-latest, windows-latest, unit) (push) Has been cancelled
CI / test (deps-minimum, ubuntu-latest, integration) (push) Has been cancelled
CI / test (deps-minimum, ubuntu-latest, unit) (push) Has been cancelled
CI / test (deps-minimum, windows-latest, integration) (push) Has been cancelled
CI / test (deps-minimum, windows-latest, unit) (push) Has been cancelled
CI / test_py314 (deps-latest, ubuntu-latest, unit) (push) Has been cancelled
CI / test_py314 (deps-latest, windows-latest, unit) (push) Has been cancelled
CI / test_py314_future (deps-latest, ubuntu-latest, unit) (push) Has been cancelled
CI / test_py314_future (deps-latest, windows-latest, unit) (push) Has been cancelled
Secret Leaks / trufflehog (push) Has been cancelled
Build documentation / build (push) Failing after 0s
CI / check_code_quality (push) Has been cancelled
CI / test (deps-latest, ubuntu-latest, integration) (push) Has been cancelled
CI / test (deps-latest, ubuntu-latest, unit) (push) Has been cancelled
CI / test (deps-latest, windows-latest, integration) (push) Has been cancelled
CI / test (deps-latest, windows-latest, unit) (push) Has been cancelled
CI / test (deps-minimum, ubuntu-latest, integration) (push) Has been cancelled
CI / test (deps-minimum, ubuntu-latest, unit) (push) Has been cancelled
CI / test (deps-minimum, windows-latest, integration) (push) Has been cancelled
CI / test (deps-minimum, windows-latest, unit) (push) Has been cancelled
CI / test_py314 (deps-latest, ubuntu-latest, unit) (push) Has been cancelled
CI / test_py314 (deps-latest, windows-latest, unit) (push) Has been cancelled
CI / test_py314_future (deps-latest, ubuntu-latest, unit) (push) Has been cancelled
CI / test_py314_future (deps-latest, windows-latest, unit) (push) Has been cancelled
Secret Leaks / trufflehog (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
# Loading methods
|
||||
|
||||
Methods for listing and loading datasets:
|
||||
|
||||
## Datasets
|
||||
|
||||
[[autodoc]] datasets.load_dataset
|
||||
|
||||
[[autodoc]] datasets.load_from_disk
|
||||
|
||||
[[autodoc]] datasets.load_dataset_builder
|
||||
|
||||
[[autodoc]] datasets.get_dataset_config_names
|
||||
|
||||
[[autodoc]] datasets.get_dataset_infos
|
||||
|
||||
[[autodoc]] datasets.get_dataset_split_names
|
||||
|
||||
## From files
|
||||
|
||||
Configurations used to load data files.
|
||||
They are used when loading local files or a dataset repository:
|
||||
|
||||
- local files: `load_dataset("parquet", data_dir="path/to/data/dir")`
|
||||
- dataset repository: `load_dataset("allenai/c4")`
|
||||
|
||||
You can pass arguments to `load_dataset` to configure data loading.
|
||||
For example you can specify the `sep` parameter to define the [`~datasets.packaged_modules.csv.CsvConfig`] that is used to load the data:
|
||||
|
||||
```python
|
||||
load_dataset("csv", data_dir="path/to/data/dir", sep="\t")
|
||||
```
|
||||
|
||||
### Text
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.text.TextConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.text.Text
|
||||
|
||||
### CSV
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.csv.CsvConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.csv.Csv
|
||||
|
||||
### JSON
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.json.JsonConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.json.Json
|
||||
|
||||
### XML
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.xml.XmlConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.xml.Xml
|
||||
|
||||
### Parquet
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.parquet.ParquetConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.parquet.Parquet
|
||||
|
||||
### Arrow
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.arrow.ArrowConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.arrow.Arrow
|
||||
|
||||
### SQL
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.sql.SqlConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.sql.Sql
|
||||
|
||||
### Images
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.imagefolder.ImageFolderConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.imagefolder.ImageFolder
|
||||
|
||||
### Audio
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.audiofolder.AudioFolderConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.audiofolder.AudioFolder
|
||||
|
||||
### Videos
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.videofolder.VideoFolderConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.videofolder.VideoFolder
|
||||
|
||||
### HDF5
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.hdf5.HDF5Config
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.hdf5.HDF5
|
||||
|
||||
### TsFile
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.tsfile.TsFileConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.tsfile.TsFile
|
||||
|
||||
### Pdf
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.pdffolder.PdfFolderConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.pdffolder.PdfFolder
|
||||
|
||||
### Nifti
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.niftifolder.NiftiFolderConfig
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.niftifolder.NiftiFolder
|
||||
|
||||
### WebDataset
|
||||
|
||||
[[autodoc]] datasets.packaged_modules.webdataset.WebDataset
|
||||
Reference in New Issue
Block a user