.. _loading-data-api:

Loading Data API
================

.. currentmodule:: ray.data

Public APIs
-----------

Arrow
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_arrow

Audio
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_audio

Avro
^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_avro

BigQuery
^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_bigquery

Binary
^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_binary_files

Catalog
^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   Catalog

   DatabricksUnityCatalog

.. autosummary::
   :nosignatures:
   :toctree: doc/
   :template: autosummary/class_without_autosummary.rst

   ReaderFormat

CSV
^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_csv

ClickHouse
^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_clickhouse

Daft
^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_daft

Dask
^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_dask

Databricks
^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_databricks_tables

Delta Lake
^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_delta

Delta Sharing
^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_delta_sharing_tables

Hudi
^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_hudi

Hugging Face
^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_huggingface

Iceberg
^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_iceberg

Images
^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_images

JSON
^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_json

Kafka
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_kafka

Lance
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_lance

MCAP (Message Capture)
^^^^^^^^^^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_mcap

Mars
^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_mars

Modin
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_modin

MongoDB
^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_mongo

NumPy
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_numpy
   read_numpy

Pandas
^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_pandas

Parquet
^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_parquet

Python Objects
^^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_items

SQL Databases
^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_sql

Snowflake
^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_snowflake

Spark
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_spark

Synthetic Data
^^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   range
   range_tensor

TFRecords
^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_tfrecords

TensorFlow
^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_tf

Text
^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_text

Torch
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_torch

Unity Catalog
^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_unity_catalog

Video
^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_videos

WebDataset
^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_webdataset

Zarr
^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   read_zarr

Partitioning API
^^^^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   datasource.Partitioning

.. autosummary::
   :nosignatures:
   :toctree: doc/
   :template: autosummary/class_without_autosummary.rst

   datasource.PartitionStyle

.. autosummary::
   :nosignatures:
   :toctree: doc/

   datasource.PathPartitionFilter
   datasource.PathPartitionParser

Shuffling API
^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   FileShuffleConfig

Developer APIs
--------------

Arrow refs
^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_arrow_refs

Datasource API
^^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   Datasource
   datasource.FileBasedDatasource
   read_datasource
   ReadTask

.. _metadata_provider:

MetadataProvider API
^^^^^^^^^^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   datasource.BaseFileMetadataProvider
   datasource.DefaultFileMetadataProvider
   datasource.FileMetadataProvider

NumPy refs
^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_numpy_refs

Pandas refs
^^^^^^^^^^^

.. autosummary::
   :nosignatures:
   :toctree: doc/

   from_pandas_refs