chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,23 @@
|
||||
# `Table::clear` and `Table::replace`
|
||||
|
||||
Calling `Table::clear` will remove all data from the underlying `Table`. Calling
|
||||
`Table::replace` with new data will clear the `Table`, and update it with a new
|
||||
dataset that conforms to Perspective's data types and the existing schema on the
|
||||
`Table`.
|
||||
|
||||
<div class="javascript">
|
||||
|
||||
```javascript
|
||||
table.clear();
|
||||
table.replace(json);
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="python">
|
||||
|
||||
```python
|
||||
table.clear()
|
||||
table.replace(df)
|
||||
```
|
||||
|
||||
</div>
|
||||
@@ -0,0 +1,47 @@
|
||||
# Construct a Table
|
||||
|
||||
Examples of constructing an empty `Table` from a schema.
|
||||
|
||||
<div class="javascript">
|
||||
|
||||
JavaScript:
|
||||
|
||||
```javascript
|
||||
var schema = {
|
||||
x: "integer",
|
||||
y: "string",
|
||||
z: "boolean",
|
||||
};
|
||||
|
||||
const table2 = await worker.table(schema);
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="python">
|
||||
|
||||
Python:
|
||||
|
||||
```python
|
||||
from datetime import date, datetime
|
||||
|
||||
schema = {
|
||||
"x": "integer",
|
||||
"y": "string",
|
||||
"z": "boolean",
|
||||
}
|
||||
|
||||
table2 = perspective.table(schema)
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="rust">
|
||||
|
||||
Rust:
|
||||
|
||||
```rust
|
||||
let data = TableData::Schema(vec![(" a".to_string(), ColumnType::FLOAT)]);
|
||||
let options = TableInitOptions::default();
|
||||
let table = client.table(data.into(), options).await?;
|
||||
```
|
||||
|
||||
</div>
|
||||
@@ -0,0 +1,87 @@
|
||||
# Loading data
|
||||
|
||||
A `Table` may also be created-or-updated by data in CSV,
|
||||
[Apache Arrow](https://arrow.apache.org/), JSON row-oriented or JSON
|
||||
column-oriented formats. In addition to these, `perspective-python` additionally
|
||||
supports `pyarrow.Table`, `polars.DataFrame` and `pandas.DataFrame` objects
|
||||
directly. These formats are otherwise identical to the built-in formats and
|
||||
don't exhibit any additional support or type-awareness; e.g., `pandas.DataFrame`
|
||||
support is _just_ `pyarrow.Table.from_pandas` piped into Perspective's Arrow
|
||||
reader.
|
||||
|
||||
`Client::table` and `Table::update` perform _coercion_ on their input for all
|
||||
input formats _except_ Arrow (which comes with its own schema and has no need
|
||||
for coercion). `"date"` and `"datetime"` column types do not have native JSON
|
||||
representations, so these column types _cannot_ be inferred from JSON input.
|
||||
Instead, for columns of these types for JSON input, a `Table` must first be
|
||||
constructed with a _schema_. Next, call `Table::update` with the JSON input -
|
||||
Perspective's JSON reader may _coerce_ a `date` or `datetime` from these native
|
||||
JSON types:
|
||||
|
||||
- `integer` as milliseconds-since-epoch.
|
||||
- `string` as a any of Perspective's built-in date format formats.
|
||||
- JavaScript `Date` and Python `datetime.date` and `datetime.datetime` are _not_
|
||||
supported directly. However, in JavaScript `Date` types are automatically
|
||||
coerced to correct `integer` timestamps by default when converted to JSON.
|
||||
|
||||
## Apache Arrow
|
||||
|
||||
The most efficient way to load data into Perspective, encoded as
|
||||
[Apache Arrow IPC format](https://arrow.apache.org/docs/python/ipc.html). In
|
||||
JavaScript:
|
||||
|
||||
```javascript
|
||||
const resp = await fetch(
|
||||
"https://cdn.jsdelivr.net/npm/superstore-arrow/superstore.lz4.arrow",
|
||||
);
|
||||
|
||||
const arrow = await resp.arrayBuffer();
|
||||
```
|
||||
|
||||
Apache Arrow input do not support type coercion, preferring Arrow's internal
|
||||
self-describing schema.
|
||||
|
||||
## CSV
|
||||
|
||||
Perspective relies on Apache Arrow's CSV parser, and as such uses mostly the
|
||||
same column-type inference logic as Arrow itself would use for parsing CSV.
|
||||
|
||||
## Row Oriented JSON
|
||||
|
||||
Row-oriented JSON is in the form of a list of objects. Each object in the list
|
||||
corresponds to a row in the table. For example:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "a": 86, "b": false, "c": "words" },
|
||||
{ "a": 0, "b": true, "c": "" },
|
||||
{ "a": 12345, "b": false, "c": "here" }
|
||||
]
|
||||
```
|
||||
|
||||
## Column Oriented JSON
|
||||
|
||||
Column-Oriented JSON comes in the form of an object of lists. Each key of the
|
||||
object is a column name, and each element of the list is the corresponding value
|
||||
in the row.
|
||||
|
||||
```json
|
||||
{
|
||||
"a": [86, 0, 12345],
|
||||
"b": [false, true, false],
|
||||
"c": ["words", "", "here"]
|
||||
}
|
||||
```
|
||||
|
||||
## NDJSON
|
||||
|
||||
[NDJSON](https://github.com/ndjson/ndjson-spec) (sometimes also referred to as
|
||||
JSONL) is a streaming-friendly format where each line is a valid JSON object,
|
||||
separated by newlines. It is commonly used in data streaming and messaging
|
||||
queues.
|
||||
|
||||
```json
|
||||
{ "a": 86, "b": false, "c": "words" }
|
||||
{ "a": 0, "b": true, "c": "" }
|
||||
{ "a": 12345, "b": false, "c": "here" }
|
||||
```
|
||||
@@ -0,0 +1,59 @@
|
||||
## Index and Limit
|
||||
|
||||
<div class="warning">`limit` cannot be used in conjunction with `index`.</div>
|
||||
|
||||
Initializing a `Table` with an `index` tells Perspective to treat a column as
|
||||
the primary key, allowing in-place updates of rows. Only a single column (of any
|
||||
type) can be used as an `index`. Indexed `Table` instances allow:
|
||||
|
||||
- In-place _updates_ whenever a new row shares an `index` values with an
|
||||
existing row
|
||||
- _Partial updates_ when a data batch omits some column.
|
||||
- _Removes_ to delete a row by `index`.
|
||||
|
||||
To create an indexed `Table`, provide the `index` property with a string column
|
||||
name to be used as an index:
|
||||
|
||||
<div class="javascript">
|
||||
|
||||
JavaScript:
|
||||
|
||||
```javascript
|
||||
const indexed_table = await perspective.table(data, { index: "a" });
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="python">
|
||||
|
||||
Python
|
||||
|
||||
```python
|
||||
indexed_table = perspective.Table(data, index="a");
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
Initializing a `Table` with a `limit` sets the total number of rows the `Table`
|
||||
is allowed to have. When the `Table` is updated, and the resulting size of the
|
||||
`Table` would exceed its `limit`, rows that exceed `limit` overwrite the oldest
|
||||
rows in the `Table`. To create a `Table` with a `limit`, provide the `limit`
|
||||
property with an integer indicating the maximum rows:
|
||||
|
||||
<div class="javascript">
|
||||
|
||||
JavaScript:
|
||||
|
||||
```javascript
|
||||
const limit_table = await perspective.table(data, { limit: 1000 });
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="python">
|
||||
|
||||
Python:
|
||||
|
||||
```python
|
||||
limit_table = perspective.Table(data, limit=1000);
|
||||
```
|
||||
|
||||
</div>
|
||||
@@ -0,0 +1,41 @@
|
||||
# Schema and column types
|
||||
|
||||
The mapping of a `Table`'s column names to data types is referred to as a
|
||||
`schema`. Each column has a unique name and a single data type, one of
|
||||
|
||||
- `float`
|
||||
- `integer`
|
||||
- `boolean`
|
||||
- `date`
|
||||
- `datetime`
|
||||
- `string`
|
||||
|
||||
A `Table` schema is fixed at construction, either by explicitly passing a schema
|
||||
dictionary to the `Client::table` method, or by passing _data_ to this method
|
||||
from which the schema is _inferred_ (if CSV or JSON format) or inherited (if
|
||||
Arrow).
|
||||
|
||||
## Type inference
|
||||
|
||||
When passing CSV or JSON data to the `Client::table` constructor, the type of
|
||||
each column is inferred automatically. In some cases, the inference algorithm
|
||||
may not return exactly what you'd like. For example, a column may be interpreted
|
||||
as a `datetime` when you intended it to be a `string`, or a column may have no
|
||||
values at all (yet), as it will be updated with values from a real-time data
|
||||
source later on. In these cases, create a `table()` with a _schema_.
|
||||
|
||||
Once the `Table` has been created, further `Table::update` calls will perform
|
||||
limited type _coercion_ based on the schema. While _coercion_ works similarly to
|
||||
_inference_, in that input data may be parsed based on the expected column type,
|
||||
`Table::update` will not _change_ the column's type further. For example, a
|
||||
number literal `1234` would be _inferred_ as an `"integer"`, but _in the context
|
||||
of an `Table::update` call on a known `"string"` column_, this will be parsed as
|
||||
the _string_ `"1234"`.
|
||||
|
||||
## `date` and `datetime` inference
|
||||
|
||||
Various string representations of `date` and `datetime` format columns can be
|
||||
_inferred_ as well _coerced_ from strings if they match one of Perspective's
|
||||
internal known datetime parsing formats, for example
|
||||
[ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) (which is also the format
|
||||
Perspective will _output_ these types for CSV).
|
||||
@@ -0,0 +1,88 @@
|
||||
# `Table::update` and `Table::remove`
|
||||
|
||||
Once a `Table` has been created, it can be updated with new data conforming to
|
||||
the `Table`'s schema. `Table::update` supports the same data formats as
|
||||
`Client::table`, minus _schema_.
|
||||
|
||||
<div class="javascript">
|
||||
|
||||
```javascript
|
||||
const schema = {
|
||||
a: "integer",
|
||||
b: "float",
|
||||
};
|
||||
|
||||
const table = await perspective.table(schema);
|
||||
table.update(new_data);
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="python">
|
||||
|
||||
```python
|
||||
schema = {"a": "integer", "b": "float"}
|
||||
|
||||
table = perspective.Table(schema)
|
||||
table.update(new_data)
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
Without an `index` set, calls to `update()` _append_ new data to the end of the
|
||||
`Table`. Otherwise, Perspective allows
|
||||
[_partial updates_ (in-place)](#index-and-limit) using the `index` to determine
|
||||
which rows to update:
|
||||
|
||||
<div class="javascript">
|
||||
|
||||
```javascript
|
||||
indexed_table.update({ id: [1, 4], name: ["x", "y"] });
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="python">
|
||||
|
||||
```python
|
||||
indexed_table.update({"id": [1, 4], "name": ["x", "y"]})
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
Any value on a `Client::table` can be unset using the value `null` in JSON or
|
||||
Arrow input formats. Values may be unset on construction, as any `null` in the
|
||||
dataset will be treated as an unset value. `Table::update` calls do not need to
|
||||
provide _all columns_ in the `Table`'s schema; missing columns will be omitted
|
||||
from the `Table`'s updated rows.
|
||||
|
||||
<div class="javascript">
|
||||
|
||||
```javascript
|
||||
table.update([{ x: 3, y: null }]); // `z` missing
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="python">
|
||||
|
||||
```python
|
||||
table.update([{"x": 3, "y": None}]) # `z` missing
|
||||
```
|
||||
|
||||
</div>
|
||||
|
||||
Rows can also be removed from an indexed `Table`, by calling `Table::remove`
|
||||
with an array of index values:
|
||||
|
||||
<div class="javascript">
|
||||
|
||||
```javascript
|
||||
indexed_table.remove([1, 4]);
|
||||
```
|
||||
|
||||
</div>
|
||||
<div class="python">
|
||||
|
||||
```python
|
||||
indexed_table.remove([1, 4])
|
||||
```
|
||||
|
||||
</div>
|
||||
Reference in New Issue
Block a user