Data is ingested into sources directly through Airfold’s Ingest API or by a pipe.

​
Creation

Define the schema for your source in a YAML file:

sales_calls.yaml
name: sales_calls
description: sales call transcripts
cols:
  ID: UUID
  Transcript: String
settings:
  engine: MergeTree()
  order_by: '`ID`'
  partition_by: tuple()

​
Properties

​
type
string

Type of source: Table or AITable, see more on AI table (Optional)

​
name
string

Name of source

​
description
string

A brief overview of the source’s content or purpose (optional)

​
cols
{name: type}
required

Defines the schema of the source as a list of columns where each column name is a key and its data type is the value (see Data Types)

​
settings
string | {key: value} | array

ClickHouse table configuration for the source, which can include ORDER BY, PARTITION BY, Table Engine, etc. The settings can be a String, a key-value pair, or an array comprising either or both. Optional.

​
Push

Push sources to your workspace using the CLI command or API.

​
CLI

For example, to push sales_calls.yaml, run:

af push sales_calls.yaml

​
Ingest API

We can also use Ingest API:

The Ingest API supports the following formats:

  • NDJSON
  • CSV
  • Parquet

For example, ingesting NDJSON is as simple as:

curl --request POST \
  --url https://api.airfold.co/v1/sales_analysis/{source} \
  --header 'Authorization: Bearer <key>' \
  --header 'Content-Type: application/json' \
  --data '[
  { "id": 123, "name": "Sarah"},
  { "id": 456, "name": "Alex"}
]'

Rate Limits

The Ingest API is rate limited to 1000 events/sec per source. If you need to ingest more events, please contact us.

​
_errors Table

Every workspace has an associated _errors source table, which contains all errors that occurred during ingestion (e.g.: if a row failed to be ingested due to schema mismatch).

This way, the whole ingestion process is uninterrupted and the error can be fixed later.

_errors is treated like any other source, so it can be queried, filtered, and joined with other sources.

For example, we can find the most common errors by source:

SELECT source, COUNT(*) AS count
FROM _errors
GROUP BY source
ORDER BY count DESC