Seamlessly integrate your postgres data with Airfold
Change Data Capture
(CDC), powered by PostgreSQL’s Logical Replication
, to track row-level inserts and updates as they happen. This means that you don’t need to configure a refresh schedule for your source. As data is inserted or updated in Postgres, it’s automatically ingested by Airfold on the fly.
wal_level
in your postgres server. In order for CDC to function, the wal_level
(Write-Ahead Logging) setting on your Postgres server must be set to logical
.
To check your current setting, run this query:
minimal
, replica
, or logical
.
Most servers default to replica
, but many modern managed Postgres providers (like Neon or Supabase) default to logical
.
If you need to change this setting, you’ll typically do so in the postgresql.conf
file, or via your provider’s UI, depending on how your Postgres instance is hosted.
role
to connect to your server. Run the following SQL to get set up:
publication
and a replication slot
.
The publication is a Postgres object that defines which tables to stream changes from. You can choose to publish specific tables or all tables in the database.
The replication slot is a mechanism that ensures Postgres retains the change data (WAL logs) until Airfold has consumed it. This guarantees that no changes are lost, even if there’s a temporary interruption in the connection.
To create the Publication
, run this SQL:
<publication_name>
is the name you want to give your new publication, and it should be followed by a comma-separated list of the tables you want to include in the sync.
Finally we can create the Slot:
Host
- The hostname of your Postgres server (e.g., db.mycompany.com
)Port
- The port your server is listening on (default is 5432
)User
- The name of the role created in previous stepsPassword
- Of the above roleDatabase
- The name of the database you are connecting toPublication
- The name of the publication created in previous stepsSlot
- The name of the Slot created in previous stepsSources
on the left side tool bar and Click the +
button.
In the popup window, click on Postgres
Use SSL
(optional): This setting enables SSL encryption for the connection between Airfold and your PostgreSQL server. When SSL is enabled, all data transmitted between Airfold and the server is encrypted, protecting sensitive information, protecting all of your data. SSL is strongly recommended for production environments!
💡 Tip:After finalizing your schema and settings, click
It’s worth spending some time on this section as both the schema and table settings cannot be changed once the table is created. Choosing optimal data types and table settings (like theprimary key
,order by
, andengine
) are crucial steps in Airfold for ensuring that your queries run with low latency. If you wish to change data types or table settings, you will have to drop the source, recreate the table with the new settings, and ingest your data again. This could be a potentially costly backfill, especially if you have a large dataset, so this is a great opportunity to slow down and get these settings right the first time! Check out the page on optimizing your schema
Create
. Your new source should now be available and data ingestion will begin momentarily!