By using Git repositories and YAML files, you can apply familiar engineering workflows like pull requests, branching, and CI/CD to your projects.

Airfold workspaces are fully defined in YAML files, which enables you to leverage version control and CI/CD similar to tools like dbt, Terraform, and other infrastructure as code systems.

Your files represent data pipelines as code with Airfold.

​
Why use Version Control?

Restore Workspace History

Each commit represents the exact state of your Airfold workspace at a given point in time.

Easily navigate to and restore any previous version of the workspace to:

  • Revert breaking changes
  • Recover pipeline configurations or source definitions that were accidentally modified

Roll Back Issues

If a recent change causes unexpected errors, simply roll back to the last known working commit.

This is helpful for avoiding downtime in production by reverting to a stable state with minimal effort.

Enable Collaboration

Git enables multiple team members to work on the same workspace simultaneously without conflicts:

  • Avoid overwriting others’ changes (a common risk when using a shared UI), where actions like multiple users pressing save on a query edit can accidentally overwrite each other’s changes
  • Git merge enables a structured way to combine contributions, reducing the risk of such conflicts and ensuring every update is reviewed

Seamless Sharing

Within your organization:

  • Store your Airfold workspace files in a shared Git repository, ensuring all team members access to latest udpates
  • Team members can work on individual tasks in separate branches, avoiding conflicts with ongoing production or staging workflows
  • Use pull requests to propose changes and review code before merging

You can also open-source your Airfold workspaces just as you would with any Git repository, allowing the broader community to reuse and adapt your workspaces.

Branch for Production and Staging

Maintain separate branches for staging and production environments:

  • Use staging for testing new pipelines, changes, or integrations
  • Deploy only validated changes to production

​
File Structure

A typical Airfold project is organized into the following directory structure:

/.airfold/config.yaml          # Configuration for CLI and workspace
/pipes                         # YAML files for pipeline definitions
  /trends_per_country.yaml
  /sales_metrics.yaml
/sources                       # YAML files for data sources and materialized views
  /events.yaml
  /sales.yaml
  /sales_metrics_mv.yaml
/endpoints                     # (Optional) Organized API endpoint definitions
  /country_trends.yaml
  /sales_over_time.yaml

Pipes: Define the sequence of SQL transformations in your data pipeline

  • Example: /pipes/sales_metrics.yaml
  • See more on pipe.yaml

Sources: Define data sources

Endpoints: (Optional) Define API endpoints for serving pipeline results.

  • Example: /endpoints/sales_over_time.yaml

​
Workflow

​
Initialize

From your project directory, use Git to initialize version control:

git init

​
Commit & Push

Commit existing workspace files to the repository:

git add .
git commit -m "Initial commit of Airfold workspace files"

Push your repository:

git remote add origin <repository_url>
git push -u origin main

​
Branch for Staging and Production

Create a branch for staging:

git branch staging

Use this staging branch to test new changes:

git checkout staging

Once changes are validated, merge into production:

git checkout main
git merge staging

​
Best Practices

Use pull requests for any workspace updates:

  • Enable code reviews before merging changes

Separate branches for staging and production:

  • main for production-ready workspace configurations
  • staging for test configurations
  • See more on Staging and Production
Staging & ProductionOptimize Queries