Airfold workspaces are fully defined in YAML files, which enables you to leverage version control and CI/CD similar to tools like dbt, Terraform, and other infrastructure as code systems.

Your files represent data pipelines as code with Airfold.

How CI/CD Works

CI/CD (Continuous Integration and Continuous Deployment) automates the process of testing, validating, and deploying changes to your projects.

In a typical CI/CD workflow:

  • Continuous Integration (CI) ensures that changes, such as updates to pipelines or configurations, are tested and validated automatically whenever new code is pushed or a pull request is opened
  • Continuous Deployment (CD) automates the deployment of validated changes to staging or production environments

Why Use CI/CD?

Implementing CI/CD for Airfold workspaces is highly recommended for production systems as it enables you to:

  • Automate the process of testing, validating, and deploying changes to Airfold pipelines
  • Track changes by leveraging version control with your CI/CD pipelines
  • Maintain consistency across various environments (e.g., staging and production)

GitHub Actions Example

GitHub Actions is one way to integrate Airfold with CI/CD pipelines.

Let’s go through an example of setting up a deployment workflow for Airlang using GitHub Actions:

deploy.yaml
name: Airfold deploy

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
  workflow_dispatch:

permissions:
  id-token: write
  contents: write
  pull-requests: write

jobs:
  airfold:
    name: Plan / Apply
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
          submodules: recursive
          token: ${{ secrets.DEPLOY_TOKEN }}

      - name: Set up Python 3.11
        uses: actions/setup-python@v4
        with:
            python-version: 3.11

      - name: Setup Airfold CLI
        uses: airfold/setup-airfold@v1
        with:
          api-token: ${{ secrets.AIRFOLD_API_KEY }}

      - name: Airfold diff
        id: diff
        if: github.event_name == 'pull_request'
        run: |
          af diff airfold
        continue-on-error: true

      - name: Airfold plan
        id: plan
        if: github.event_name == 'pull_request'
        run: |
          af push airfold --dry-run
        continue-on-error: true

      - name: Airfold Status
        if: steps.plan.outcome == 'failure'
        run: exit 1

      - name: Airfold push and apply
        if: (github.ref == 'refs/heads/main' && github.event_name == 'push') || github.event_name == 'workflow_dispatch'
        run: af push airfold

on

push: Runs the workflow whenever changes are pushed to the main branch

pull_request: Executes the workflow when a pull request is opened or updated for the main branch

workflow_dispatch: Allows the workflow to be triggered manually from the Actions tab in GitHub

permissions

Grants the workflow specific permissions:

  • id-token: Required for OIDC authentication
  • contents: Allows the workflow to read/write repository contents
  • pull-requests: Enables the workflow to comment on pull requests

jobs

jobs: Defines a single job named Plan / Apply

runs-on: ubuntu-latest: Specifies the environment where the job will run

1. Checkout Repository

uses: actions/checkout@v3
with:
  fetch-depth: 0
  submodules: recursive
  token: ${{ secrets.DEPLOY_TOKEN }}
  • Clones the repository, fetching all commit history and submodules
  • Authenticates using a secret DEPLOY_TOKEN

2. Set Up Python 3.11

uses: actions/setup-python@v4
with:
    python-version: 3.11
  • Installs and configures Python 3.11 for the workflow

3. Setup Airfold CLI

uses: airfold/setup-airfold@v1
with:
  api-token: ${{ secrets.AIRFOLD_API_KEY }}
  • Installs the Airfold CLI and configures it using the API key

4. Airfold diff

id: diff
if: github.event_name == 'pull_request'
run: |
  af diff airfold
continue-on-error: true
  • Runs af diff to compare the current and previous states of the Airfold configuration
  • Executes only for pull requests
  • continue-on-error: true: Prevents failures in this step from stopping the workflow

5. Airfold plan

id: plan
if: github.event_name == 'pull_request'
run: |
  af push airfold --dry-run
continue-on-error: true
  • Performs a dry run of af push to simulate changes

6. Airfold Status

if: steps.plan.outcome == 'failure'
run: exit 1
  • Stops the workflow if the plan step fails, ensuring only valid changes proceed

7. Airfold push and apply

if: (github.ref == 'refs/heads/main' && github.event_name == 'push') || github.event_name == 'workflow_dispatch'
run: af push airfold

Pushes changes to Airfold workspace when:

  • Code is pushed to main
  • The workflow is triggered manually through workflow_dispatch

GitHub Actions

GitHub Actions enable you to define workflows as code using YAML configuration files in your repository.

High-level overview of a GitHub Actions workflow:

  1. Define a Workflow

  2. Specify Triggers

  3. Define Jobs

See more on the GitHub Actions Documentation.

Airfold workspaces are fully defined in YAML files, which enables you to leverage version control and CI/CD similar to tools like dbt, Terraform, and other infrastructure as code systems.

Your files represent data pipelines as code with Airfold.

How CI/CD Works

CI/CD (Continuous Integration and Continuous Deployment) automates the process of testing, validating, and deploying changes to your projects.

In a typical CI/CD workflow:

  • Continuous Integration (CI) ensures that changes, such as updates to pipelines or configurations, are tested and validated automatically whenever new code is pushed or a pull request is opened
  • Continuous Deployment (CD) automates the deployment of validated changes to staging or production environments

Why Use CI/CD?

Implementing CI/CD for Airfold workspaces is highly recommended for production systems as it enables you to:

  • Automate the process of testing, validating, and deploying changes to Airfold pipelines
  • Track changes by leveraging version control with your CI/CD pipelines
  • Maintain consistency across various environments (e.g., staging and production)

GitHub Actions Example

GitHub Actions is one way to integrate Airfold with CI/CD pipelines.

Let’s go through an example of setting up a deployment workflow for Airlang using GitHub Actions:

deploy.yaml
name: Airfold deploy

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
  workflow_dispatch:

permissions:
  id-token: write
  contents: write
  pull-requests: write

jobs:
  airfold:
    name: Plan / Apply
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
          submodules: recursive
          token: ${{ secrets.DEPLOY_TOKEN }}

      - name: Set up Python 3.11
        uses: actions/setup-python@v4
        with:
            python-version: 3.11

      - name: Setup Airfold CLI
        uses: airfold/setup-airfold@v1
        with:
          api-token: ${{ secrets.AIRFOLD_API_KEY }}

      - name: Airfold diff
        id: diff
        if: github.event_name == 'pull_request'
        run: |
          af diff airfold
        continue-on-error: true

      - name: Airfold plan
        id: plan
        if: github.event_name == 'pull_request'
        run: |
          af push airfold --dry-run
        continue-on-error: true

      - name: Airfold Status
        if: steps.plan.outcome == 'failure'
        run: exit 1

      - name: Airfold push and apply
        if: (github.ref == 'refs/heads/main' && github.event_name == 'push') || github.event_name == 'workflow_dispatch'
        run: af push airfold

on

push: Runs the workflow whenever changes are pushed to the main branch

pull_request: Executes the workflow when a pull request is opened or updated for the main branch

workflow_dispatch: Allows the workflow to be triggered manually from the Actions tab in GitHub

permissions

Grants the workflow specific permissions:

  • id-token: Required for OIDC authentication
  • contents: Allows the workflow to read/write repository contents
  • pull-requests: Enables the workflow to comment on pull requests

jobs

jobs: Defines a single job named Plan / Apply

runs-on: ubuntu-latest: Specifies the environment where the job will run

1. Checkout Repository

uses: actions/checkout@v3
with:
  fetch-depth: 0
  submodules: recursive
  token: ${{ secrets.DEPLOY_TOKEN }}
  • Clones the repository, fetching all commit history and submodules
  • Authenticates using a secret DEPLOY_TOKEN

2. Set Up Python 3.11

uses: actions/setup-python@v4
with:
    python-version: 3.11
  • Installs and configures Python 3.11 for the workflow

3. Setup Airfold CLI

uses: airfold/setup-airfold@v1
with:
  api-token: ${{ secrets.AIRFOLD_API_KEY }}
  • Installs the Airfold CLI and configures it using the API key

4. Airfold diff

id: diff
if: github.event_name == 'pull_request'
run: |
  af diff airfold
continue-on-error: true
  • Runs af diff to compare the current and previous states of the Airfold configuration
  • Executes only for pull requests
  • continue-on-error: true: Prevents failures in this step from stopping the workflow

5. Airfold plan

id: plan
if: github.event_name == 'pull_request'
run: |
  af push airfold --dry-run
continue-on-error: true
  • Performs a dry run of af push to simulate changes

6. Airfold Status

if: steps.plan.outcome == 'failure'
run: exit 1
  • Stops the workflow if the plan step fails, ensuring only valid changes proceed

7. Airfold push and apply

if: (github.ref == 'refs/heads/main' && github.event_name == 'push') || github.event_name == 'workflow_dispatch'
run: af push airfold

Pushes changes to Airfold workspace when:

  • Code is pushed to main
  • The workflow is triggered manually through workflow_dispatch

GitHub Actions

GitHub Actions enable you to define workflows as code using YAML configuration files in your repository.

High-level overview of a GitHub Actions workflow:

  1. Define a Workflow

  2. Specify Triggers

  3. Define Jobs

See more on the GitHub Actions Documentation.