CI/CD

Airfold workspaces are fully defined in YAML files, which enables you to leverage version control and CI/CD similar to tools like dbt, Terraform, and other infrastructure as code systems.

Your files represent data pipelines as code with Airfold.

​

How CI/CD Works

CI/CD (Continuous Integration and Continuous Deployment) automates the process of testing, validating, and deploying changes to your projects.

In a typical CI/CD workflow:

• Continuous Integration (CI) ensures that changes, such as updates to pipelines or configurations, are tested and validated automatically whenever new code is pushed or a pull request is opened
• Continuous Deployment (CD) automates the deployment of validated changes to staging or production environments

​

Why Use CI/CD?

Implementing CI/CD for Airfold workspaces is highly recommended for production systems as it enables you to:

• Automate the process of testing, validating, and deploying changes to Airfold pipelines
• Track changes by leveraging version control with your CI/CD pipelines
• Maintain consistency across various environments (e.g., staging and production)

​

GitHub Actions Example

GitHub Actions is one way to integrate Airfold with CI/CD pipelines.

Let’s go through an example of setting up a deployment workflow for Airlang using GitHub Actions:

name: Airfold deploy

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
  workflow_dispatch:

permissions:
  id-token: write
  contents: write
  pull-requests: write

jobs:
  airfold:
    name: Plan / Apply
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          fetch-depth: 0
          submodules: recursive
          token: ${{ secrets.DEPLOY_TOKEN }}

      - name: Set up Python 3.11
        uses: actions/setup-python@v4
        with:
            python-version: 3.11

      - name: Setup Airfold CLI
        uses: airfold/setup-airfold@v1
        with:
          api-token: ${{ secrets.AIRFOLD_API_KEY }}

      - name: Airfold diff
        id: diff
        if: github.event_name == 'pull_request'
        run: |
          af diff airfold
        continue-on-error: true

      - name: Airfold plan
        id: plan
        if: github.event_name == 'pull_request'
        run: |
          af push airfold --dry-run
        continue-on-error: true

      - name: Airfold Status
        if: steps.plan.outcome == 'failure'
        run: exit 1

      - name: Airfold push and apply
        if: (github.ref == 'refs/heads/main' && github.event_name == 'push') || github.event_name == 'workflow_dispatch'
        run: af push airfold

​

on

push: Runs the workflow whenever changes are pushed to the main branch

pull_request: Executes the workflow when a pull request is opened or updated for the main branch

workflow_dispatch: Allows the workflow to be triggered manually from the Actions tab in GitHub

​

permissions

Grants the workflow specific permissions:

• id-token: Required for OIDC authentication
• contents: Allows the workflow to read/write repository contents
• pull-requests: Enables the workflow to comment on pull requests

​

jobs

jobs: Defines a single job named Plan / Apply

runs-on: ubuntu-latest: Specifies the environment where the job will run

1. Checkout Repository

uses: actions/checkout@v3
with:
  fetch-depth: 0
  submodules: recursive
  token: ${{ secrets.DEPLOY_TOKEN }}

• Clones the repository, fetching all commit history and submodules
• Authenticates using a secret DEPLOY_TOKEN

2. Set Up Python 3.11

uses: actions/setup-python@v4
with:
    python-version: 3.11

• Installs and configures Python 3.11 for the workflow

3. Setup Airfold CLI

uses: airfold/setup-airfold@v1
with:
  api-token: ${{ secrets.AIRFOLD_API_KEY }}

• Installs the Airfold CLI and configures it using the API key

4. Airfold diff

id: diff
if: github.event_name == 'pull_request'
run: |
  af diff airfold
continue-on-error: true

• Runs af diff to compare the current and previous states of the Airfold configuration
• Executes only for pull requests
• continue-on-error: true: Prevents failures in this step from stopping the workflow

5. Airfold plan

id: plan
if: github.event_name == 'pull_request'
run: |
  af push airfold --dry-run
continue-on-error: true

• Performs a dry run of af push to simulate changes

6. Airfold Status

if: steps.plan.outcome == 'failure'
run: exit 1

• Stops the workflow if the plan step fails, ensuring only valid changes proceed

7. Airfold push and apply

if: (github.ref == 'refs/heads/main' && github.event_name == 'push') || github.event_name == 'workflow_dispatch'
run: af push airfold

Pushes changes to Airfold workspace when:

• Code is pushed to main
• The workflow is triggered manually through workflow_dispatch

​

GitHub Actions

GitHub Actions enable you to define workflows as code using YAML configuration files in your repository.

High-level overview of a GitHub Actions workflow:

1. Define a Workflow

2. Specify Triggers

3. Define Jobs

See more on the GitHub Actions Documentation.