Vai al contenuto principale

CI/CD for Ilum Interactive Services with GitHub Actions

This guide demonstrates how to automate the deployment of Ilum Interactive Services Utilizzando GitHub Actions with self-hosted runners running directly on your Kubernetes cluster. By combining the Ilum REST API with a CI/CD pipeline, you can deploy and update services automatically on every push or merge.

Quick Summary
  • Self-Hosted Runners: Deploy GitHub Actions Runner Controller (ARC) on Kubernetes so workflows execute inside your cluster with direct access to Ilum.
  • Service Code in Git: Store your Ilum service implementation (service.py) in a GitHub repository.
  • Automated Deployment: A GitHub Actions workflow creates/updates Ilum service groups via the REST API on every push to principale.
  • Update via Pull Requests: Modify service logic in a feature branch, open a PR, merge — and the service is automatically redeployed with the new code.

Prerequisiti​

  • A running Ilum instance on Kubernetes (with all pods healthy)
  • kubectl e helm configured for your cluster
  • A GitHub account with permissions to create repositories and personal access tokens
  • cert-manager installed on the cluster

Step 1: Install cert-manager​

Actions Runner Controller requires cert-manager for TLS certificate management. Install it if not already present:

Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml

Wait for all cert-manager pods to be ready:

Wait for cert-manager
kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s

Step 2: Create a GitHub Personal Access Token​

The self-hosted runner needs a GitHub token to register itself with your repository.

  1. Navigate to GitHub → Settings → Developer Settings → Personal access tokens → Tokens (classic).
  2. Clic Generate new token (classic).
  3. Set a descriptive name (e.g., ServiceManagementToken).
  4. Select the repo scope (full control of private repositories).
  5. Clic Generate token and copy the value immediately — you won't see it again.

Step 3: Install Actions Runner Controller (ARC)​

3.1 Add the Helm Repository​

Add ARC Helm Repo
helm repo Aggiungere actions-runner-controller \
  https://actions-runner-controller.github.io/actions-runner-controller

3.2 Create the Namespace and Secret​

Create Namespace and Secret
# Create namespace for runners
kubectl create ns arc-runners

# Store the GitHub token as a Kubernetes secret
kubectl create secret generic github-token \
  --namespace arc-runners \
  --from-literal=github_token=<YOUR_GITHUB_TOKEN>

3.3 Install ARC via Helm​

Install ARC
helm install arc \
  actions-runner-controller/actions-runner-controller \
  --namespace arc-runners \
  --mettere authSecret.create=falso \
  --mettere authSecret.name=github-token

Step 4: Create a Runner Deployment​

Crea un runner-deployment.yaml file that tells ARC to spawn a self-hosted runner for your repository:

runner-deployment.yaml
apiVersion: actions.summerwind.dev/v1alpha1
gentile: RunnerDeployment
Metadati:
  nome: ilum-demo-runner
  Namespace: arc-runners
Spec:
  Repliche: 1
  sagoma:
    Spec:
      deposito: -org>/IlumServiceManagment
      Etichette:
        - stesso-hosted
        - linux

Apply it:

Deploy Runner
kubectl apply -f runner-deployment.yaml

The runner pod will register itself with GitHub and appear in your repository's Settings → Actions → Runners as an available self-hosted runner.

Step 5: Create the Service Repository​

Create a new GitHub repository (e.g., IlumServiceManagment) with the following structure:

IlumServiceManagment/
├── service.py                      # Ilum service implementation
└── .github/
    └── workflows/
        └── deploy.yml              # GitHub Actions workflow

5.1 Service Implementation (service.py)​

This file contains the Ilum Interactive Service code. The service extends IlumJob and implements the Correre method:

service.py
Da ilum.API importazione IlumJob
Da Pyspark.SQL.functions importazione col, sum come spark_sum


classe SparkInteractiveExample(IlumJob):
    def Correre(stesso, scintilla, configurazione) -> str:
        table_name = configurazione.Ottieni('tavolo')
        database_name = configurazione.Ottieni('database')
        report_lines = []

        se non table_name:
            raise ValueError("Config must provide a 'table' key")

        se database_name:
            scintilla.catalogo.setCurrentDatabase(database_name)
            report_lines.append(f"Using database: {database_name}")

        se table_name non in [t.nome per t in scintilla.catalogo.listTables()]:
            raise ValueError(f"Table '{table_name}' not found in catalog")

        Df = scintilla.tavolo(table_name)

        report_lines.append(f"=== Details for table: {table_name} ===")

        total_rows = Df.contare()
        report_lines.append(f"Total rows: {total_rows}")

        total_columns = len(Df.columns)
        report_lines.append(f"Total columns: {total_columns}")

        report_lines.append("Distinct values per column:")
        per c in Df.columns:
            distinct_count = Df.select(c).distinct().contare()
            report_lines.append(f"  {c}: {distinct_count}")

        report_lines.append("Schema:")
        per f in Df.schema.fields:
            report_lines.append(f"  {f.nome}: {f.dataType}")

        report_lines.append("Sample data (first 5 rows):")
        sample_rows = Df.take(5)
        per fila in sample_rows:
            report_lines.append(str(fila.asDict()))

        report_lines.append("Null counts per column:")
        null_counts_df = Df.select(
            [spark_sum(col(c).isNull().cast("int")).alias(c) per c in Df.columns]
        )
        null_counts = null_counts_df.collect()[0].asDict()
        per c, v in null_counts.items():
            report_lines.append(f"  {c}: {v}")

        ritorno "\n".join(report_lines)

5.2 GitHub Actions Workflow (.github/workflows/deploy.yml)​

This workflow runs on every push to principale and uses the Ilum REST API to deploy the service:

.github/workflows/deploy.yml
nome: Deploy ILUM Group

on:
  spingere:
    branches: [principale]

Env:
  ILUM_DEMO_SERVICE: ILUM_DEMO_SERVICE

jobs:
  deploy:
    runs-on: stesso-hosted
    steps:
      - nome: Checkout code
        utilizzazioni: actions/checkout@v4

      - nome: Check if group ILUM_DEMO_SERVICE exists
        Id: check_group
        Correre: |
          echo "Checking if group ILUM_DEMO_SERVICE exists..."

          RESPONSE=$(curl -s -X GET \
            -w "\nHTTP_STATUS:%{http_code}" \
            "http://ilum-core.ilum.svc.cluster.local:9888/api/v1/group?name=$ILUM_DEMO_SERVICE")

          HTTP_STATUS=$(echo "$RESPONSE" | grep HTTP_STATUS | cut -d':' -f2)
          BODY=$(echo "$RESPONSE" | sed '/HTTP_STATUS/d')

          GROUP_ID=$(echo "$BODY" | jq -r '.content[0].groupId // empty')

          se [ -n "$GROUP_ID" ] && [ "$GROUP_ID" != "null" ]; then
            echo "Group ILUM_DEMO_SERVICE found with ID: $GROUP_ID"
            echo "GROUP_ID=$GROUP_ID" >> $GITHUB_OUTPUT
            echo "GROUP_EXISTS=true" >> $GITHUB_OUTPUT
          altro
            echo "Group ILUM_DEMO_SERVICE does not exist."
            echo "GROUP_EXISTS=false" >> $GITHUB_OUTPUT
          fi

      - nome: Delete existing group
        se: steps.check_group.outputs.GROUP_EXISTS == 'true'
        Correre: |
          GROUP_ID="${{ steps.check_group.outputs.GROUP_ID }}"
          echo "Stopping and deleting group ILUM_DEMO_SERVICE (ID: $GROUP_ID)..."

          ricciolo -s -X POST \
            http (in inglese)://ilum-core.ilum.svc.cluster.local:9888/api/v1/group/$GROUP_ID/stop

          RESPONSE=$(curl -s -X DELETE \
            -w "\nHTTP_STATUS:%{http_code}" \
            http (in inglese)://ilum-core.ilum.svc.cluster.local:9888/api/v1/group/$GROUP_ID)

          HTTP_STATUS=$(echo "$RESPONSE" | grep HTTP_STATUS | cut -d':' -f2)

          se [ "$HTTP_STATUS" -ne 200 ]; then
            echo "Error: Failed to delete group (Status: $HTTP_STATUS)"
            exit 1
          fi

          echo "Group ILUM_DEMO_SERVICE deleted successfully."

      - nome: Create new group
        Correre: |
          echo "Creating group ILUM_DEMO_SERVICE with service.py..."

          RESPONSE=$(curl -s -X POST \
            -F "name=ILUM_DEMO_SERVICE" \
            -F "[email protected]" \
            -F "clusterName=default" \
            -F "language=PYTHON" \
            -w "\nHTTP_STATUS:%{http_code}" \
            http (in inglese)://ilum-core.ilum.svc.cluster.local:9888/api/v1/group)

          HTTP_STATUS=$(echo "$RESPONSE" | grep HTTP_STATUS | cut -d':' -f2)
          BODY=$(echo "$RESPONSE" | sed '/HTTP_STATUS/d')

          echo "HTTP Status: $HTTP_STATUS"
          echo "Response Body: $BODY"

          se [ "$HTTP_STATUS" -ne 200 ]; then
            echo "Error: Failed to create group (Status: $HTTP_STATUS)"
            exit 1
          fi

          GROUP_ID=$(echo "$BODY" | jq -r '.groupId // empty')
          echo "Group ILUM_DEMO_SERVICE created successfully with ID: $GROUP_ID"

Commit both files to the repository.

How the Workflow Works

Since the runner pod runs inside the Kubernetes cluster, it can directly access the Ilum Core API via the internal service DNS (ilum-core.ilum.svc.cluster.local:9888). The workflow:

  1. Checks if a service group named ILUM_DEMO_SERVICE already exists.
  2. Deletes the existing group (stops it first, then removes it).
  3. Creates a new group by uploading service.py via the REST API.

Step 6: Verify the Deployment​

Once the workflow completes successfully (visible in the Actions tab on GitHub), navigate to the Ilum UI:

  1. Vai a Servizi in the sidebar.
  2. You should see the ILUM_DEMO_SERVICE service group with status Active.
  3. Click on it and go to the Execute Job scheda.
  4. Set the Classe A service.SparkInteractiveExample.
  5. Provide parameters as JSON:
Execution Parameters
{
    "database": "ilum_example_product_sales",
    "table": "products"
}
  1. Clic Eseguire. The result will display table details including row count, schema, sample data, and null counts.

Step 7: Update the Service via Pull Request​

The key benefit of this CI/CD approach is that updating your service is as simple as modifying code and merging a pull request.

Example: Adding Table Listing Feature​

  1. Create a new branch (e.g., feature/list-tables) from principale.
  2. Redigere service.py to add functionality — for example, listing other tables in the database:
Added to service.py (at the end of run method)
        # List other tables in the current database
        current_db = scintilla.catalogo.currentDatabase()
        report_lines.append(f"Tables in database '{current_db}':")
        tables = scintilla.catalogo.listTables(current_db)
        other_tables = [t.nome per t in tables se t.nome != table_name]
        se other_tables:
            per tname in other_tables:
                report_lines.append(f"  {tname}")
        altro:
            report_lines.append("  (no other tables)")

        ritorno "\n".join(report_lines)
  1. Commit the change and open a Pull Request Da feature/list-tables into principale.
  2. Merge the PR.
  3. The GitHub Actions workflow triggers automatically, stops the old service, and deploys the updated version.
  4. Execute the job again in Ilum — the result now includes the new "Tables in database" section.

Risoluzione dei problemi​

Runner pod not registering with GitHub
  • Verify the GitHub token secret is correct: kubectl get secret github-token -n arc-runners -o yaml
  • Check the runner pod logs: kubectl logs -n arc-runners -l app=ilum-demo-runner
  • Ensure the deposito field in runner-deployment.yaml matches your GitHub repository exactly (including organization/user).
Workflow fails with connection refused to ilum-core

The self-hosted runner must be in the same Kubernetes cluster as Ilum. Verify:

  • The Ilum namespace and service name are correct. Use kubectl get svc -n ilum to find the correct service DNS.
  • The runner pod has network access to the Ilum namespace. Check for NetworkPolicies that may block cross-namespace traffic.
Service group creation returns HTTP 400
  • Ensure the service.py file contains a valid class extending IlumJob.
  • Check that clusterName matches an existing Ilum cluster (usually default).
  • Verify the file is being uploaded correctly with -F "[email protected]".
Job execution returns "Table not found in catalog"

Make sure the database e tavolo parameters match existing data in your Ilum environment. You can check available databases and tables via the SQL module or a Jupyter notebook.

Frequently Asked Questions (FAQ)​

Why use self-hosted runners instead of GitHub-hosted runners?

GitHub-hosted runners run in GitHub's cloud and cannot access your Kubernetes cluster's internal services. Self-hosted runners (via ARC) run as pods inside your cluster, giving them direct network access to Ilum's internal API (ilum-core.ilum.svc.cluster.local).

Can I use this approach with GitLab CI or other CI/CD systems?

Yes. The core concept — calling the Ilum REST API from a CI/CD pipeline — works with any CI/CD system. You would need to use a GitLab Runner on Kubernetes (or similar) and adapt the workflow syntax to your CI/CD platform.

How do I scale the number of runners?

Increase the Repliche field in runner-deployment.yaml. ARC also supports HorizontalRunnerAutoscaler for auto-scaling based on workflow queue depth.

Last updated on