Vai al contenuto principale

Run dbt Core on Spark (Kubernetes)

This guide explains how to set up dbt Core con Apache Spark running on a Kubernetes cluster. Using Ilum as the execution engine, you can run scalable data transformation pipelines directly on your data lake.

You have two primary ways to connect dbt to Spark on Ilum:

Thrift Server vs. Spark Connect​

CaratteristicaMethod 1: Spark Thrift Server (Legacy)Method 2: Spark Connect (Modern)
ProtocolJDBC/ODBC (via HiveDriver)gRPC (via Spark Connect)
Connection TypeMetodo: parsimoniaMetodo: sessione
ArchitetturaRequires a dedicated Thrift Server podConnects directly to Spark Driver
PerformanceHigher latency (row-based serialization)High performance (Arrow-based)
Ideale perBI Tools (Tableau, PowerBI), Legacy appsData Engineering, Python/dbt pipelines
New to Spark Connect?

For a deep dive into the architecture, check out our Spark Connect on Kubernetes Guide.

Prerequisiti​

Before starting, ensure your development environment is ready:

  • Kubernetes Cluster: You need a running K8s cluster (GKE, EKS, AKS, or Minikube).
  • Tools:
  • Knowledge: Basic understanding of dbt projects and Spark concepts.

How to Configure dbt with Spark on Kubernetes​

Choose your preferred connection method:

Step 1: Deploy Spark Thrift Server​

Deploy Ilum with the SQL module (acting as a scalable Thrift server) and Hive Metastore enabled:

Helm Install
helm repo Aggiungere ilum https://charts.ilum.cloud
helm install ilum ilum/ilum \
  --mettere ilum-hive-metastore.enabled=vero \
  --mettere ilum-core.metastore.enabled=vero \
  --mettere ilum-core.metastore.type=alveare \
  --mettere ilum-sql.enabled=vero \
  --mettere ilum-core.sql.enabled=vero

Step 2: Connect to the Thrift Service​

1. Identify the service:

Get Service
kubectl get servizio

Trova il servizio con "sql-thrift-binary" nel nome.

2. Port-forward:

Port Forward
kubectl port-forward svc/ilum-sql-thrift-binary 10009:10009

Questo rende il server Thrift disponibile all'indirizzo localhost:10009.

3. Test with Beeline (optional):

beeline -u "jdbc:hive2://localhost:10009/default"

Correre:

MOSTRARE TABELLE;

Aspettatevi un elenco vuoto o tabelle esistenti.

Configuring and Running dbt​

1. Clean Environment (if needed):

Uninstall Conflict
pip uninstall dbt-spark pyspark -y

2. Install dbt and dependencies:

Install dbt-spark
pip install Pyspark==3.5.8
pip install dbt-core
pip install "dbt-spark[PyHive,session]"
pip install --upgrade economia

3. Verify installation:

Verify dbt
DBT --version

Create dbt Project​

1. Initialize a dbt project:

Init Project
ilum_dbt_project di inizializzazione dbt

2. Answer the setup prompts:

Setup Prompts
Which database? 1 (scintilla)
host: localhost
Desired authentication method: 3 (economia)
port: 10009
schema: default
threads: 1

This creates the ilum_dbt_project directory and a profiles.yml file in ~/.dbt/.

Configure dbt for Ilum​

Redigere ~/.dbt/profiles.yml to include both Thrift and Spark Connect targets:

~/.dbt/profiles.yml
ilum_dbt_project:
  bersaglio: economia  # Default target
  Uscite:
    economia:
      digitare: scintilla
      metodo: economia
      ospite: localhost
      porto: 10009
      schema: default
      Discussioni: 1
      connect_retries: 5
      connect_timeout: 60
      connect_args:
        URL: "jdbc:hive2://localhost:10009/default; transportMode=binario; hive.server2.transport.mode=binario"
        autista: "org.apache.hive.jdbc.HiveDriver"
        Auth: "NESSUNO"
    
    spark_connect:
      digitare: scintilla
      metodo: sessione
      ospite: localhost
      porto: 15002
      schema: default
      Discussioni: 1

Switch between targets:

Run dbt
# Use Thrift (default)
dbt run

# Use Spark Connect
dbt run --target spark_connect

# Or set default in dbt_project.yml
# target: spark_connect

  • Connessione di prova:

    Debug dbt
    cd ilum_dbt_project
    Debug DBT

    Assicurati che non vengano visualizzati errori, indicando una connessione riuscita al server Thrift.

Create a Model to Write Data​

  • Crea modello:

  • Modelli/sample_data.sql

    Modelli/sample_data.sql
    {{ config(Materializzata='tavolo') }}

    SELEZIONARE 
      Id,
      nome
    DA (
      VALORI 
        (1, 'Alice'),
        (2, 'Bob')
    ) COME t(Id, nome)

  • Modello di esecuzione:

    Run sample_data
    dbt run --select sample_data

Create a Model to Read Data​

  • Crea modello:

  • Modelli/read_data.sql

    Modelli/read_data.sql
    {{ config(Materializzata='tavolo') }}

    SELEZIONARE 
      Id,
      nome,
      LUNGHEZZA(nome) COME name_length
    DA {{ riferimento('sample_data') }}

  • Modello di esecuzione:

    Run read_data
    dbt run --select read_data

Verify Results​

1. Monitor Job in Ilum UI:

  • Access the Ilum UI (URL provided in your Ilum setup, e.g. port-forward)

  • Navigate to the Jobs section

  • Look for the job named ilum-sql-spark-engine

  • Check job status, logs, and execution details to confirm successful processing

    2. Query with Beeline:

    beeline -u "jdbc:hive2://localhost:10009/default"

    3. Run query:

    SELEZIONARE * DA default.read_data;

    Expected output:

    +----+-----+------------+
    | id | nome| name_length|
    +----+-----+------------+
    | 1 | Alice| 5 |
    | 2 | Bob | 3 |
    +----+-----+------------+

Troubleshooting dbt-spark Connections​

Common issues when connecting dbt to Spark on Kubernetes:

Error: "ThriftTransportException: Could not connect to localhost:10009"

Cause: The port forwarding tunnel is down or the Thrift Server pod is not running. Soluzione:

  1. Check if the Thrift pod is running: kubectl get pods -l app.kubernetes.io/name=ilum-sql
  2. Restart port-forwarding: kubectl port-forward svc/ilum-sql-thrift-binary 10009:10009
Error: "grpc._channel._InactiveRpcError: failed to connect to all addresses"

Cause: Your local dbt client cannot reach the Spark Connect gRPC port (15002). Soluzione:

  • Ensure you have port-forwarded the Driver Pod, not the Service (unless using NodePort).
  • Verify you are using Metodo: sessione in profiles.yml.
Error: "AnalysisException: Table or view not found"

Cause: Hive Metastore connectivity issue. Soluzione:

  • Ensure ilum-core.metastore.enabled=true was set during Helm install.
  • Check if the schema (database) exists in Spark: spark.sql("SHOW DATABASES").show()

Orchestrazione​

For production orchestration using Apache Airflow, see the dedicated guide: Orchestrate dbt with Airflow

Last updated on