Create and Connect a Remote GKE Cluster to Data Lakehouse

Introduzione

Ilum empowers you to manage a powerful multi-cluster setup from a single, central control plane. While Ilum automates the deployment and configuration of its core components, setting up the underlying infrastructure requires precise coordination.

This guide provides a comprehensive walkthrough for setting up a multi-cluster architecture on Motore di Google Kubernetes (GKE). You will learn how to:

1. Provision a central control plane (Master Cluster).
2. Set up a dedicated execution environment (Remote Cluster).
3. Establish secure communication between them using client certificates and ingress rules.

This guide walks you through the steps required to launch your first Ilum Job on a remote cluster. We use Google Kubernetes Engine (GKE) as an example, but you can follow the same flow with any Kubernetes distribution.

Prerequisiti

Before starting the tutorial, make sure you have:

1. Access to a Google Cloud project with billing enabled.
2. kubectl installed and configured on your machine (version compatible with your GKE cluster).
3. Timone installed (v3+).
4. Le Google Cloud CLI (gcloud) installed and initialized (you can run gcloud auth login e gcloud config list without errors).
5. Le gke-gcloud-auth-plugin installed and available in your PATH so that kubectl can authenticate to GKE clusters.
6. Permissions in the target Google Cloud project to:
   • create and manage GKE clusters (e.g. Kubernetes Engine Cluster Admin or equivalent),
   • create and use Cloud Storage buckets if you plan to use GCS for data.

What you'll accomplish in this guide:

Step	Task	Purpose
1	Create two GKE clusters	Set up master (control plane) and remote (job execution)
2	Install Ilum on master	Deploy Ilum's core components
3	Set up authentication	Create secure credentials for remote cluster access
4	Register remote cluster	Add cluster to Ilum's management interface
5	Configure networking	Enable communication between clusters
6	Run your first job	Verify the multi-cluster setup works

---

Step 1. Provision Master and Remote GKE Clusters

The foundation of a multi-cluster setup consists of two distinct entities:

• Master Cluster: Hosts the Ilum control plane (UI, API, Scheduler).
• Remote Cluster: dedicated environment where Ilum executes the Spark Jobs dispatched from the master.

Creare un progetto

• Open Google Cloud Console.
• Click the Project selector in the top-left corner.
• Clic New Project.
• Enter a project name and (if applicable) select an Organization/Folder.
• Clic Creare.
• Select the newly created project in the project selector.

Enable Google Kubernetes Engine API

• In the Console search bar, type Motore Kubernetes.
• Open Motore Kubernetes.
• Clic Abilitare to enable the Google Kubernetes Engine API for the selected project.

Switch to the chosen project in gcloud

• In the Console, open the project selector and copy the Project ID.
• In your terminal, set this project as active:

Set Project

gcloud config mettere project PROJECT_ID

• (Optional) If you plan to create clusters in a specific region often, set a default region:

Set Default Region

gcloud config mettere compute/region europe-central2

This avoids errors requiring --region / --zone.

Creare un cluster

Create the master cluster first:

Create Master Cluster

gcloud container clusters create master-cluster \
  --tipo-macchina=n1-standard-8 \
  --num-nodes=1

Create the remote cluster with a different name:

Create Remote Cluster

gcloud container clusters create remote-cluster \
  --tipo-macchina=n1-standard-4 \
  --num-nodes=1

important

Resource Requirements & Architecture:

Why two clusters? The master cluster runs Ilum's control plane (UI, API, scheduler). The remote cluster executes your Spark jobs. This separation allows independent scaling and multi-cluster management from one interface.

Sizing:

• Master cluster: This example uses n1-standard-8 (8 vCPU, 30 GB RAM) for testing only. Minimum recommended: 12 vCPUs and 48 GB RAM (e.g., n1-standard-12). Production environments with many users need significantly more.
• Remote cluster: This example uses n1-standard-4 (4 vCPU, 15 GB RAM) for testing only. Production workloads require larger machines (e.g., n1-standard-16+) and multiple nodes depending on your Spark job requirements.

Step 2. Install Ilum Control Plane on Master Cluster

Once your clusters are running, the next step is to deploy the Ilum platform on the master cluster.

Passare alla configurazione del cluster master in kubectl

Switch Context

# Ottieni i contesti kubectl e trova quello con il tuo cluster
kubectl config get-contexts
# switch to it using this command
MASTER_CLUSTER_CONTEXT del contesto d'uso di kubectl config

Crea lo spazio dei nomi e passa ad esso in kubectl

Setup Namespace

kubectl create spazio dei nomi ilum
kubectl config set-context --current --namespace=ilum

Install Ilum using Helm charts:

Installa Ilum

helm repo Aggiungere ilum https://charts.ilum.cloud
helm install ilum -n ilum ilum/ilum

nota

What just happened? You've installed Ilum's control plane on the master cluster. You can now access Ilum's interface to manage jobs across multiple clusters.

Step 3. Configure Authentication for Remote Cluster Access

important

Goal: Generate secure credentials (client certificates) that authorize the Ilum control plane to deploy and manage resources on the cluster remoto.

Importante: All steps in this section must be performed while connected to the cluster remoto (not the master cluster).

Why this is needed:

When you created the remote cluster with gcloud, it was automatically added to your kubeconfig with your Google Cloud account credentials. However, Ilum doesn't have access to your Google account.

Ilum needs its own authentication method - client certificates - to connect to and manage the remote cluster. This section creates those certificates.

Step	Action	Purpose
1	Create Client Key	Generate private key for authentication
2	Create CSR	Request identity verification from cluster
3	Register & Approve CSR	Cluster admin validates and signs request
4	Get Signed Certificate	Retrieve the "ID card" for Ilum
5	Grant Admin Permissions	Authorize Ilum to manage cluster resources
✓	Risultato	Ilum can securely manage the cluster

This is similar to how kubectl authenticates to clusters, but we're doing it programmatically for Ilum.

mancia

Before you start: Make sure you're connected to the cluster remoto context:

Switch to Remote Context

kubectl config use-context gke_<your-project>_<region>_remote-cluster

You can verify with: kubectl config current-context

Step 3.1: Create a Client Key

What we're doing: Generating a private key that will be used for authentication.

First, create a dedicated directory for the certificates and work there:

Create Workspace

mkdir -p ~/remote-cluster
cd ~/remote-cluster

Now generate the private key:

Generate Private Key

openssl genpkey -algorithm RSA -out client.key

danger

Security: This private key is like a password. Keep client.key secure and never commit it to version control.

Step 3.2: Create Certificate Request

What we're doing: Creating a Certificate Signing Request (CSR) that asks the remote cluster to verify and sign our identity.

Sostituire Il mio utente with the username of your choice:

Create CSR

openssl req -new -key client.key -out csr.csr -subj "/CN=myuser"

mancia

What is a CSR? A Certificate Signing Request asks the Kubernetes cluster to create a signed certificate for your user. The cluster will verify and sign it, creating a trusted identity.

Step 3.3: Encode the CSR

What we're doing: Converting the CSR to base64 format, which is required by Kubernetes.

Encode CSR

cat csr.csr | base64 | tr -d '\n'

Copy the output - you'll need it in the next step.

Step 3.4: Register CSR in Kubernetes

What we're doing: Submitting the CSR to the remote cluster for approval.

Creare csr.yaml (replace Il mio utente and paste your encoded CSR):

csr.yaml

apiVersion: certificates.k8s.io/v1
gentile: CertificateSigningRequest
Metadati:
  nome: Il mio utente  # Replace with your username
Spec:
  richiesta: >
  signerName: kubernetes.io/kube-apiserver-cliente
  Usi:
    - autenticazione client

Apply the CSR to the cluster remoto:

Register CSR

kubectl apply -f csr.yaml

Step 3.5: Approve and Retrieve the Certificate

What we're doing: As cluster admin, we approve our own CSR and retrieve the signed certificate.

Approve the CSR:

Approve CSR

Certificato kubectl Approva MyUser

Retrieve the signed certificate:

Retrieve Certificate

kubectl get csr myuser -o jsonpath='{.status.certificate}' | base64 --decode > cliente.crt

nota

What happened? You've registered your certificate request with Kubernetes, approved it (as cluster admin), and retrieved the signed certificate. This certificate + private key pair is now your authentication credential for the remote cluster.

Step 3.6: Get CA Certificate

What we're doing: Extracting the cluster's CA certificate. This will be needed for the optional test in the next section, and later in Step 4 when adding the cluster to Ilum.

Get and save the CA certificate:

Get CA Certificate

kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 --decode > ca.crt

Verify the certificate was saved correctly:

Verify CA Certificate

openssl x509 -in ca.crt -text -noout | head -n 5

mancia

Why do we need this?

• CA Certificate: Verifies the cluster's identity (prevents man-in-the-middle attacks)
• Flags: --minify gets only current context, --raw gets actual data instead of file paths

Step 3.7: Grant Permissions with Role Binding

What we're doing: Giving the certificate user (myuser) admin permissions on the remote cluster.

Creare rolebinding.yaml:

apiVersion: rbac.authorization.k8s.io/v1
gentile: ClusterRoleBinding
Metadati:
  nome: Il mio utente-Admin-rilegatura
Soggetti:
  - gentile: Utente
    nome: Il mio utente  # Match your username
    apiGroup: rbac.authorization.k8s.io
roleRef:
  gentile: Ruolo del cluster
  nome: grappolo-Admin
  apiGroup: rbac.authorization.k8s.io

Apply the role binding to the cluster remoto:

Apply Role Binding

kubectl apply -f rolebinding.yaml

avvertimento

Security: We're granting cluster-admin for simplicity. In production, create a custom role with only the permissions Ilum needs:

• Create/delete pods, services, configmaps
• Read secrets
• Manage persistent volumes

Step 3.8: Configure Service Account Permissions

What we're doing: Granting permissions to the ServiceAccount that Spark pods will use.

important

What is a Service Account? It's the Kubernetes equivalent of a user for pods. Spark driver pods use it to create executor pods and manage resources in the cluster.

By default, pods use the default ServiceAccount in their namespace, but it doesn't have sufficient privileges to create other pods.

Creare sa_role_binding.yaml:

sa_role_binding.yaml

apiVersion: rbac.authorization.k8s.io/v1
gentile: ClusterRoleBinding
Metadati:
  nome: ilum-default-Admin-rilegatura
roleRef:
  apiGroup: rbac.authorization.k8s.io
  gentile: Ruolo del cluster
  nome: grappolo-Admin
Soggetti:
  - gentile: ServiceAccount
    nome: default
    Namespace: ilum  # Namespace where Spark jobs will run

Apply the role binding to the cluster remoto:

Apply SA Role Binding

kubectl apply -f sa_role_binding.yaml

avvertimento

Production Security: We're granting cluster-admin to the default ServiceAccount for simplicity. In production:

1. Create a dedicated ServiceAccount (e.g., spark-driver) with minimal permissions
2. Specify it in your cluster configuration:

   spark.kubernetes.authenticate.driver.serviceAccountName=spark-driver
3. Grant only the permissions needed: create/delete pods, read configmaps/secrets, etc.

(Optional) Test the Client Certificates

mancia

Why this step? This optional step verifies that the certificates work correctly before we give them to Ilum in Step 4.

We'll create a temporary kubectl context that uses the certificates (instead of your Google Cloud credentials) to confirm they have the correct permissions.

1. Verify certificates are in the directory:

All certificate files should already be in ~/remote-cluster/ from the previous steps:

List Workspace Content

ls ~/remote-cluster/
# You should see: ca.crt  client.crt  client.key  csr.csr  csr.yaml

2. Get your remote cluster name:

Get Cluster Name

kubectl config view -o jsonpath='{.clusters[*].name}' | tr ' ' '\n' | grep remoto

You should see something like: gke_my-project_europe-central2_remote-cluster

Copy this cluster name.

3. Add the certificate-based user to kubectl:

Set Credentials

kubectl config set-credentials myuser-cert \
  --client-certificate=$HOME/remote-cluster/client.crt \
  --client-key=$HOME/remote-cluster/client.key

This creates a new user in your kubeconfig that uses certificates instead of your Google account.

4. Create a test context:

Set Context

kubectl config set-context test-remote-certs \
  --cluster=<paste-your-remote-cluster-name-here> \
  --namespace=default \
  --user=myuser-cert

Sostituire with the cluster name from step 2.

5. Test the certificate-based authentication:

Test Authentication

kubectl config use-context test-remote-certs
kubectl get pods

Expected output:

Nessuna risorsa trovata nello spazio dei nomi predefinito.

If you see this (or a list of pods), the certificates work! ✅

6. Switch back to your normal context:

Revert Context

# Find your original context
kubectl config get-contexts

# Switch back (replace with your actual context name)
kubectl config use-context gke_<your-project>_<region>_remote-cluster

nota

What we just verified: The client certificates can successfully authenticate to the remote cluster with admin permissions. In Step 4, we'll provide these same certificates to Ilum so it can manage the cluster.

---

Step 4. Register the Remote Cluster in Ilum UI

With your certificates generated and permissions granted, you are now ready to connect the remote cluster to the Ilum control plane.

What you'll provide: All the credentials and connection details you created in Step 3:

Item	File/Value	Purpose
CA Certificate	ca.crt	Verifies cluster identity
Client Certificate	cliente.crt	Your authentication credential
Client Key	client.key	Private key for authentication
Server URL	From kubectl	Cluster API endpoint
Username	Il mio utente	User identity

Access Ilum UI

Before you can add the remote cluster to Ilum, you need to access the Ilum web interface.

1. Switch to the master cluster context:

Switch Context

kubectl config use-context gke_<your-project>_<region>_master-cluster

You can verify with: kubectl config current-context

2. Set up port forwarding to Ilum UI:

Port Forward UI

kubectl port-forward -n ilum svc/ilum-ui 9777:9777

Keep this terminal window open - it needs to run continuously.

3. Open Ilum in your browser:

Open http://localhost:9777 in your web browser.

4. Log in:

• Username: Admin
• Password: Admin

Now you're ready to add the remote cluster to Ilum!

---

Demo: Adding a Kubernetes Cluster

Guida a schermo intero

Vai alla pagina di creazione del cluster

• Passare alla sezione Cluster nel carico di lavoro
• Clic Nuovo cluster bottone

Specificare le impostazioni generali:

• Scegli un nome e una descrizione che desideri.
• Impostare il tipo di cluster su Kubernetes
• Scegli una versione di spark

La versione di Spark viene specificata selezionando un'immagine per i processi Spark.

Ilum richiede l'uso di immagini proprie, che includono tutte le dipendenze necessarie preinstallate.

Ecco un elenco di tutte le immagini disponibili:

Specificare le configurazioni di spark

Le configurazioni Spark specificate a livello di cluster verranno applicate a ogni singolo processo Ilum distribuito in tale cluster.

Ciò può essere utile se si desidera evitare di ripetere le configurazioni. Ad esempio, se si usa Iceberg come catalogo Spark, è possibile configurarlo una sola volta a livello di cluster e quindi tutti i processi Ilum distribuiti in questo cluster useranno automaticamente queste configurazioni.

Important: Set the namespace for Spark jobs

Nel Parametri section, add:

spark.kubernetes.namespace: ilum

nota

This ensures all Spark jobs run in the ilum namespace where you created the external services (ilum-core, ilum-grpc, ilum-minio) in Step 5. Without this, jobs would run in default namespace and couldn't connect to master cluster services.

Aggiungere archivi

Default cluster storage will be used by Ilum to store all the files required to run Ilum Jobs both provided by user and by Ilum itself.

You can choose any type of storage: S3, GCS, WASBS, HDFS.

mancia

For detailed instructions on setting up storage (especially GCS), see the Crea spazio di archiviazione guide.

Quick setup in the UI:

• Fare clic sull'icona "Aggiungi spazio di archiviazione" bottone
• Specify the name and choose the storage type (S3, GCS, etc.)
• Scegliere Benna scintilla - bucket principale utilizzato per archiviare i file dei lavori Ilum
• Scegliere Bucket di dati - obbligatorio nel caso in cui si utilizzi il formato spark di Ilum Tables
• Specify storage endpoint and credentials
• Fare clic sull'icona "Invia" bottone

È possibile aggiungere un numero qualsiasi di archivi e i processi Ilum verranno configurati per utilizzare ciascuno di essi.

Vai alla sezione Kubernetes e specifica le configurazioni di Kubernetes

Qui, forniamo a Ilum tutti i dettagli necessari per connettersi al nostro cluster Kubernetes. Il processo è simile alla connessione al cluster tramite lo strumento kubectl, ma in questo caso viene eseguito tramite l'interfaccia utente.

You will need to provide the following:

Field	What to provide	File location
Url	Cluster API endpoint (text field)	⚠️ Switch to remote cluster first, then: kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
CaCert	Upload CA certificate file	~/remote-cluster/ca.crt (from Step 3.6)
ClientKey	Upload client private key file	~/remote-cluster/client.key (from Step 3.1)
ClientCert	Upload client certificate file	~/remote-cluster/client.crt (from Step 3.5)
Username	Username (text field)	Il mio utente (or whatever you used in Step 3.2)

Once you have specified these items, Ilum will be able to access your Kubernetes cluster.

Inoltre, puoi:

• Configurare Kubernetes in modo che richieda un parola d’ordine per l'utente. In questo caso, sarà necessario specificare la password qui nell'interfaccia utente.
• Aggiungi un passphrase durante la creazione della chiave client. In questo caso, sarà necessario specificare la passphrase qui nell'interfaccia utente.
• Specificare l'icona algoritmo chiave utilizzato nel client.key. Questo non è obbligatorio, poiché queste informazioni sono solitamente memorizzate all'interno della chiave stessa. Tuttavia, in alcuni casi potrebbe essere necessario definirlo in modo esplicito.

Infine, puoi fare clic sul pulsante Invia per aggiungere un cluster.

---

Infine, puoi fare clic sul pulsante Invia per aggiungere un cluster.

Step 5. Configure Multi-Cluster Networking

avvertimento

The Challenge: While Ilum can now dispatch jobs to the remote cluster, those jobs need a way to report their status, logs, and metrics back to the master cluster.

Why jobs need to communicate:

Component	Action	Target Service	Ubicazione
Lavori Spark	Send status updates	Ilum Core (gRPC)	Master Cluster
Lavori Spark	Send metrics & logs	MinIO Event Log	Master Cluster
Scheduled Jobs	Trigger new jobs	Nucleo di Ilum	Master Cluster

The Solution: Expose Ilum's services from the master cluster and create DNS aliases in the remote cluster.

Step 5.1: Configure Firewall

Prima di tutto, devi consentire il traffico in entrata nel tuo cluster master GKE. Il progetto Google Cloud consente al traffico di accedere tramite il suo firewall. Il firewall può essere gestito tramite regole. Per consentire al cluster remoto di accedere ai servizi del cluster master, è necessario aggiungere questa regola a Il progetto del nostro master-cluster:

Create Firewall Rule

gcloud compute firewall-rules create allow-ingress-traffic \
  --network default \
  --direction INGRESS \
  --action ALLOW \
  --rules tcp:80,tcp:443 \
  --source-ranges 0.0.0.0/0 \
  --description "Allow HTTP/HTTPS"

danger

Security Warning: --source-ranges 0.0.0.0/0 allows traffic from anywhere. In production, restrict this to your remote cluster's IP range:

--source-ranges 10.0.0.0/8  # Example: your cluster's CIDR

Step 5.2: Expose Services from Master Cluster

nota

What is LoadBalancer? It's a Kubernetes service type that provisions a public IP address, making the service accessible from outside the cluster (including from your remote cluster).

È necessario esporre i servizi nel cluster master al mondo esterno. A tale scopo, modificare il tipo in LoadBalancer. Ilum semplifica questo processo utilizzando le configurazioni Helm.

To expose Ilum Core, gRPC, and MinIO to the outside world, run the following command in your terminal:

Expose Services

helm upgrade ilum -n ilum ilum/ilum \
  --mettere ilum-core.service.type="LoadBalancer" \
  --mettere ilum-core.grpc.service.type="LoadBalancer" \
  --mettere minio.service.type="LoadBalancer" \
  --valori-di-riutilizzo

Dopodiché dovresti attendere qualche minuto e quindi controllare i tuoi servizi eseguendo

Check Services

kubectl ottieni servizi

Per vedere questo:

Here you can notice the ilum-core, ilum-grpc and minio services changed their type to LoadBalancer and got public IP. Now you can go to http://<public-ip>:9888/api/v1/group per verificare se è tutto a posto.

Step 5.3: Create External Services in Remote Cluster

mancia

What is ExternalName? It creates a local DNS alias in the remote cluster. When a job tries to connect to nucleo dell'ilum: 9888, Kubernetes redirects it to the public IP. Jobs use familiar service names as if everything was in one cluster.

Switch to remote cluster context:

Before creating external services, make sure you're working on the cluster remoto:

Switch to Remote Cluster

kubectl config use-context gke_<your-project>_<region>_remote-cluster

Creare external_services.yaml in the remote cluster:

nota

Use the EXTERNAL-IP addresses from the kubectl ottieni servizi output in your master cluster (the public IPs shown in the screenshot above). Replace the placeholders below with these actual IP addresses.

external_services.yaml

apiVersion: v1
gentile: Servizio
Metadati:
  nome: ilum-nucleo
  Namespace: ilum  # Must match the namespace where jobs will run
Spec:
  digitare: ExternalName
  externalName: 34.118.72.123  # Replace with actual EXTERNAL-IP from master cluster
  Porte:
    - porto: 9888
      targetPort: 9888

---
apiVersion: v1
gentile: Servizio
Metadati:
  nome: ilum-grpc
  Namespace: ilum  # Must match the namespace where jobs will run
Spec:
  digitare: ExternalName
  externalName: 34.118.72.124  # Replace with actual EXTERNAL-IP from master cluster
  Porte:
    - porto: 9999
      targetPort: 9999

---
apiVersion: v1
gentile: Servizio
Metadati:
  nome: ilum-minio
  Namespace: ilum  # Must match the namespace where jobs will run
Spec:
  digitare: ExternalName
  externalName: 34.118.72.125  # Replace with actual EXTERNAL-IP from master cluster
  Porte:
    - porto: 9000
      targetPort: 9000

important

Before applying:

1. Replace the example IPs (34.118.72.123, etc.) with your actual EXTERNAL-IP addresses Da kubectl ottieni servizi in the master cluster
2. Make sure the namespace: ilum matches the namespace where your Spark jobs will run in the remote cluster
3. If you haven't created the ilum namespace in the remote cluster yet, create it first:
   Create Namespace

   kubectl create spazio dei nomi ilum

Apply the external services:

Apply External Services

kubectl apply -f external_services.yaml

nota

Multi-Cluster Bridge Complete: Jobs running on the remote cluster can now reach Ilum's services on the master cluster using familiar service names (ilum-core, ilum-grpc, ilum-minio), even though they're actually connecting over the internet via public IPs.

Components in Multi-Cluster Architecture

Components compatible with multi-cluster (with additional networking):

Component	Purpose	Requires Exposure
Hive Metastore	Metadata management for tables	✅ Yes
Marquez	Data lineage tracking	✅ Yes
Server della cronologia	Spark application history	✅ Yes
Grafite	Metrics collection	✅ Yes

Components restricted to single-cluster:

Component	Ragione
Cubo Prometeo Pila	Prometheus needs direct pod access for metrics scraping. Challenging with dynamic pods across clusters.
Loki e Promtail	Promtail collects logs similarly to Prometheus. Same multi-cluster limitations.

Informazioni

All other Ilum services not listed above are cluster-independent and work in both single- and multi-cluster setups.

---

Step 6. Verify the Multi-Cluster Setup

The final step is to validate your configuration by running a real Spark job on the remote cluster.

Step 1: Access Ilum UI

1. Switch to master cluster context:

   kubectl config use-context gke_<your-project>_<region>_master-cluster

2. Set up port-forwarding:

   kubectl port-forward -n ilum svc/ilum-ui 9777:9777

   Keep this terminal window open.

3. Open Ilum in browser:

   • Navigate to http://localhost:9777
   • Login with default credentials: amministratore:admin

Step 2: Create a Test Job

1. Navigate to Jobs section in Ilum UI

2. Click "New Job +" bottone

3. Configure the job:

   • Nome: RemoteClusterTest
   • Job Type: Spark Job
   • Cluster: Select your cluster remoto (not master)
   • Classe: org.apache.spark.examples.SparkPi
   • Language: Scala

4. Add Resources:

   • Vai a Risorse tab
   • Jars: Upload one of these:
     • Spark 4 / Scala 2.13 (default): spark-examples_2.13-4.1.1.jar
     • Spark 3 / Scala 2.12: spark-examples_2.12-3.5.8.jar

5. Submit the job

Step 3: Verify Execution

If everything is configured correctly:

✅ Job starts successfully - Pods are created in the remote cluster

✅ Logs appear - You can see Spark initialization and execution logs

✅ Job completes - Final output shows: Pi is roughly 3.14...

Check remote cluster pods:

Check Remote Pods

kubectl config use-context gke_<your-project>_<region>_remote-cluster
kubectl get pods -n ilum

You should see Spark driver and executor pods running or completed.

mancia

For detailed job configuration options, see the Run Simple Spark Job guide.

---

Troubleshooting & FAQ

Here are solutions to common issues you might encounter when connecting a remote GKE cluster.

Why is my Spark Job stuck in "Pending" state?

This usually happens if the remote cluster lacks resources or can't pull images.

• Check Resources: Ensure your node pool has enough CPU/RAM.
• Check Events: Run kubectl get events -n ilum on the remote cluster to see scheduling errors.

Why can't the remote cluster connect to Ilum Core?

If the job runs but fails to report status:

• Verify Firewall: Ensure the master cluster's firewall allows ingress on ports 9888 (Core), 9999 (gRPC), and 9000 (MinIO).
• Check DNS: Verify the ExternalName services in the remote cluster resolve to the master's public IP.

Can I use a private GKE cluster?

Yes, but you will need to configure VPC Peering or a VPN between the master and remote networks instead of using public LoadBalancers.

---

Passaggi successivi

Congratulations! You've successfully set up a robust multi-cluster Ilum environment. You can now:

1. Deploy Spark jobs to your remote cluster from the Ilum UI
2. Monitor job execution across all clusters from one interface
3. Scale horizontally by adding more remote clusters using the same process
4. Optimize costs by using different machine types for different workloads

important

Production Checklist:

• Restrict firewall rules to specific IP ranges
• Create custom RBAC roles instead of cluster-admin
• Set up TLS certificates for LoadBalancer services
• Configure resource quotas and limits
• Abilitare monitoring and alerting
• Document your cluster configurations