Vai al contenuto principale

Run Spark Submit (spark-submit) on Kubernetes

Un semplice lavoro Spark in Ilum funziona esattamente come uno inviato tramite lo standard scintilla-invio , ma con ulteriori miglioramenti per facilità d'uso, configurazione e integrazione con strumenti esterni.

È possibile utilizzare il file JAR con esempi di Spark dall'installazione locale di Spark o da qualsiasi file JAR personalizzato.

Di seguito è riportata una guida passo passo per configurare ed eseguire un semplice processo Spark utilizzando scintilla-invio on Ilum. This guide demonstrates the core configuration needed and shows how to monitor your job’s progress within the Ilum platform. For a complete overview of Ilum's architecture, check the Architecture Overview.

Quick Start (TL;DR)

How do I run a Spark job on Kubernetes with scintilla-invio?

To run a Spark job on Ilum (Kubernetes), ensure Java 17 and Spark are installed, upload your JAR, and run:

Quick Start: Spark Submit on K8s
./bin/spark-submit \
  --padrone k8s://http://<ilum-core-address>:<ilum-core-port>  \
  --deploy-mode cluster \
  --classe org.apache.spark.examples.SparkPi \
  --conf spark.driver.memory=4g \
  --conf spark.ilum.cluster=default \
  --conf spark.kubernetes.container.image=ilum/spark:4.1.2 \
  --conf spark.kubernetes.submission.waitAppCompletion=vero \
  s3a://ilum-files/ilum/default/spark-examples_2.13-4.1.2.jar

Note: Replace with your actual Ilum Core endpoint.

Step-by-Step Guide​

1. Prerequisites​

  • Assicurati che Java 17 sia installato e impostato correttamente nel tuo JAVA_HOME.
  • Download and extract the appropriate version of Apache Spark:
Download Spark 4
wget https://archive.apache.org/dist/spark/spark-4.1.2/spark-4.1.2-bin-hadoop3.tgz
tar -xzf spark-4.1.2-bin-hadoop3.tgz
cd spark-4.1.2-bin-hadoop3

2. Connect to Ilum​

If Ilum is deployed on Kubernetes, forward the service port to your local machine to make Ilum accessible at ospite locale:9888.

Forward Core
kubectl port-forward svc/ilum-core 9888:9888
Production Tip

Se stai comunicando dall'interno dello stesso cluster Kubernetes, puoi utilizzare gli indirizzi di servizio basati su DNS Kubernetes (ad esempio, http://ilum-core.namespace.svc.cluster.local) or expose services using Ingress.

3. Submit Your Spark Job​

Choose the submission method that best fits your workflow:

This method is suitable for quick local testing.

1. Upload your JAR File

For demonstration, we assume the JAR is uploaded manually to MinIO.

Locate the example JAR: examples/jars/spark-examples_2.13-4.1.2.jar

Upload it to MinIO (bucket ilum-files, path ilum/default/): s3a://ilum-files/ilum/default/spark-examples_2.13-4.1.2.jar

2. Submit via REST

Limitation

spark.ilum.pyRequisiti is not supported in this mode, as REST does not support PySpark submissions.

Run the following command:

REST Submit (Spark 4)
./bin/spark-submit \
  --padrone spark://localhost:9888 \
  --deploy-mode cluster \
  --classe org.apache.spark.examples.SparkPi \
  --conf spark.master.rest.enabled=vero \
  --conf spark.ilum.cluster=default \
  --conf spark.app.name=my-spark-job \
  s3a://ilum-files/ilum/default/spark-examples_2.13-4.1.2.jar

Parameters:

ParameterDescrizione
--padroneIlum Core address via REST (e.g. spark://localhost:9888).
--conf spark.master.rest.enabled=veroEnables REST submission.
s3a://...JAR file path in MinIO.
Expected Output
Esecuzione di Spark utilizzando il protocollo di invio dell'applicazione REST.
25/03/12 12:58:01 INFO RestSubmissionClient: Invio di una richiesta per lanciare un'applicazione in spark://localhost:9888.
25/03/12 12:58:03 INFO RestSubmissionClient: Invio creato correttamente come 20250312-1158-qdnioef2rny. Stato di invio del polling...
25/03/12 12:58:03 INFO RestSubmissionClient: Invio di una richiesta per lo stato di presentazione 20250312-1158-qdnioef2rny in spark://localhost:9888.
25/03/12 12:58:03 INFO RestSubmissionClient: Stato del driver 20250312-1158-qdnioef2rny è ora INVIATO.
25/03/12 12:58:03 INFO RestSubmissionClient: Il driver è in esecuzione sul worker ILUM all'indirizzo ILUM_UI_ADDRESS/workloads/details/job/20250312-1158-qdnioef2rny.
25/03/12 12:58:03 INFO RestSubmissionClient: Il server ha risposto con CreateSubmissionResponse:
{
  "action" : "CreateSubmissionResponse",
  "serverSparkVersion" : "4.1.2",
  "submissionId" : "20250312-1158-qdnioef2rny",
  "successo" : vero
}
25/03/12 12:58:03 INFO ShutdownHookManager: Hook di spegnimento chiamato
25/03/12 12:58:03 INFO ShutdownHookManager: Eliminazione della directory /tmp/spark-fa2603be-488a-4e2a-9b7f-5e49825d379b

4. Monitor and Troubleshoot​

Using the Ilum UI:

  • Monitoraggio dell'avanzamento del processo: Track executors, memory usage, and job stages.
  • Review Results: Access logs and the integrated Spark History Server.
  • Troubleshoot: Diagnose failures by checking detailed executor logs.

For more details on monitoring metrics, see the Monitoring Guide.

Comparison: Classic spark-submit vs Ilum Approach​

Running Spark directly on Kubernetes requires significant administrative effort. Ilum simplifies this by automating infrastructure management.

Traditional Approach (Native Spark on K8s) vs Ilum​

CaratteristicaNative Spark on K8sIlum (Managed Spark)
SetupManual Docker image build & complex scintilla-invio args.Automated. Use existing JARs; Ilum handles images.
ConfigurazioneVerbose (Service Accounts, Volumes, Secrets).Simplified. Minimal args; configs are injected automatically.
ImmagazzinamentoManual Hadoop/S3 configuration per job.Integrated. Automatic credential injection for S3/GCS/Azure.
MonitoraggioCLI-based (kubectl logs), ephemeral.Centralized UI. Persistent logs, metrics, and history.
OsservabilitàBasic Spark UI (if exposed).Avanzato. Derivazione dei dati, detailed resource metrics.

Key Benefits of Ilum:

  1. Automatic Image Selection: Ilum selects a compatible Spark Docker image matching the cluster version.
  2. Advanced Observability: Ilum provides deep lineage observability and advanced monitoring capabilities.
  3. Simplified Configuration: Reduce scintilla-invio parameters by 3x-4x.
  4. Integrated Storage Access: Credentials for all configured storages are automatically injected.
  5. Instant Monitoring: Logs and metrics (CPU/RAM) appear in the Ilum UI immediately.

For a developer, this means less time fighting with infrastructure and error-prone configurations, and more time delivering business logic.

For advanced customization, refer to the official Spark documentation.

Last updated on