Scheduling Apache Spark Jobs

Scheduling in Ilum allows you to automate the execution of Apache Spark jobs on Kubernetes clusters at specified intervals using CRON expressions. This is essential for setting up reliable ETL pipelines, regular data analysis, or maintenance tasks that need to run without manual intervention.

Example JAR Source

You can use the jar file with spark examples from one of these links:

Spark 4 (default)

Spark 4 / Scala 2.13: spark-examples_2.13-4.1.1.jar

Spark 3

Spark 3 / Scala 2.12: spark-examples_2.12-3.5.8.jar

Step-by-Step Guide: Scheduling a Spark Job

1. Navigate to Schedules: Access the Orari section in your Ilum dashboard.

2. Create New Schedule: Click the New Schedule + button to start setting up your automated job.

3. Fill Out Schedule Details:

   • Scheda Generale:

     • Nome: Entrare ScheduledMiniReadWriteTest
     • Cluster: Select your target cluster
     • Classe: Entrare org.apache.spark.examples.MiniReadWriteTest
     • Language: Select Scala

   • Tabella Temporizzazione:

     • CRON Expression: Select the Costume scheda.
     • Custom expression: Entrare @daily
     Timing

     This configuration will trigger the job to run once every day at midnight. You can adjust this to any valid CRON expression (e.g., 0 */12 * * * for every 12 hours).

   • Configuration Tab:

     • Arguments: Entrare /opt/spark/examples/src/main/resources/kv1.txt

   • Scheda Risorse:

     • Jars: Upload the JAR file from the link above.

   • Memory Tab:

     • Leave all settings at their default values for this example.

4. Submit and Monitor:

   • Clic Invia to create the schedule.
   • You can see your new schedule in the list.
   • When the scheduled time arrives, a new job instance will be launched automatically. You can view these instances in the Lavori section.

Schedule Configuration Reference

Below is a detailed breakdown of all available settings, organized by tab as they appear in the UI.

Generale

Parameter	Descrizione
Nome	A unique identifier for the schedule.
Grappolo	The target cluster where the scheduled jobs will be executed.
Classe	The fully qualified class name of the application (e.g., org.apache.spark.examples.SparkPi) or the filename for Python scripts.
Lingua	The programming language used for the job (Scala o Pitone).
Descrizione	An optional description to explain the purpose of this schedule.
Max Retries	The maximum number of times Ilum will attempt to restart the job if it fails.

Timing

Parameter	Descrizione
Start Time	(Optional) The specific date and time when the schedule should become active. If left blank, it starts immediately.
End Time	(Optional) The specific date and time when the schedule should stop triggering new jobs.
CRON Expression	Defines the frequency of the job execution. You can use the visual builders (Minutes, Hourly, Daily, Weekly, Monthly) or select Costume to enter a standard Unix-style CRON expression (e.g., 0 12 * * * for noon daily).

Configurazione

Parameter	Descrizione
Parametri	Key-value pairs for configuring Spark properties (e.g., spark.executor.memory).
Arguments	Command-line arguments passed to the main method of your application.
Tag	Custom labels to categorize and filter your scheduled jobs.

Risorse

Parameter	Descrizione
Vasi	Additional JAR files to be included in the classpath.
Files	Auxiliary files to be placed in the working directory of each executor.
PyFiles	Python dependencies (.zip, .egg, .Py) for Python jobs.
Fabbisogno	Additional Python packages to install.
Spark Packages	Maven coordinates for Spark JAR packages to be downloaded.

Memoria

Parameter	Descrizione
Executors	The number of executor instances to allocate.
Core del driver	The number of CPU cores assigned to the driver.
Core dell'esecutore	The number of CPU cores assigned to each executor.
Memoria del driver	The amount of RAM allocated to the driver.
Memoria dell'esecutore	The amount of RAM allocated to each executor.
Allocazione dinamica	Enables automatic scaling of executors based on workload.

Domande frequenti

Details

Can I schedule PySpark jobs using Ilum?

Yes, Ilum fully supports scheduling for both Scala/Java (JARs) and Python (PySpark) jobs. Simply select "Python" as the language in the General tab and provide your script.

Details

How does the retry mechanism work?

If a scheduled job fails, Ilum can automatically attempt to restart it based on the "Max Retries" configuration. This ensures transient issues don't break your pipelines.

Details

What CRON formats are supported?

Ilum supports standard Unix-style CRON expressions (e.g., 0 12 * * *) as well as predefined macros like @daily, @hourly, etc.