Vai al contenuto principale

Moduli

Ilum is built around a small set of always-on platform services and a curated catalogue of optional modules. Each module is a Helm sub-chart that adds a capability to the platform: an additional execution engine (Trino, Flink), an additional catalog (Nessie, Unity Catalog), a notebook environment (JupyterHub, Zeppelin), an orchestrator (Airflow, Kestra, Mage, n8n, NiFi), a BI tool (Superset, Streamlit), an AI/ML stack (MLflow, LangFuse), or an observability component (Kube Prometheus stack, Loki, Promtail).

Modules can be turned on at install time through the umbrella Helm chart, or managed at runtime through the in-product Modules registry, which is backed by the dedicated ilum-api microservice.

Le ilum-api microservice​

ilum-api is the module-management microservice that drives module install, upgrade, and disable operations at runtime. It is a peer service to ilum-core (the main platform REST API) and ilum-ui (the web frontend), and ships as the helm_api sub-chart inside the umbrella Helm chart.

Key characteristics:

  • Stateless: Holds no in-memory state. Tolerates pod restarts and scales horizontally.
  • Cluster-scoped RBAC: Required for cross-namespace Helm operations.
  • Long termination grace period: Lets in-flight Helm operations complete before pod shutdown (default 600 seconds).
  • Health endpoint: /api/v1/health.

When an operator toggles a module from the in-product Modules registry, ilum-api translates the action into the corresponding Helm install, upgrade, or disable operation against the cluster.

Roadmap: MCP and open APIs​

Future releases will extend ilum-apicon Model Context Protocol (MCP) capabilities and open APIs so that external systems and AI agents can drive module operations programmatically. This positions ilum-api as the platform-wide extensibility surface for Ilum, beyond its current module-management role.

The Modules registry​

The frontend Modules registry catalogues every module Ilum can install, organised by category:

  • Engines: Spark, DuckDB, Trino, Flink.
  • Cataloghi : Hive Metastore, Project Nessie, Unity Catalog, DuckLake.
  • Immagazzinamento : MinIO, S3, GCS, Azure Blob, HDFS.
  • Notebook : Jupyter, JupyterHub (Enterprise), Apache Zeppelin.
  • Orchestrazione : Apache Airflow, Kestra, Mage, n8n, Apache NiFi.
  • BI and visualization: Apache Superset, Streamlit.
  • AI and ML: MLflow, LangFuse, AI Data Analyst.
  • Governance: Marquez (lineage), Unity Catalog.
  • Osservabilità : Kube Prometheus stack, Grafana, Loki, Promtail, Graphite exporter.
  • Security and identity: Ory Hydra (Ilum as IdP), OpenLDAP.

Each entry is gated by a permission key (for example AIRFLOW_READ ) and surfaces in the navigation only for users with the corresponding role.

Module lifecycle​

The lifecycle of an optional module is:

  1. Discovery: An operator opens the Modules view in the Ilum UI.
  2. Enablement: The operator toggles a module on. ilum-api issues a Helm install or upgrade against the cluster, applying defaults and any operator overrides.
  3. Provisioning: Kubernetes schedules the module's pods, and dependent resources (PVCs, services, ingresses) are created.
  4. Exposure: When the module is healthy, the frontend exposes it in the navigation, behind the relevant permission.
  5. Upgrade: As new Ilum releases ship updated module versions, operators trigger upgrades through the same UI flow. ilum-api runs the corresponding Aggiornamento dell'elmo .
  6. Disable: Toggling a module off triggers a helm uninstall. Persistent state (Postgres tables, MinIO buckets, attached volumes) is retained unless explicitly cleaned up.

For operators who prefer GitOps, every module is also configurable directly through the umbrella Helm chart values (refer to All-in-One configuration).

Module configuration​

Module-specific configuration lives in the umbrella Helm chart. For example, to enable Trino with a custom catalog at install time:

Trino : 
  Abilitato :  vero 
  Cataloghi : 
    ilum-delta :  | 
connector.name=delta_lake 
      hive.metastore.uri=thrift://ilum-hive-metastore:9083

For per-module reference, refer to:

Why modules​

The modules architecture exists to keep deployments small and focused. A development cluster can run with ilum-core , ilum-ui , ilum-api, Spark, DuckDB, MinIO, and PostgreSQL in under 12 GB of memory. A full enterprise deployment can layer on Trino, Nessie, Unity Catalog, Airflow, Superset, MLflow, LangFuse, and the kube-prometheus-stack, and grow each component independently.

ilum-api makes this modularity dynamic: optional capabilities can be added or removed without redeploying the platform.

Last updated on