Vai al contenuto principale

Hadoop Day-2 Operations: Patching, Certificates, Configuration

After a Classic migration, the cluster is operated through Apache Ambari (Ilum build), directly or through Ilum Cluster Ops. Everyday operations (rolling restarts, Restart All, service checks, maintenance mode, decommission and recommission of nodes) are performed there. Bifrost adds the operations that a version-aligned estate needs on top: package patching from controlled mirrors, certificate renewal from the customer's PKI, control of configuration drift, upgrades of the management plane itself, and an acceptance drill that proves the day-2 surface before go-live.

Common Behaviour​

  • Plan first. Every day-2 command plans by default and acts only with --execute.
  • Management plane access. Commands that act through Ambari take --ambari-host, --ambari-port, --ambari-tlse --ambari-ca-bundle. The Ambari admin password is read from BIFROST_AMBARI_PASSWORD.
  • Evidence. Every run records its plan, its actions, and its results in the cluster's state directory ($BIFROST_STATE_DIR/<cluster>/).
  • Health gates. Commands that restart daemons use the rolling plan described below.

Rolling Restarts with Health Gates​

Commands that restart daemons (patch, renew-certificates, daemon-limits, impala-ha) share one rolling plan. Daemons are restarted in dependency order, standby before active, one host at a time:

  1. ZooKeeper, followers first and the leader last.
  2. Ranger Admin, then Ranger KMS.
  3. HDFS: JournalNodes one at a time with an edit-log roll after each, then the standby NameNode, a graceful failover, the former active NameNode, and the DataNodes.
  4. YARN: the standby ResourceManager, the active ResourceManager, then the NodeManagers.
  5. MapReduce history server, HBase (masters, then RegionServers), Hive (metastore, HiveServer2).
  6. Kafka, Solr, Livy, Oozie, Zeppelin, Atlas, and the Spark history servers.
  7. Impala (statestore, catalog, daemons), then Hue.

After each step, a health gate checks the affected service: ZooKeeper quorum, HDFS high-availability state and safe mode, the JournalNode quorum, YARN, HBase including hbase:meta, Kafka under-replicated partitions, Solr, Impala, and the TLS certificate chain. A gate that fails stops the roll. A gate that cannot measure its service also stops the roll; an unmeasured check is never counted as a pass.

Patching​

A package update is a two-step operation. mirror-lock locks the local package mirror on every node on one release of a module; checksums are verified, and superseded releases are kept for a downgrade. patch then rolls the release through Ambari.

# 1. Lock every node's mirror on the release
bifrost classic mirror-lock --cluster prod01 \
  --prefix '<package-prefix>' \
  --release '<version>-<release>' \
  --from /srv/bifrost/rpms/<module> \
  --execute

# 2. Roll it through the cluster
bifrost classic patch --cluster prod01 \
  --package '<package-glob>' \
  --component KAFKA_BROKER \
  --strategy rolling \
  --ambari-host ambari.example.internal \
  --execute
  • Baseline. patch records the state of the selected components first and refuses to start when they are not steady.
  • Strategies. rolling handles one host at a time: stop, upgrade from the mirror, start, then a health check. coordinated stops all selected components, patches every host, and starts them together.
  • Host order. --hosts sets the order, for example a canary host first and standby before active.
  • Replaced files. After the roll, every daemon that still runs a file the update replaced is restarted through the rolling plan, including daemons of components that were not selected.
  • Downgrade. --rollback-to installs an exact earlier version and release; lock the mirror on that release first.

Certificate Renewal​

bifrost classic renew-certificates renews the TLS material of the adopted cluster from files the customer's PKI issued. Bifrost does not generate keys or certificate requests.

The upload directory has this layout:

<upload>/
  ca/                    chain.pem or *.pem: the trusted CA certificates (required)
  hosts/<fqdn>/          cert.pem + key.pem (optional key.pw), or host.jks / host.p12 + host.pw
  hosts/_shared/         one wildcard or multi-SAN certificate for every host it covers
  MANIFEST.sha256        optional checksums, verified when present

File types are detected by content. For JKS and PKCS12 stores, the key password must equal the store password.

# Verify the upload and plan the roll
bifrost classic renew-certificates --cluster prod01 --from /srv/pki/renewal

# Stage, roll, and verify, one host at a time
bifrost classic renew-certificates --cluster prod01 --from /srv/pki/renewal --execute
  • Plan run. Verifies the upload on the controller: the certificate and key belong together, the chain is complete, the names cover the host, nothing has expired, and the key opens. It prints which components roll on each host and writes nothing.
  • Execution. On each host, the new material is staged and the previous files are moved aside, never deleted. The TLS components are then restarted through the rolling plan, and liveness and the certificate chain are checked before the next host.
  • Refusals. The command refuses before staging anything when a host is not covered by the upload, when the set of trusted CA certificates changes without --accept-anchor-change, when a TLS component cannot be restarted, when the cluster is not steady, or when a store password would change.
  • Rollback. --rollback-to <state-dir>/renew/<run>/manifest.json --execute restores the material that run replaced and restarts the same components.
  • Scope. Renewal changes no configuration value. The Ambari agent certificates and the Kerberos keytabs are not part of it.

Configuration Plane Control​

The takeover certifies the configuration Ambari holds as the adopted baseline. Changes made later outside Bifrost become visible and reversible:

  • validate-post names every change made after certification by service, configuration type, and key. A typical source is the Ambari stack advisor, which can rewrite carried values when a configuration wizard is used.
  • bifrost classic restore-config-plane puts every changed service back to its certified version through Ambari configuration versions. It restarts no daemon, and it refuses a service that a Bifrost command changed after certification.
  • bifrost classic refresh-clients re-renders client configuration files that Ambari reports as stale, without restarting any daemon.
  • bifrost classic certify-config-plane records a new baseline after an intended change. --declare-adopted-types asks the stack advisor to keep the carried values.
bifrost classic restore-config-plane --cluster prod01 --ambari-host ambari.example.internal --refresh-clients --execute

Ambari Upgrades​

The management plane is upgraded, or downgraded, from a new Ambari bundle:

bifrost classic stage-ambari --cluster prod01 \
  --from /srv/bifrost/ambari-bundle-new \
  --roll-packages --reapply-service-defs \
  --execute

The Ambari agents are rolled one host at a time, each confirmed by a fresh heartbeat. The server host is rolled last, followed by the Ambari schema upgrade, the re-application of the service definitions, and one server restart. Before anything changes, a PostgreSQL Ambari database on the server host is backed up automatically. For an Oracle or remote database, the database team's backup is referenced with --ambari-db-backup-ref.

Stack packages are upgraded with mirror-lock e patch, and the Atlas server with bifrost classic atlas-upgrade.

Day-2 Acceptance Drill​

bifrost classic day2-drill exercises the day-2 surface of the management plane on the adopted cluster before go-live. Every block is bracketed by canary captures before and after, every restart is proven by a new process, and every result is recorded.

BlockCovers
B0Null control: records which signatures change without any operation.
B1Control plane: agent and server restarts, access roles, LDAP synchronization, alerts.
B2Ambari upgrade through stage-ambari --roll-packages.
B3Restart All for every service.
B4Rolling restarts under tenant load.
B5High-availability failover and recovery of stopped daemons through automatic restart.
B6Version updates with mirror-lock e patch, including a downgrade.
B7Configuration changes and their reversal, configuration groups, YARN queues.
B8Decommission and recommission of NodeManagers and RegionServers, and a role move.
B10Features of the Ilum build: per-component Java home, query termination, snapshots, automatic actions, package mirrors.
B11Maintenance mode and host reboots.
B12Stop All and Start All.
B13Acceptance: full canary against the baseline, validate-ambari, validate-post, the rollback plan, and the configuration plane.
bifrost classic day2-drill --cluster prod01 --ambari-host ambari.example.internal --execute
  • Opt-in. --execute requires bifrost_day2_drill_allowed: true in the inventory, because the drill restarts and reconfigures the cluster.
  • Selection and resume. --blocks selects blocks; --resume <run-id> continues an interrupted run at its next step, and a block that passed is not run again.
  • Evidence. Every step's result, the canary reports, and the tenant load record are kept under $BIFROST_STATE_DIR/<cluster>/day2-drill/<run-id>/.

Other Operations​

CommandPurpose
impala-ha enable o impala-ha disableAdds or removes the second Impala statestore and catalog service on --host, restarts Impala, and waits for one active and one standby instance.
daemon-limitsMeasures every daemon's process and open-file limits against the limits it had on the source, raises them where needed, and restarts the daemons that ran below them.
restart-kmsRestarts Ranger KMS one host at a time. HSM-backed key stores require --hsm-ack with the HSM team's agreement.
replication-activateRuns a dry run of each imported replication schedule and resumes the schedules that read their source.
atlas-upgradeUpgrades the Atlas server package with HBase snapshots, a version check, and a search index rebuild.

Node Lifecycle​

Nodes are decommissioned and recommissioned through Ambari: NodeManagers through the ResourceManager, RegionServers by draining their regions first, and DataNodes through HDFS decommissioning.

Passaggi successivi​

Last updated on