2026-09-30 15:33:30

KVM_system_update/Readme.md

@@ -0,0 +1,161 @@
+# Dom0 System Update Scripts
+
+These scripts update Linux guests managed by KVM on Dom0, collect OS and
+kernel information before and after the updates, compare the results, and
+email a summary.
+
+## Requirements
+
+Run the scripts as `root` on Dom0, from the directory containing the scripts
+and `excluded_vm.txt`. The scripts use the current working directory for their
+input and output files.
+
+Required tools:
+
+- Bash
+- `virsh` (libvirt/KVM)
+- `ssh` and working root SSH access from Dom0 to each VM
+- `hostnamectl`
+- `dnf`
+- `python3`
+- `tee`, `find`, and `gzip`
+- `msmtp` (for `check_update.sh`)
+
+The scripts expect eligible guests to be Linux systems that support
+`hostnamectl` and `dnf`. Exclude Windows guests.
+
+## Configuration and input
+
+### `excluded_vm.txt`
+
+Place one VM name per line. Lines beginning with `#`, blank lines, and inline
+comments are ignored. VM names must match the names reported by `virsh` and
+must also resolve as SSH hostnames, for example `linux-vm.example.com`.
+
+If the file is missing, the scripts warn and process all discovered VMs.
+
+### `WAIT_FOR_START`
+
+Both scripts wait after each VM is confirmed started, even if it was already
+running. Set `WAIT_FOR_START` to a non-negative integer number of minutes; the
+default is `1`.
+
+```bash
+WAIT_FOR_START=5 ./system_update.sh
+WAIT_FOR_START=5 ./check_update.sh
+```
+
+After this wait, the scripts check SSH connectivity with up to six attempts,
+spaced 10 seconds apart.
+
+### `EMAIL_RECIPIENT`
+
+`check_update.sh` sends the summary to `vplesnila@gmail.com` by default. Override
+the address with an environment variable:
+
+```bash
+EMAIL_RECIPIENT=admin@example.com ./check_update.sh
+```
+
+Email is sent by `msmtp -t`; configure msmtp on Dom0 separately. The message
+subject is `Update system on <Dom0 static hostname>`, and its HTML body displays
+the summary tables in a monospace `<pre>` block.
+
+## Scripts and recommended order
+
+### `system_update.sh`
+
+Run this script first:
+
+```bash
+./system_update.sh
+```
+
+For every non-excluded VM returned by `virsh list --all --name`, it starts the
+VM if necessary, waits `WAIT_FOR_START` minutes, checks SSH, and records the
+VM's `Operating System` and `Kernel` values from `hostnamectl` **before** the
+update. It then runs `dnf update -y` over SSH and requests a graceful VM
+shutdown. After all VMs, it records Dom0's before values and runs `dnf update
+-y` on Dom0. This script does not reboot Dom0.
+
+The script overwrites `status.before` once at the beginning of each execution.
+It contains a JSON object with a `systems` array. Each collected system has
+this shape:
+
+```json
+{
+  "vm_name": "linux-vm.example.com",
+  "before": {
+    "operating_system": "Example Linux 9",
+    "kernel": "5.14.0-example"
+  }
+}
+```
+
+Dom0 is recorded with `"vm_name": "Dom0"`. A VM whose startup, SSH connection,
+or hostname query fails may not have a record in the file. Update failures are
+logged; the script continues processing.
+
+The script writes operational output to `log/system_update_YYYY-MM-DD_HH_MM.log`.
+
+### `check_update.sh`
+
+Run this after `system_update.sh` has successfully created `status.before`:
+
+```bash
+./check_update.sh
+```
+
+For each non-excluded VM, it starts the VM if needed, waits
+`WAIT_FOR_START` minutes, checks SSH, records its `hostnamectl` values as
+**after** values, and requests a graceful shutdown. It then records Dom0's
+after values. It does not run updates.
+
+Each run overwrites `status.after`, which uses the same `systems` array format
+as `status.before`, with an `after` object instead of a `before` object:
+
+```json
+{
+  "vm_name": "linux-vm.example.com",
+  "after": {
+    "operating_system": "Example Linux 9",
+    "kernel": "5.14.0-example"
+  }
+}
+```
+
+The script joins records from `status.before` and `status.after` by `vm_name`
+and prints two comparison tables:
+
+- `operating_system`: VM name, before OS, after OS, and `Changed`
+- `kernel`: VM name, before kernel, after kernel, and `Changed`
+
+`Changed` is `NO` for equal values and `YES` otherwise. Missing values are shown
+as `not available` and therefore compare as changed.
+
+The tables are saved to `log/summary_YYYY-MM-DD_HH_MM.log` and included in the
+summary email. Summary files older than 60 days are purged at script startup.
+
+## Logs and retention
+
+Both scripts create the `log/` directory when it does not exist. Filenames
+include local Dom0 date and time to the minute; two executions of the same
+script within one minute can use the same filename.
+
+**Current retention caveat:** `system_update.sh` writes files named
+`system_update_*.log`, but its cleanup commands currently match
+`dnf_update_all_*.log` and `.log.gz`. Consequently, those commands do not
+compress or delete the `system_update_*.log` files. The 60-day summary purge
+in `check_update.sh` applies to `summary_*.log` files.
+
+## Operational notes
+
+- These scripts use the directory from which they are launched, not the
+  directory containing the script, for `excluded_vm.txt`, status files, and
+  `log/`.
+- VM shutdowns are requested through `virsh shutdown`; the scripts do not wait
+  for a VM to finish shutting down.
+- `check_update.sh` requires a valid `status.before` JSON file. Run
+  `system_update.sh` first so both files describe the same update cycle.
+- Review the generated logs and comparison tables for per-VM errors before
+  relying on the update results.