162 lines
5.2 KiB
Markdown
162 lines
5.2 KiB
Markdown
# Dom0 System Update Scripts
|
|
|
|
These scripts update Linux guests managed by KVM on Dom0, collect OS and
|
|
kernel information before and after the updates, compare the results, and
|
|
email a summary.
|
|
|
|
## Requirements
|
|
|
|
Run the scripts as `root` on Dom0, from the directory containing the scripts
|
|
and `excluded_vm.txt`. The scripts use the current working directory for their
|
|
input and output files.
|
|
|
|
Required tools:
|
|
|
|
- Bash
|
|
- `virsh` (libvirt/KVM)
|
|
- `ssh` and working root SSH access from Dom0 to each VM
|
|
- `hostnamectl`
|
|
- `dnf`
|
|
- `python3`
|
|
- `tee`, `find`, and `gzip`
|
|
- `msmtp` (for `check_update.sh`)
|
|
|
|
The scripts expect eligible guests to be Linux systems that support
|
|
`hostnamectl` and `dnf`. Exclude Windows guests.
|
|
|
|
## Configuration and input
|
|
|
|
### `excluded_vm.txt`
|
|
|
|
Place one VM name per line. Lines beginning with `#`, blank lines, and inline
|
|
comments are ignored. VM names must match the names reported by `virsh` and
|
|
must also resolve as SSH hostnames, for example `linux-vm.example.com`.
|
|
|
|
If the file is missing, the scripts warn and process all discovered VMs.
|
|
|
|
### `WAIT_FOR_START`
|
|
|
|
Both scripts wait after each VM is confirmed started, even if it was already
|
|
running. Set `WAIT_FOR_START` to a non-negative integer number of minutes; the
|
|
default is `1`.
|
|
|
|
```bash
|
|
WAIT_FOR_START=5 ./system_update.sh
|
|
WAIT_FOR_START=5 ./check_update.sh
|
|
```
|
|
|
|
After this wait, the scripts check SSH connectivity with up to six attempts,
|
|
spaced 10 seconds apart.
|
|
|
|
### `EMAIL_RECIPIENT`
|
|
|
|
`check_update.sh` sends the summary to `vplesnila@gmail.com` by default. Override
|
|
the address with an environment variable:
|
|
|
|
```bash
|
|
EMAIL_RECIPIENT=admin@example.com ./check_update.sh
|
|
```
|
|
|
|
Email is sent by `msmtp -t`; configure msmtp on Dom0 separately. The message
|
|
subject is `Update system on <Dom0 static hostname>`, and its HTML body displays
|
|
the summary tables in a monospace `<pre>` block.
|
|
|
|
## Scripts and recommended order
|
|
|
|
### `system_update.sh`
|
|
|
|
Run this script first:
|
|
|
|
```bash
|
|
./system_update.sh
|
|
```
|
|
|
|
For every non-excluded VM returned by `virsh list --all --name`, it starts the
|
|
VM if necessary, waits `WAIT_FOR_START` minutes, checks SSH, and records the
|
|
VM's `Operating System` and `Kernel` values from `hostnamectl` **before** the
|
|
update. It then runs `dnf update -y` over SSH and requests a graceful VM
|
|
shutdown. After all VMs, it records Dom0's before values and runs `dnf update
|
|
-y` on Dom0. This script does not reboot Dom0.
|
|
|
|
The script overwrites `status.before` once at the beginning of each execution.
|
|
It contains a JSON object with a `systems` array. Each collected system has
|
|
this shape:
|
|
|
|
```json
|
|
{
|
|
"vm_name": "linux-vm.example.com",
|
|
"before": {
|
|
"operating_system": "Example Linux 9",
|
|
"kernel": "5.14.0-example"
|
|
}
|
|
}
|
|
```
|
|
|
|
Dom0 is recorded with `"vm_name": "Dom0"`. A VM whose startup, SSH connection,
|
|
or hostname query fails may not have a record in the file. Update failures are
|
|
logged; the script continues processing.
|
|
|
|
The script writes operational output to `log/system_update_YYYY-MM-DD_HH_MM.log`.
|
|
|
|
### `check_update.sh`
|
|
|
|
Run this after `system_update.sh` has successfully created `status.before`:
|
|
|
|
```bash
|
|
./check_update.sh
|
|
```
|
|
|
|
For each non-excluded VM, it starts the VM if needed, waits
|
|
`WAIT_FOR_START` minutes, checks SSH, records its `hostnamectl` values as
|
|
**after** values, and requests a graceful shutdown. It then records Dom0's
|
|
after values. It does not run updates.
|
|
|
|
Each run overwrites `status.after`, which uses the same `systems` array format
|
|
as `status.before`, with an `after` object instead of a `before` object:
|
|
|
|
```json
|
|
{
|
|
"vm_name": "linux-vm.example.com",
|
|
"after": {
|
|
"operating_system": "Example Linux 9",
|
|
"kernel": "5.14.0-example"
|
|
}
|
|
}
|
|
```
|
|
|
|
The script joins records from `status.before` and `status.after` by `vm_name`
|
|
and prints two comparison tables:
|
|
|
|
- `operating_system`: VM name, before OS, after OS, and `Changed`
|
|
- `kernel`: VM name, before kernel, after kernel, and `Changed`
|
|
|
|
`Changed` is `NO` for equal values and `YES` otherwise. Missing values are shown
|
|
as `not available` and therefore compare as changed.
|
|
|
|
The tables are saved to `log/summary_YYYY-MM-DD_HH_MM.log` and included in the
|
|
summary email. Summary files older than 60 days are purged at script startup.
|
|
|
|
## Logs and retention
|
|
|
|
Both scripts create the `log/` directory when it does not exist. Filenames
|
|
include local Dom0 date and time to the minute; two executions of the same
|
|
script within one minute can use the same filename.
|
|
|
|
**Current retention caveat:** `system_update.sh` writes files named
|
|
`system_update_*.log`, but its cleanup commands currently match
|
|
`dnf_update_all_*.log` and `.log.gz`. Consequently, those commands do not
|
|
compress or delete the `system_update_*.log` files. The 60-day summary purge
|
|
in `check_update.sh` applies to `summary_*.log` files.
|
|
|
|
## Operational notes
|
|
|
|
- These scripts use the directory from which they are launched, not the
|
|
directory containing the script, for `excluded_vm.txt`, status files, and
|
|
`log/`.
|
|
- VM shutdowns are requested through `virsh shutdown`; the scripts do not wait
|
|
for a VM to finish shutting down.
|
|
- `check_update.sh` requires a valid `status.before` JSON file. Run
|
|
`system_update.sh` first so both files describe the same update cycle.
|
|
- Review the generated logs and comparison tables for per-VM errors before
|
|
relying on the update results.
|