programming/KVM_system_update/Readme.md

# Dom0 System Update Scripts

These scripts update Linux guests managed by KVM on Dom0, collect OS and
kernel information before and after the updates, compare the results, and
email a summary.

## Requirements

Run the scripts as `root` on Dom0, from the directory containing the scripts
and `excluded_vm.txt`. The scripts use the current working directory for their
input and output files.

Required tools:

- Bash
- `virsh` (libvirt/KVM)
- `ssh` and working root SSH access from Dom0 to each VM
- `hostnamectl`
- `dnf`
- `python3`
- `tee`, `find`, and `gzip`
- `msmtp` (for `check_update.sh`)

The scripts expect eligible guests to be Linux systems that support
`hostnamectl` and `dnf`. Exclude Windows guests.

## Configuration and input

### `excluded_vm.txt`

Place one VM name per line. Lines beginning with `#`, blank lines, and inline
comments are ignored. VM names must match the names reported by `virsh` and
must also resolve as SSH hostnames, for example `linux-vm.example.com`.

If the file is missing, the scripts warn and process all discovered VMs.

### `WAIT_FOR_START`

Both scripts wait after each VM is confirmed started, even if it was already
running. Set `WAIT_FOR_START` to a non-negative integer number of minutes; the
default is `1`.

```bash
WAIT_FOR_START=5 ./system_update.sh
WAIT_FOR_START=5 ./check_update.sh
```

After this wait, the scripts check SSH connectivity with up to six attempts,
spaced 10 seconds apart.

### `EMAIL_RECIPIENT`

`check_update.sh` sends the summary to `vplesnila@gmail.com` by default. Override
the address with an environment variable:

```bash
EMAIL_RECIPIENT=admin@example.com ./check_update.sh
```

Email is sent by `msmtp -t`; configure msmtp on Dom0 separately. The message
subject is `Update system on <Dom0 static hostname>`, and its HTML body displays
the summary tables in a monospace `<pre>` block.

## Scripts and recommended order

### `system_update.sh`

Run this script first:

```bash
./system_update.sh
```

For every non-excluded VM returned by `virsh list --all --name`, it starts the
VM if necessary, waits `WAIT_FOR_START` minutes, checks SSH, and records the
VM's `Operating System` and `Kernel` values from `hostnamectl` **before** the
update. It then runs `dnf update -y` over SSH and requests a graceful VM
shutdown. After all VMs, it records Dom0's before values and runs `dnf update
-y` on Dom0. This script does not reboot Dom0.

The script overwrites `status.before` once at the beginning of each execution.
It contains a JSON object with a `systems` array. Each collected system has
this shape:

```json
{
  "vm_name": "linux-vm.example.com",
  "before": {
    "operating_system": "Example Linux 9",
    "kernel": "5.14.0-example"
  }
}
```

Dom0 is recorded with `"vm_name": "Dom0"`. A VM whose startup, SSH connection,
or hostname query fails may not have a record in the file. Update failures are
logged; the script continues processing.

The script writes operational output to `log/system_update_YYYY-MM-DD_HH_MM.log`.

### `check_update.sh`

Run this after `system_update.sh` has successfully created `status.before`:

```bash
./check_update.sh
```

For each non-excluded VM, it starts the VM if needed, waits
`WAIT_FOR_START` minutes, checks SSH, records its `hostnamectl` values as
**after** values, and requests a graceful shutdown. It then records Dom0's
after values. It does not run updates.

Each run overwrites `status.after`, which uses the same `systems` array format
as `status.before`, with an `after` object instead of a `before` object:

```json
{
  "vm_name": "linux-vm.example.com",
  "after": {
    "operating_system": "Example Linux 9",
    "kernel": "5.14.0-example"
  }
}
```

The script joins records from `status.before` and `status.after` by `vm_name`
and prints two comparison tables:

- `operating_system`: VM name, before OS, after OS, and `Changed`
- `kernel`: VM name, before kernel, after kernel, and `Changed`

`Changed` is `NO` for equal values and `YES` otherwise. Missing values are shown
as `not available` and therefore compare as changed.

The tables are saved to `log/summary_YYYY-MM-DD_HH_MM.log` and included in the
summary email. Summary files older than 60 days are purged at script startup.

## Logs and retention

Both scripts create the `log/` directory when it does not exist. Filenames
include local Dom0 date and time to the minute; two executions of the same
script within one minute can use the same filename.

**Current retention caveat:** `system_update.sh` writes files named
`system_update_*.log`, but its cleanup commands currently match
`dnf_update_all_*.log` and `.log.gz`. Consequently, those commands do not
compress or delete the `system_update_*.log` files. The 60-day summary purge
in `check_update.sh` applies to `summary_*.log` files.

## Operational notes

- These scripts use the directory from which they are launched, not the
  directory containing the script, for `excluded_vm.txt`, status files, and
  `log/`.
- VM shutdowns are requested through `virsh shutdown`; the scripts do not wait
  for a VM to finish shutting down.
- `check_update.sh` requires a valid `status.before` JSON file. Run
  `system_update.sh` first so both files describe the same update cycle.
- Review the generated logs and comparison tables for per-VM errors before
  relying on the update results.