*********************** Hypervisor Abstraction *********************** ADARE supports multiple virtualisation backends through a factory-and-registry pattern. The hypervisor layer is responsible for creating, starting, stopping, and destroying VMs, as well as executing commands inside the guest, taking snapshots, and transferring files. .. contents:: On this page :local: :depth: 2 Factory Pattern =============== The entry point is ``adare.hypervisor``, which maintains a global ``_HYPERVISOR_REGISTRY`` dictionary. Each backend registers itself by calling ``register_hypervisor(name, manager_class, vm_class)``. Registration is **lazy**: when ``get_hypervisor_manager()`` is called, the requested backend module is imported for the first time and its ``register()`` function populates the registry. This avoids importing libvirt or VBoxManage wrappers until they are actually needed. .. code-block:: python # In adare.hypervisor.qemu.__init__ def register(): from adare.hypervisor import register_hypervisor from adare.hypervisor.qemu.manager import QEMUManager from adare.hypervisor.qemu.vm import QEMUVM register_hypervisor('qemu', QEMUManager, QEMUVM) The caller only needs: .. code-block:: python from adare.hypervisor import get_hypervisor_manager manager = get_hypervisor_manager('qemu') # or 'virtualbox' If no name is passed, the ``DEFAULT_HYPERVISOR`` from ``adare.config`` is used. Abstract Base Classes ===================== Two abstract classes define the contract that every backend must fulfil. AbstractHypervisorManager ------------------------- Defined in ``adare.hypervisor.base.manager``. Provides thread-safe queued execution of hypervisor commands through a worker thread and command queue. Key abstract methods: - ``_worker_loop()`` -- processes commands from ``self._cmd_queue`` in a dedicated thread. - ``run(func, *args, **kwargs)`` -- enqueue a synchronous function and block until it completes. - ``run_async(func, *args, **kwargs)`` -- run an async function directly. - ``import_vm_async(vm_file_path, vm_name, ...)`` -- high-level VM import. AbstractVM ---------- Defined in ``adare.hypervisor.base.vm``. Represents a single virtual machine and exposes lifecycle, guest-control, and import/export operations. Lifecycle methods: ``create``, ``start``, ``stop``, ``destroy``, ``get_state``, ``vm_exists``, ``wait_until_fully_booted``. Guest control: ``run_command``, ``copy_from_guest``, ``queue_command``, ``execute_queued_commands``, ``cleanup_background_processes``. Import/export: ``create_from_ovf_or_ova``. Static lookup: ``get_vm_by_name``, ``get_vm_uuid_by_name``, ``verify_vm_exists_by_uuid``, ``get_vm_info_by_uuid``. QEMU Implementation =================== The primary and recommended backend. QEMU VMs are managed through libvirt for domain lifecycle and the QEMU Guest Agent (QGA) for in-guest command execution. Mixin Architecture ------------------ ``QEMUVM`` uses multiple inheritance to compose its capabilities from focused mixin classes: .. code-block:: python class QEMUVM( RegistryMixin, # VM UUID registry (JSON persistence) ConfigurationMixin, # VM config load/save (QEMUVMConfig) DiskManagementMixin, # qcow2 disk operations CommandExecutionMixin, # QGA command execution SnapshotMixin, # qcow2 internal snapshots via libvirt NetworkingMixin, # Port forwarding, network config AbstractVM, # Abstract base contract ): ... Each mixin lives in ``adare.hypervisor.qemu.mixins`` and addresses a single concern. Libvirt Integration ------------------- ``QEMUManager`` opens a libvirt connection (``qemu:///system`` on Linux, ``qemu:///session`` on macOS) and uses it for: - Defining and undefining domains (XML-based VM definitions). - Starting, stopping, and destroying VMs. - Creating and restoring qcow2 internal snapshots. - Querying VM state. The libvirt XML is generated by ``libvirt_xml_builder.py`` from a ``QEMUVMConfig`` dataclass. The builder handles architecture-specific details (x86_64 vs aarch64), UEFI firmware selection, VirtioFS device configuration, SMB share paths, and network port forwarding. .. _instance-firmware-state: Instance-scoped firmware state ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ADARE undefines and redefines the same domain constantly -- the XML is rebuilt on every start, and a cold-boot retry destroys and restarts the domain -- while the VM *instance* behind it is long-lived and reused across runs. Two pieces of guest state live outside the disk image and are governed by undefine **flags** rather than by the run overlay, so both are easy to reset by accident: .. list-table:: :header-rows: 1 :widths: 22 38 40 * - State - Where it lives - libvirt default on undefine * - UEFI NVRAM varstore - ``/-nvram.fd`` - Refuses the undefine outright * - Emulated TPM (swtpm) - ``~/.config/libvirt/qemu/swtpm//`` - Depends on ``persistent_state`` (see below) ``adare.hypervisor.qemu.libvirt_undefine`` centralises the policy: ``keep_firmware_state_flags()`` (``KEEP_NVRAM | KEEP_TPM``) for anything that will redefine the domain, and ``delete_firmware_state_flags()`` for genuine instance removal, so neither file outlives the instance it belonged to. The TPM half needs two more pieces to actually work, and is worth spelling out because each piece failing silently looks like a plausible success: 1. **``persistent_state='yes'`` on the TPM backend element** (````, ``libvirt_xml_builder._add_tpm``). Without it -- libvirt's default -- the swtpm state is treated as ephemeral and is **deleted on any undefine no matter which undefineFlags are passed**, so ``KEEP_TPM`` is a no-op. This was ADARE's actual behaviour before this attribute was added: Windows 11 requires a TPM, so every Windows domain declares ````, and every cold boot -- including every cold-boot retry -- got a **brand-new vTPM** (new EK, new SRK, cleared owner auth) because ``swtpm_setup`` manufactures one on every define. With ``persistent_state='yes'`` set, the state survives undefine by default, and ``KEEP_TPM``/``TPM`` behave as the table implies. 2. **A domain UUID that is stable across runs of the same environment.** swtpm keys the state directory by domain UUID, and ADARE undefines/redefines a *run-scoped* ``vm_name`` -- a fresh instance name is minted per experiment run (``QEMULifecycleStrategy.prepare_vm_for_experiment``). A UUID picked with plain ``uuid.uuid4()`` at config-creation time therefore differed on every run even with ``KEEP_TPM`` correctly applied: the state was "kept" under a UUID the next run's domain never pointed at again, which is an orphan, not a reused vTPM. ``ConfigurationMixin._domain_uuid_for`` derives the UUID from the environment's own stable id instead, so every run of the same environment gets the same domain UUID and therefore the same swtpm state directory. Both matter: persistent_state alone still orphans one swtpm directory per run; a stable UUID alone still gets the state deleted on every undefine. Get either wrong and the failure is invisible except in ``~/.cache/libvirt/qemu/log/-swtpm.log``, where a repeated ``Starting vTPM manufacturing`` line is the tell -- one per boot attempt -- or by counting ``~/.config/libvirt/qemu/swtpm/`` directories against removed Windows VMs. An irreproducible-per-boot vTPM is wrong for two independent reasons: * **Reproducibility.** Every TPM-derived artifact (owner auth, TPM-bound keys, PCR measurements, ``Get-Tpm`` output) differs between runs and does not match the sealed base image -- irreproducible by construction, in a framework whose purpose is reproducible artifacts. * **Fidelity.** A real machine has exactly one TPM for its lifetime. VM Creator ---------- The ``adare.hypervisor.qemu.vm_creator`` package supports fully automated VM creation from ISO images. Separate creator classes handle: - **Linux** (``linux_creator.py``) -- Ubuntu autoinstall via cloud-init. - **Windows** (``windows_creator.py``) -- unattended install via ``autounattend.xml`` templates for Windows 10, 11, and 11 ARM64. - **Manual** (``manual_creator.py``) -- interactive installation. VirtualBox Support ================== The VirtualBox backend (``adare.hypervisor.virtualbox``) follows the same abstract interface. It registers ``VirtualBoxManager`` and ``VirtualBoxVM`` with the factory and uses ``VBoxManage`` commands for all operations: shared folders for file transfer, ``VBoxManage guestcontrol`` for in-guest commands, and ``VBoxManage snapshot`` for snapshot management. This backend predates the QEMU implementation and is maintained for compatibility. QEMU is the default for new projects. VM Lifecycle ============ Regardless of hypervisor, an experiment run follows this lifecycle: 1. **Prepare** -- create or look up the VM instance, allocate ports and instance name. 2. **Setup networking** -- configure port forwarding for the WebSocket agent connection. 3. **Setup file transfer** -- prepare the mechanism for getting files into the guest (see :doc:`file-sharing`). 4. **Start and initialise** -- boot the VM, wait for the guest OS to become accessible, mount shared directories or upload files. 5. **Install agent** -- install ``adarevm`` and ``adarelib`` wheels inside the guest (skipped when correct versions are already present). 6. **Start agent** -- launch the ``adarevm`` WebSocket server inside the guest. 7. **Execute experiment** -- run playbook actions (GUI clicks, keyboard input, shell commands, tests) via WebSocket tool calls. 8. **Retrieve artifacts** -- collect logs, screenshots, and forensic artifacts from the guest. 9. **Cleanup** -- stop the VM, remove ephemeral resources. This lifecycle is defined by ``AbstractVMLifecycleStrategy`` (``adare.hypervisor.base.lifecycle``) and implemented by ``QEMULifecycleStrategy`` (``adare.hypervisor.qemu.lifecycle``).