A few days ago I had the need to access a VM disk to perform a file synchronization for a migration from a physical server to a virtual machine managed with Bhyve on FreeBSD.
The disk configuration I use for the Bhyve hypervisor is based on ZFS rather than raw images which allows creating a dedicated volume for each VM, optimizing disk space along with compression, encryption, snapshots and all the advantages that ZFS can provide (and disadvantages). You can also opt for the qcow format in Linux/KVM and each technology provides its advantages and disadvantages.
But let us focus on the title of the post. Memory is fragile and with so many infinite options that each technology offers, sometimes some options and configurations are forgotten.
To perform the file synchronization on the destination VM my goal was to mount the VM volume on the hypervisor and from it perform an rsync from the old machine (still in production) to the mounted volume with the idea of minimizing downtime while IP changes and so on are made.
In ZFS for each dataset of volume type that is created, in the /dev directory a device will appear to access it as if it were just another disk:
# ls -la /dev/zvol/zroot/bhyve/testvm/disk0
crw-r----- 1 root operator 0xe3 10 nov. 17:16 /dev/zvol/zroot/bhyve/testvm/disk0
This ZFS feature is very convenient to be able to manipulate and perform actions on the volume as with any other device.
The problem that arose for me (and may serve as a reminder) is that accessing the device was initially not possible:
# gpart show /dev/zvol/zroot/bhyve/testvm/disk0
gpart: No such geom: /dev/zvol/zroot/bhyve/testvm/disk0.
The reason is that by default, when vm-bhyve creates the ZFS volume it sets the volmode as “dev” and this mode does not allow a complete view of the device created in the system:
volmode=default|full|geom|dev|none
This property specifies how volumes should be exposed to the OS.
Setting it to full exposes volumes as fully fledged block devices,
providing maximal functionality. The value geom is just an alias for
full and is kept for compatibility. Setting it to dev hides its
partitions. Volumes with property set to none are not exposed outside
ZFS, but can be snapshotted, cloned, replicated, etc, that can be
suitable for backup purposes. Value default means that volumes
exposition is controlled by system-wide tunable zvol_volmode, where
full, dev and none are encoded as 1, 2 and 3 respectively. The default
value is full.
The solution is very simple and is fixed with a simple command:
# zfs set volmode=full zroot/bhyve/testvm/disk0
And now:
# gpart show /dev/zvol/zroot/bhyve/testvm/disk0
=> 2048 4192223 zvol/zroot/bhyve/testvm/disk0 GPT (10G) [CORRUPT]
2048 6144 14 bios-boot (3.0M)
8192 253952 15 efi (124M)
262144 3932127 1 linux-data (1.9G)
We have full access to the disk, and the partitions also appear which is what we need to be able to mount them on the hypervisor:
# ls -la /dev/zvol/zroot/bhyve/testvm/disk0*
crw-r----- 1 root operator 0xdd 10 nov. 17:28 /dev/zvol/zroot/bhyve/testvm/disk0
crw-r----- 1 root operator 0xe2 10 nov. 17:28 /dev/zvol/zroot/bhyve/testvm/disk0p1
crw-r----- 1 root operator 0xe4 10 nov. 17:28 /dev/zvol/zroot/bhyve/testvm/disk0p14
crw-r----- 1 root operator 0xe5 10 nov. 17:28 /dev/zvol/zroot/bhyve/testvm/disk0p15
Now, mounting the desired partition is as simple as:
# mkdir /mnt/testvm
# mount -t ext2fs -o rw /dev/zvol/zroot/bhyve/testvm/disk0p1 /mnt/testvm/
Which gives us access to the VM disk at /mnt/testvm.
Remember to perform this operation always with the VM stopped and never start it without verifying that the volume has been unmounted.
# umount /mnt/testvm