To make the costs associated with any cloud service provider profitable, the majority (if not all) play with selling more resources than they physically have in their infrastructure, which is what is commonly known as “overprovisioning”. The idea plays on the fact that not all customers will consume the maximum of assigned resources, thus allowing the sale of more disk space, RAM or CPU than is actually available.
With this approach each provider will then have to be careful to offer a good service, optimizing costs while delivering what it promises without falling into resource saturation failures, but that is another topic.
In this post I will talk about how to optimize disk usage of raw image files and as an example I will use Proxmox, a very good platform for installing a complete hypervisor with which to manage your containers and virtual VMs.
Proxmox offers multiple options for storing the “disks” of its containers/VMs over different “storage backends”: ZFS, btrfs, LVM, dir, Ceph… each with its advantages and disadvantages recommended according to the use case of each project.
The simplest is to use a local directory, generally /var/lib/vz where to store raw image files for each container or in the case of VMs as an option qcow2 images. We will focus on the raw format which are basically sparse files which although they can be defined with a “virtual” space of several GBs their actual content will initially be minimal and will grow as data is written to them.
The problem is that this type of file, by default only “grows” and does not automatically free the space occupied by deleted data. In this way you can have raw files that although their actual data content is for example 20 GB, after several months of use creating and deleting data, their size on the hypervisor disk may be much larger.
To remedy this problem we can use the fstrim application. This program frees unused blocks from mounted filesystems. The procedure is simple, we mount the raw file, run fstrim on the mount point and unmount the raw file. With this simple operation we will be able to play with better guarantees and optimize disk usage to allow “selling” more optimally more space than the hypervisor actually has.
Here is a simple script to perform this operation on Proxmox in an automated way:
#!/bin/bash
vmid=$(ls /etc/pve/nodes/`hostname`/lxc/)
list=$(echo "$vmid" | sed -r 's/.conf//g')
for i in $list; do
pct mount $i
fstrim /var/lib/lxc/$i/rootfs
pct unmount $i
done