Run Generic Virtual Machines from your NixOS Configuration

The ctrl-os.vms module allows you to declaratively define and run generic virtual machines in your NixOS configuration. It is not limited to NixOS VMs.

Prerequisistes

ctrl-os.vms is tested with Linux VMs only. Other operating systems may work as well. VM images must provided as RAW images.

This guide is based on the Ubuntu Cloud Image. To allow network access, you will need to know the name of the network interface that provides access to the Internet.

Be sure to enable the CTRL-OS modules overlay as documented here.

Then we need to prepare the Ubuntu image. We download the image and convert it to a RAW image format. Add the following to your NixOS configuration files to automate the process:

  image = pkgs.fetchurl {
    # You can pick any image here. We use a specific version (not "current"), so we can pin the dependency.
    url = "https://cloud-images.ubuntu.com/noble/20260108/noble-server-cloudimg-amd64.img";
    sha256 = "sha256-AHhsCTan3ZGmsHlBymC7VmUpdeDnL52s9zyIetpCCWY=";
  };
  imageRaw = pkgs.runCommand "ubuntu-image.raw" {
    nativeBuildInputs = [ pkgs.qemu-utils ];
  } ''
    qemu-img convert -O raw ${image} $out
  '';

To allow internet access for the VM, a gateway interface must be specified. Choose a physical network device connected to the Internet for this to work:

  ctrl-os.vms = {
    gatewayInterface = "eth0";
  };

Configuring Virtual Machines

This section gives examples how to setup VMs. It's not meant as an exhaustive documentation. For that refer to the documentation of the underlying VM management software, the Separation Control Layer (SCL). The documentation of the SCL also goes into detail how to interface with the SCL to inspect VMs, volumes, and networks.

Configuring a Basic VM

The actual VMs are configured via ctrl-os.vms.virtualMachines. In the following sample, the pre-configured default network is used. Refer to extended networking for an advanced networking setup, or a network separation between different VMs.

  ctrl-os.vms = {
    gatewayInterface = "eth0";
    virtualMachines = {
      demo = {
        # The image to boot (must be a RAW image for now).
        image = imageRaw; # We use the previously prepared disk image.
        imageSize = 4096; # Specify the disk size (MiB) of the virtual machine image here.

        # The characteristics of the VM.
        cores = 2; # Specify the amount of CPU cores the VM should have.
          memorySize = 2048; # MiB
          network = "default";

          # Start the VM at boot time.
          autoStart = true;
        };
      };
    };

With this configuration, the VM is started, but no access from the host to the VM is possible. We will dive into how access from the host can be allowed in the next section.

Allow Access to the VM network

VMs are separated from the host network via network namespaces. The following diagram gives a conceptual overview how the gateway

ctrl-os.vms network

Each network of ctrl-os.vms represents a separate network namespace. A network can have multiple VMs assigned to it. Within a network, VMs can communicate with each other. To access the internet a virtual network interface is provided for each network. The virtual network interface has a gatewayIP assigned. VMs can configure the gatewayIP as gateway to the internet. The VM network is connected to the internet via the interface specified in the ctrl-os.vms.gatewayInterface option.

On the host, a virtual network interface is provided, with the externalIP assigned to it. Access to VMs is possible via the externalIP and port forwarding rules.

ctrl-os.vms offers a default network with the following parameters:

externalIP: The IP address that the host can use to access the virtual machine network. Default: 192.168.10.1
gatewayIP: The gateway in the VM network that virtual machines can use to access the internet (not the host local network). Default: 192.168.20.1
internalNetmask: The netmask of the internal network namespace. Default: 255.255.255.0

Info

Only IPv4 is supported at the moment.

In addition, the network options of ctrl-os.vms offers the following configuration parameters to allow accesses from the host:

allowedTCPPorts
allowedUDPPorts

For both options the following pattern is used to allow port forwarding from a host port to the VM port: <hostPort:vmIp:vmPort>

Given the example from above, ssh access to the virtual machine is established via a TCP port forwarding rule 2222:192.168.20.2:22.

The complete configuration is shown below:

  ctrl-os.vms = {
    gatewayInterface = "eth0";
    networks.default.allowedTCPPorts = [
      "2222:192.168.20.2:22" # Enable ssh access on host IP 192.168.10.1:2222 to VM 192.168.20.2:22
    ];

    # ...
  };

In order for this to work the VM needs to actually start a SSH server at port 22.

Configure the VM Using `cloud-init`

The configuration of the VM is possible in ctrl-os.vms via cloud-init. ctrl-os.vms offers the following options to automate VM configuration:

cloudInitUserConfigFile
cloudInitNetworkConfigFile

The example from above is extended by the VM configuration as follows:

  ctrl-os.vms = let
      yaml = pkgs.formats.yaml { };
  in {
    gatewayInterface = "eth0";
    networks.default.allowedTCPPorts = [
      "2222:192.168.20.2:22" # Enable ssh access on host IP 192.168.10.1:2222 to guest virtual machine 192.168.20.2:22
    ];

    virtualMachines = {
      demo = {
        # As before.
        image = imageRaw;
        imageSize = 4096;
        cores = 2;
        memorySize = 2048;
        autoStart = true;
        network = "default";

        cloudInitUserConfigFile = yaml.generate "cloud-init-user-config.yaml" {
          system_info.default_user.name = "nixos"; # The user name inside the virtual machine.
          password = "nixos"; # The initial password of the nixos user.

          # For demo and testing purposes only!
          chpasswd.expire = false; # Disable password expiry.
          ssh_pwauth = true; # Enable password authentication.
        };

        cloudInitNetworkConfigFile = yaml.generate "cloud-init-network-config.yaml" {
          version = 2;
          ethernets.id0 = {
            # Configure the VM IP for every ethernet device starting with 'enp'
            match.name = "enp*";

            addresses = [
              # Configure the VM IP to 192.168.20.2 (matches the port forwarding rule from above).
              "192.168.20.2/24"
            ];

            # Configure the internalIP of the network as gateway IP to allow internet access.
            gateway4 = "192.168.20.1";
          };
        };
      };
    };
  };

The example above directly configures the cloud-init configuration files in place using Nix helpers. Specifying the configuration is also possible via files on disk. This makes it possible to provide the VMs with secrets:

  ctrl-os.vms.virtualMachines.demo = {
    cloudInitUserConfigFile = "/path/to/cloud-init-user-config.yaml";
    cloudInitNetworkConfigFile = "/path/to/cloud-init-network-config.yaml";
  };

Now you can access the virtual machine via the following command:

ssh -p 2222 nixos@192.168.10.1

Extended Networking for Virtual Machines

In certain scenarios, we want to separate VMs into different networks.

The following snippet configures a network named demo. It is also possible to disable the default network by setting it to null.

  ctrl-os.vms = {
    gatewayInterface = "eth0";
    networks = {
      default = null;
      demo = {
        externalIP = "192.168.44.1";
        gatewayIP = "10.0.0.1";
        internalNetmask = "255.0.0.0";
        allowedTCPPorts = [
          "2222:10.0.0.2:22"
        ];
        allowedUDPPorts = [
          "8000:10.0.0.3:8000"
        ];
      };
    };
  };