Proxmox VM migration without losing the public IP: BGP, FRR and NetBox, with an open-source network-reconciler

Proxmox HA will move a failed VM to a healthy node in under a minute. It will not move the public IP. The router still sends traffic to the old node, the NAT rule still lives on the old node, and from the customer's side the service is down until someone fixes the routing by hand. On a three-node lab we measured what happens when the routing is fixed automatically instead: a live migration of a VM on local LVM storage cost 3 lost pings out of 45. A VM on LINSTOR shared storage lost none.

This article explains the setup: FRR speaking BGP between the nodes and a router, NetBox as the source of truth for public IPs, and a small daemon we wrote, the network-reconciler, that applies NAT and route changes on the node where the VM is running. Both the daemon and the event bus it depends on are released as open source.

The problem: the VM moves, the network does not

Virtual machines in a cluster often run services or API endpoints that must be reachable on public IPs. On a small cluster you configure this once per VM, on the node the VM lives on, and forget about it. As the cluster grows, two things break this model.

The first is scale: dozens of VMs, each with a NAT rule and a static route on a specific node, becomes a spreadsheet nobody trusts. The second is VM migration. When a node fails, is shut down for maintenance, or a VM is simply moved to balance load, the VM is no longer where the router expects it. Depending on the network design, restoring reachability means reconfiguring (and sometimes restarting) the router, deleting the NAT entry on the old node, creating it on the new one, and possibly pulling the correct configuration from an external source first. Each of these steps is a place for a typo at 3 a.m.

We wanted the public IP to follow the VM with no manual step and with downtime measured in single seconds. That is what the network-reconciler does.

What the network-reconciler does

The network-reconciler is a daemon that runs on every Proxmox node. It listens to three data sources (below), and whenever a VM starts, stops, or migrates, it does two things on the node where the VM now runs:

• Writes an nftables NAT rule so that traffic for the public IP is forwarded to the VM's internal IP (DNAT), and the VM's outgoing traffic is sourced from the public IP (SNAT).
• Adds the public IP as a /32 address on the node's loopback interface. FRR redistributes connected routes into BGP, so the router learns within seconds which node currently owns that IP.

On the node the VM left, the same rules are removed. No configuration on the router changes: it only receives a new BGP route.

Three data sources

To make sure the latest network data is always promptly and correctly applied, the reconciler checks three sources.

1. proxmox-eventbus: cluster events in real time

proxmox-eventbus is an internally developed daemon that runs on each Proxmox node, watches the local task log via inotify, enriches every event from /etc/pve, and publishes it as a CloudEvents message on an embedded NATS bus. The NATS instances form a full mesh across the nodes, so there is no separate broker, load balancer, or DNS to operate, and mTLS reuses the certificates Proxmox already has in /etc/pve. Subjects follow the pattern pve.cluster.node.kind.vmid.action.phase, so a subscriber can listen to exactly "migration finished for VM 999" and nothing else.

The network-reconciler subscribes to VM start, stop, and migration events. If a VM migrates away from the node, the reconciler removes its NAT rule and loopback address; if a VM migrates in, it fetches the VM's network configuration (unless already pre-fetched) and applies it. proxmox-eventbus must be installed on every node you want to monitor; it is shipped as a .deb package for amd64 and arm64 and starts as a systemd unit.

2. NetBox: source of truth for public IPs

NetBox is where we document which public IP belongs to which VM. It has a well-documented API and configurable webhooks, so the reconciler is notified every time an IP address, a virtual machine, or an interface changes. All you need is an event rule for those object types and a webhook attached to that rule. If enabled in the reconciler config, both are created automatically.

3. Shared config files: the fallback

NetBox can be unreachable because of a node outage or a failed update, and sometimes an admin wants a quick change without opening NetBox. For both cases the reconciler persists every VM configuration it has received in a separate file named by VM ID. If a VM config is not available from NetBox, it looks for that file and applies it. An admin can also create the file by hand for a test and override it in NetBox later.

Because VMs move between nodes, these files live in /etc/pve/network-reconciler. File synchronisation is then handled by Proxmox itself, and every online node in the cluster sees the same files.

Network setup

Once proxmox-eventbus and network-reconciler are installed on each node, and NetBox is available (in one of the VMs is fine, though not required), what remains is one node with FRR acting as the router. The router and the nodes must share one BGP autonomous system number; in this example it is 65050. All nodes you want to include are listed as neighbours. The prefix lists make sure the router only accepts private-range /32s from the nodes and announces nothing back.

frr defaults traditional
hostname edge-router
service integrated-vtysh-config
!
ip prefix-list ACCEPT-PRIVATE seq 10 permit 172.16.0.0/12 le 32
!
ip prefix-list REJECT-ALL seq 10 deny 0.0.0.0/0 le 32
!
router bgp 65050
 bgp router-id 10.101.50.10
 no bgp default ipv4-unicast
 !
 neighbor PVE-PEERS peer-group
 neighbor PVE-PEERS remote-as 65050
 neighbor PVE-PEERS timers 10 30
 !
 neighbor 10.101.50.200 peer-group PVE-PEERS
 neighbor 10.101.50.201 peer-group PVE-PEERS
 neighbor 10.101.50.202 peer-group PVE-PEERS
 !
 address-family ipv4 unicast
  neighbor PVE-PEERS activate
  neighbor PVE-PEERS prefix-list ACCEPT-PRIVATE in
  neighbor PVE-PEERS prefix-list REJECT-ALL out
 exit-address-family
!
line vty

The reconciler deliberately does not touch the FRR configuration: it is network-dependent, and we prefer to keep routing changes in the admin's hands. After installing the network-reconciler service, the admin adjusts the node's FRR configuration according to the provided example file so that it matches the autonomous system configured on the router. The example below only needs the hostname and IP changed per node. The key line is "redistribute connected": every /32 the reconciler puts on the loopback becomes a BGP route. BFD is enabled so the router notices a dead node faster than the BGP hold timer alone would allow.

frr version 10.4.1
frr defaults datacenter
hostname pve01
log syslog informational
service integrated-vtysh-config
!
router bgp 65050
 bgp router-id 10.101.50.200
 no bgp default ipv4-unicast
 coalesce-time 1000
 neighbor BGP peer-group
 neighbor BGP remote-as 65050
 neighbor BGP bfd
 neighbor 10.101.50.10 peer-group BGP
 !
 address-family ipv4 unicast
  neighbor BGP activate
  neighbor BGP soft-reconfiguration inbound
  redistribute connected
 exit-address-family
exit
!
line vty

Exposing a VM on a public IP

Our lab is a three-node cluster (pve01, pve02, pve03) with two test VMs. VM 998 is on local LVM storage and gets its network config from a shared config file. VM 999 is on a LINSTOR shared storage pool and gets its config from NetBox. The router is a separate FRR instance in AS 65050.

When a NetBox configuration for a VM is created and the webhook reaches the reconciler, two changes happen on the node.

First, the node needs to know that traffic addressed to the external IP is actually meant for an internal IP, so the node's NAT table is updated. For a node with IP 10.101.50.200 running a VM with internal IP 10.101.50.251, exposed as 172.16.1.1, the nftables ruleset looks like this. Both prerouting and postrouting are needed for a working connection in both directions.

table inet network-reconciler {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                ip daddr 172.16.1.1 dnat ip to 10.101.50.251
        }
 
        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr 10.101.50.251 snat ip to 172.16.1.1
        }
}

Second, the location of the external IP must reach the router. The reconciler assigns 172.16.1.1/32 to the node's loopback interface; FRR redistributes it into BGP, and the router now knows which node to send traffic to. No static route, no router restart.

What happens during a migration

When a migration is triggered by a node failure or by an admin, proxmox-eventbus notifies the reconciler instances on both the source and the destination node. The destination fetches the network data for that VM (from NetBox or the config file) and prepares it. When the "migration sync completed" message arrives, the destination applies the prepared NAT rule and loopback address, and the source removes its copies. To see how this affects reachability, we pinged the public IP continuously during the migration.

VM 998: local LVM storage

The first example is VM 998 on normal LVM storage, migrated purely by Proxmox (the disk is copied during the migration).

table inet network-reconciler {
        chain prerouting {
                type nat hook prerouting priority dstnat; policy accept;
                ip daddr 172.16.1.1 dnat ip to 10.101.50.251
        }
 
        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr 10.101.50.251 snat ip to 172.16.1.1
        }
}

Three packets (sequence 41 to 43) were lost and latency spiked to about 1 ms right after. Three lost pings at a one-second interval means roughly three seconds in which the service was unreachable. The cause is not the routing. Without replicated storage the disk has to be moved for the migration to work; Proxmox does this with qm move-disk, which runs a QEMU drive-mirror job in the background and polls QEMU for its status at one-second intervals. During the cutover I/O is paused, and it usually takes two polls, sometimes three, before QEMU reports success. That is the two to three seconds you see in the ping log.

VM 999: LINSTOR shared storage

The second example is VM 999 on a shared storage pool managed by LINSTOR, which keeps the volume replicated on all pooled nodes. The migration does not need to move the disk at all: LINSTOR promotes the new node to primary and the VM resumes immediately. Every packet was delivered and the delay is not visible in the ping log.

64 bytes from 172.16.1.2: icmp_seq=37 ttl=64 time=0.645 ms
64 bytes from 172.16.1.2: icmp_seq=38 ttl=64 time=0.485 ms
64 bytes from 172.16.1.2: icmp_seq=39 ttl=64 time=0.661 ms
64 bytes from 172.16.1.2: icmp_seq=40 ttl=64 time=1.14 ms
64 bytes from 172.16.1.2: icmp_seq=44 ttl=64 time=0.816 ms
64 bytes from 172.16.1.2: icmp_seq=45 ttl=64 time=0.484 ms
^C
--- 172.16.1.2 ping statistics ---
45 packets transmitted, 42 received, 6.66667% packet loss, time 45074ms

The takeaway: with shared storage such as LINSTOR, DRBD or Ceph, the network side of a migration is not the bottleneck. The routing update is faster than the migration itself. If you run Ceph, our Linux defaults for Ceph performance article covers the storage-side tuning.

When you do not need this

Be honest about scale before adding another daemon to your nodes.

• Small cluster, few public IPs, migrations are rare: a static route per node plus a documented NAT rule is fine, and one less moving part.
• You already run Proxmox SDN with an EVPN zone (core SDN packages ship by default since Proxmox VE 8.1). SDN solves VM-to-VM overlay networking across nodes and can export EVPN routes to an external BGP peer via exit nodes. If that already covers your public-IP requirements, stay with it. The network-reconciler targets a different case: plain bridged VMs with NAT on the node, public IPs documented in NetBox, and no overlay network.
• Your public IPs are floating IPs from a cloud or hosting provider with its own API. Use that API; do not add BGP to the mix.
• You have no NetBox and do not plan to run one. A NetBox URL and token are mandatory in the reconciler config; the config-file fallback covers NetBox being down, not NetBox being absent.

Known limitations

• The reconciler runs as root (it needs to write to /etc/pve for the shared config files), but by design it never edits the FRR configuration. That remains a manual, one-time step per node.
• If the reconciler on the destination node is not running, the migration succeeds but the public IP is not moved. Monitor the systemd unit like any other cluster service.
• The ping tests above are from a three-node lab. The migration delay on local storage depends on disk size and network throughput; the three-second figure is for a 20 GB disk over the lab network.
• Tested on Proxmox VE 9.2.3 with FRR 10.4.1 on the nodes and FRR 8.4.4 on the router, and NetBox 4.6.4 using v1 API tokens; v2 token support is planned.
• IPv4 only. IPv6 is not supported at the moment.

Get the code

Both tools are published on Hostzero's GitHub.

• proxmox-eventbus: Go, Apache-2.0, .deb packages for amd64 and arm64. The post-install hook creates the service user, adds it to www-data so it can read /etc/pve, and starts the systemd unit.
• network-reconciler: Go, Apache-2.0. Runs as a systemd unit; configuration in /etc/network-reconciler/config.yaml, NetBox token in /etc/network-reconciler/environment. Deploy to all nodes with make deploy after setting the HOSTS variable in the Makefile, or build a package with make package and install it per node, then systemctl enable --now network-reconciler. Requires a Proxmox cluster (tested with PVE 9), a router node running FRR, proxmox-eventbus on every node, and a NetBox v4 instance.

Issues and pull requests are welcome. If you run this on a cluster larger than ours, we want to hear the numbers.

What we do at Hostzero

We run and support Proxmox clusters for EU companies from our data centre in Frankfurt, including migrations from VMware. The network-reconciler came out of that work: the customer setups where VMs move between nodes daily and nobody should have to touch a router afterwards. If you are planning a cluster, or are migrating from VMware to Proxmox, see our Proxmox VE migration and support service. For the shared-storage side, our Ceph planning and operations team handles the storage that makes the zero-loss migration above possible.

FAQ

Does Proxmox HA move the public IP of a VM automatically?
No. HA restarts or migrates the VM on another node, but any NAT rule or static route pointing at the old node stays where it was. Reachability on the public IP has to be restored separately, which is what the network-reconciler automates.

Why BGP and not a shared virtual IP with keepalived or VRRP?
VRRP works for one IP shared by a fixed pair of hosts. Here any VM can land on any node, and each VM has its own public IP. Announcing each IP as a /32 route from whichever node owns it scales to any number of IPs and nodes without pre-defining pairs.

How much downtime does a migration cause with this setup?
In our lab, three lost pings (about three seconds) for a VM on local LVM storage, and no lost packets for a VM on LINSTOR shared storage. The routing update itself is faster than the migration.

Do I have to run NetBox?
Yes, a NetBox v4 URL and token are required in the configuration. If NetBox is unreachable, the reconciler falls back to per-VM config files in /etc/pve/network-reconciler, which Proxmox synchronises across the cluster, so a NetBox outage does not block migrations.

Is this the same as Proxmox SDN?
No. Proxmox SDN builds overlay networks (VLAN, VXLAN, EVPN) between VMs across nodes. The network-reconciler handles a narrower job: keeping a public IP with NAT on the node reachable when a bridged VM migrates, driven by NetBox.

Is the code production-ready?
proxmox-eventbus is at v0.1.0 and we run it on our own clusters. Treat both tools as early releases: read the code, test on a lab cluster first, and open an issue when something breaks.

Talk to an engineer

Planning a Proxmox cluster, or dealing with one where VMs and IPs have drifted apart? Talk to an engineer: 30 minutes, no sales pitch.