https://blog.cabroneria.com/tags/dhcp/ Recent content in Dhcp on > Cabronerias de Matu Hugo en-us Mon, 06 Apr 2026 09:38:00 -0400 https://blog.cabroneria.com/post/008_fun_pfsense_dhcp_outage/ Mon, 06 Apr 2026 09:38:00 -0400 https://blog.cabroneria.com/post/008_fun_pfsense_dhcp_outage/ <h2 id="summary">Summary</h2> <p>On 2026-04-06 at approximately 09:38 EDT, the Kea DHCP service on pfSense (26.03-RELEASE) stopped serving leases on all interfaces. The root cause was a corruption of the <code>&lt;dhcpd&gt;</code> block in <code>/cf/conf/config.xml</code>, which caused pfSense to generate an empty Kea interfaces list. Service was restored by activating a previous ZFS boot environment.</p> <hr> <h2 id="timeline">Timeline</h2> <table> <thead> <tr> <th>Time (EDT)</th> <th>Event</th> </tr> </thead> <tbody> <tr> <td>08:54</td> <td>config.xml drops ~15KB — <code>&lt;dhcpd&gt;</code> block wiped (config-1775480086.xml &gt; config-1775480087.xml)</td> </tr> <tr> <td>08:54–09:37</td> <td>Multiple config saves at reduced size; pfSense regenerates Kea config with empty interfaces each time</td> </tr> <tr> <td>09:38</td> <td>Kea begins logging <code>DHCPSRV_NO_SOCKETS_OPEN</code>; DHCP stops serving leases</td> </tr> <tr> <td>09:38–09:41</td> <td>Kea restarts repeatedly, fails each time</td> </tr> <tr> <td>~09:45</td> <td>Operator detects outage; accesses box via WireGuard VPN</td> </tr> <tr> <td>~10:00</td> <td>Investigation begins</td> </tr> <tr> <td>~10:30</td> <td>Root cause identified: empty <code>&lt;dhcpd&gt;</code> in config.xml</td> </tr> <tr> <td>~10:45</td> <td>Config restore from backup attempted; Kea config generator continues producing empty interfaces</td> </tr> <tr> <td>~11:00</td> <td>ZFS boot environment rollback to <code>default_20260405014509</code> initiated</td> </tr> <tr> <td>~11:10</td> <td>Service restored</td> </tr> </tbody> </table> <hr> <h2 id="root-cause">Root Cause</h2> <p>A pfSense package operation (pfBlockerNG or Suricata reload/apply) triggered a config write at 08:54 that silently cleared the entire <code>&lt;dhcpd&gt;</code> section from <code>/cf/conf/config.xml</code>. This is a pfSense 26.x bug: package-initiated config saves can clobber unrelated service configuration blocks.</p>