Fixing SIP 504 Server Timeout Errors in High-Availability SBC Clusters

Problem

Inbound SIP INVITE requests time out at the HA SBC cluster boundary, generating a '504 Server Timeout' back to upstream carriers during active-standby switchover or when downstream media gateways fail to respond within default transaction timers.

Symptoms

• Upstream carriers receive 'SIP/2.0 504 Server Timeout' during HA state transitions.
• Call setup failures spike immediately following a VRRP or Pacemaker VIP migration.
• SIP transaction timers (Timer B / Timer F) expire waiting for '100 Trying' or '180 Ringing' from internal application servers.
• Packet captures show responses from internal nodes being sent to the standby SBC node instead of the new active master.

Root cause

Asymmetric network routing post-failover caused by stale upstream ARP caches (missing Gratuitous ARP), coupled with unsynchronized SIP Transaction Module (TM) state between cluster nodes and overly aggressive fr_timer transaction timeouts.

Resolution steps

1. step #1

   Inspect active call traffic and log files on the active SBC to identify which transaction timer is triggering the 504 timeout.

   grep -E "504 Server Timeout|fr_timer|fr_inv_timer" /var/log/kamailio.log

2. step #2

   Verify Virtual IP (VIP) assignment and force immediate Gratuitous ARP (GARP) broadcasts in Keepalived configuration to prevent stale router ARP tables upon failover.

   vrrp_instance VI_1 {
       state MASTER
       interface eth0
       virtual_router_id 51
       priority 101
       advert_int 1
       garp_master_delay 1
       garp_master_repeat 5
       garp_master_refresh 10
       virtual_ipaddress {
           192.168.10.50/24 dev eth0
       }
   }

3. step #3

   Enable Distributed Message Queue (DMQ) or cluster replication modules to synchronize SIP transaction state across SBC nodes in real time.

   # Kamailio DMQ and TM Module Sync Setup
   loadmodule "dmq.so"
   loadmodule "dmq_usrloc.so"
   loadmodule "tm.so"
   
   modparam("dmq", "server_address", "sip:192.168.10.11:5090")
   modparam("dmq", "notification_address", "sip:192.168.10.12:5090")
   modparam("tm", "fr_timer", 3000)
   modparam("tm", "fr_inv_timer", 30000)

4. step #4

   Configure conntrackd to replicate Netfilter state table entries across cluster members, ensuring stateful firewalls do not drop asymmetric return traffic.

   cat << 'EOF' > /etc/conntrackd/conntrackd.conf
   Sync {
       Mode FTFW {
           ResendQueueSize 10240
       }
       UDP {
           IPv4_address 192.168.10.11
           IPv4_Destination_Address 192.168.10.12
           Port 3780
           Interface eth1
       }
   }
   EOF
   systemctl restart conntrackd

5. step #5

   Adjust the default upstream carrier failover route logic to quickly redistribute calls to a secondary internal media server if the primary fails to respond within 2 seconds, preventing 504 responses.

   failure_route[INTERNAL_FAILURE] {
       if (t_check_status("504|408")) {
           if (t_next_hop_carrier()) {
               t_on_failure("INTERNAL_FAILURE");
               t_relay();
               exit;
           }
       }
   }

6. step #6

   Trigger a controlled HA failover while running a live SIP trace to confirm active call state retention and zero 504 errors.

   sngrep -d eth0 -w /tmp/ha_failover.pcap "sip.Response.status == 504"
   systemctl stop keepalived

Verification evidence

sngrep or tcpdump output demonstrates INVITE forwarded southbound, but reply packets arrive at the standby node's MAC address or netfilter drops them due to missing conntrack states, triggering 'TM: timer routine: timeout' in the active SBC log before sending a 504 back to the originator.