Duplicating UDP Streams to Dark Deploy Services

When developing new code swapping out services, it is often useful to do a dark deploy of the new code and activate it at a later time. If you plan ahead, activating and deactivating a new compontent can be done easier than completely replacing a component. Especially if the replacement goes bad, then you have a lot of hoops to jump through to roll back, rather than just reconfigure a few things.

One challenge is if you have a service that listens on a port and are replacing it with a different service that needs to listen on the same port. One use case I ran across for this recently was in testing out a replacement for statsd.

Statsd usually sits at a pretty critical pathway in production, as it is responsible for sending metrics to a system that lets people view graphs and create monitoring rules. Swapping out a monitoring system is the type of project that has a long tail and will not be done in one shot. What we need is a way to duplicate the metrics going to statsd, so they get sent to statsd and to the new system.

This involves a couple of steps:

  1. Reconfigure statsd to listen on a different port
  2. Know what port your new system listens on
  3. Create a listener on the regular statsd port that will forward traffic to both statsd and the new service.

For #3, I tried looking through iptables configuration - I figured something may exist to handle that but the only thing I saw (IPTables Tee) could only send copies to a separate IP address and not a separate port. If there's an IPTables way to do this, I would be really interested.

What I came up with was this small piece of code that listens on a port and duplicates the traffic to a number of ports, depending on the command line arguments.

./udp-duplicator 0.0.0.0:8125 127.0.0.1:8126 127.0.0.1:8127

The above will listen on port 8125 and send traffic to both port 8126 and 8127 on the local host. If you wanted to send to 3 or 4 ports, or different systems you could do that with this code. This should be run inside whatever service manager you are using (upstart, systemd, etc).

Below is the actual service code. It is fairly simple. There is not any error handling as it just dies and relies on the system service manager to start it back up.

package main

import "net"
import "os"

func main() {
        targetConns := make([]net.Conn, 0)

        for target := range os.Args[2:] {
                conn, _ := net.Dial("udp", os.Args[target])

                targetConns = append(targetConns, conn)
        }

        pc, err := net.ListenPacket("udp", os.Args[1])
        if err != nil {
                panic(err)
        }
        defer pc.Close()

        buffer := make([]byte, 65536)

        for {
                n, _, err := pc.ReadFrom(buffer)
                if err != nil {
                        panic(err)
                }

                for _, target := range os.Args[2:] {
                        conn, _ := net.Dial("udp", target)

                        conn.Write(buffer[:n])
                        conn.Close()
                }

        }
}

Comments

Pages

Tags