Lossy Inbound Links

Posted on 4 mins read

Here’s a short companion walkthrough to my Performance @Scale talk on making a lossy inbound link, in the style of the Magic Modem. For a full OS router OS built around this with global ISP traffic shaping details and a simple UI, check out manaOS

VM Setup and OS Install

Download debian-9.0.0-amd64-netinst.iso (or similar, should work with 9.4.0), and VirtualBox if you need them.

VirtualBox configuration

New Virtual Machine:
  Expert Mode
  Name: lldemo
  Type: Linux
  Version: Debian (64-bit)
  Memory Size: 512MB
  Hard disk: Create a virtual hard disk now
    Hard disk file type: VDI
    Storage on physical hard disk: Dynamically allocated
    File location and size: 8GB
  Create

File menu, "Host Network Manager..."
  Create
    This example uses 192.168.56.1/24 for the Host, adjust as needed
  Under "Adapter", "Configure Adapter Manually", IPv4 Address: 192.168.56.1 with mask 255.255.255.0
  Choose the "DHCP Server" tab
    Ensure "Enable Server" is checked
    Server Address: 192.168.56.2, Mask 255.255.255.0
    Lower Address Bound: 192.168.53.3
    Upper Address Bound: 192.168.56.254
  Select "Apply"

Right click lldemo, choose "Settings..."
  Choose the "Storage" tab
    Click the empty CD/DVD
    Click the CD/DVD icon next to the "Optical Drive: IDE Secondary Master" menu
    "Choose Virtual Optical Disk File", select the debian image
  Choose the "Network" tab
    Configure Adapter 2, Enable, Host-only Adapter, Name: vboxnet0

OS Installation

Start the image, Install, perform a normal debian install, enp0s3 should correspond to VirtualBox Adapter 1, while enp0s8 corresponds to VirtualBox Adapter 2 (Host-only adapter on vboxnet0)

  • Primary interface should be enp0s3
  • Creating the sudoable user “wiz” with password “wiz” for this example
  • For software selection, select only “SSH server”

The machine should reboot and leave you at the login prompt. Login and configure the second network interface:

  sudo vim.tiny /etc/network/interfaces
  # add these two lines for enp0s8, similar to enp0s3:
    allow-hotplug enp0s8
    iface enp0s8 inet dhcp
  sudo reboot   # or just restart networking

After reboot, check the IP address that the VM retrieved, it should be 192.168.56.3:

  ip addr show dev enp0s8

From the host machine, ping and ssh should now work:

  ping 192.168.56.3
  # for the rest of this example, keep this ssh open:
  ssh -v -o "DynamicForward 1080" wiz@192.168.56.3

# Setup forwarding and masquerading:
sudo sysctl -w net.ipv4.conf.all.forwarding=1
sudo iptables -t nat -A POSTROUTING -o enp0s3 -jMASQUERADE

# Add the IFB interface for inbound (internet download) shaping
sudo modprobe -v ifb numifbs=1
sudo ip link set up dev ifb0

# Send inbound (internet download) traffic to the ifb:
sudo tc qdisc add dev enp0s3 ingress
sudo tc filter add dev enp0s3 parent ffff: \
    protocol ip u32 match u32 0 0 flowid 1:1 \
    action mirred egress redirect dev ifb0

# Setup negligible network emulation, 2 special traffic classes:
sudo tc qdisc add dev ifb0 root handle 1: htb
sudo tc class add dev ifb0 parent 1: classid 1:2 htb rate 100Mbit
sudo tc qdisc add dev ifb0 parent 1:2 handle 12: netem delay 0ms loss 0%
sudo tc class add dev ifb0 parent 1: classid 1:3 htb rate 100Mbit
sudo tc qdisc add dev ifb0 parent 1:3 handle 13: netem delay 0ms loss 0%
# can always reset to scratch with:
#   tc qdisc del dev ifb0 root
# and reinstall the above network classes

# Assign all traffic to class 1:3:
sudo tc filter add dev ifb0 protocol ip u32 match ip src 0.0.0.0/0 classid 1:3

# Ensure there are no running instances of Chrome, then start one that uses the proxy:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
    --incognito --proxy-server="socks5://127.0.0.1:1080" \
    --host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE 127.0.0.1"

# Set all traffic in class 1:3 to moderately horrible:
tc qdisc replace dev ifb0 parent 1:3 handle 13: netem delay 2394ms loss 11% rate 4Mbit

# Reset class 1:3 to negligible delay:
tc qdisc replace dev ifb0 parent 1:3 handle 13: netem delay 0ms

# Assign routes on AWS EC2 in us-east-1 to class 1:2:
sudo apt-get install -y jq curl
curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | \
  jq '.prefixes[] | select(.region == "us-east-1" and .service == "EC2") | .ip_prefix' | \
  tr -d '"' | \
  xargs -I "%" printf "tc filter add dev ifb0 protocol ip u32 match ip src % classid 1:2\n" | \
  sudo bash

# Set most traffic to slow, but traffic to AWS EC2 in us-east-1 to fast:
sudo tc qdisc replace dev ifb0 parent 1:3 handle 13: netem delay 1000ms loss 5% rate 1Mbit
sudo tc qdisc replace dev ifb0 parent 1:2 handle 12: netem delay 5ms loss 1% rate 10Mbit

# Configure your networks to taste, adjust the parameters to those collected
#   by your RUM metrics with per-packet modeling