Setup a VMWare machine to simulate low speed TCP/IP network

When we design a multi-tiers application in high speed network environment, we do not know if the data communicate performance is up to expectation if running on real network. The cost of develop the application in real network is high. There is a cost effective solution to simulate various kind of bandwidth and traffic conditions for WAN network.

Previously, I learned from DUMMYNET and NistNet has the solution. But it is very hard to deploy the solution for development. I finally found the TC (Traffic control) in iproute package of Linux distribution able to do the task. It is very easy to use.

Combine the TC and routing strategy in Linux network, we may design a masqueraded router that able to let us simulate low speed TCP/IP network in development stage.

We do not need a real machine to configure the router. Use VMWARE workstation to setup a virtual machine running a masqueraded router.

Some Theory

To realize the simulation:

1. Configure 2 networks and make sure both networks may reach each others. 
2. Bind a server application to one of the network
3. Bind a client application to another network
4. Make sure client may reach server and server response to client promptly
5. We may then enforce a traffic control rule on one of the network
6. Establish a connection from client to server and observe how the traffic control rule affect the connection quality.
7. Tune the traffic control rule and continue monitor the connection quality.

Install and Configure VMWare Workstation

1. Install and Start VMWare workstation.  The following illustration use VMware Workstation 6.5.3 as example.
2. In VMWare Workstation window, click Edit | Virtual Network Editor…
3. A “Virtual Network Editor” dialog prompt out, switch to page “Host Virtual Adapters”:
   
    
4. There may be some VMWare network adapter created by default.  In this topic, I need 2 VMware network adapter connect as “Host only”.  For illustration purpose, I add 2 VMware network adapter connected to VMnet1 and VMnet2.
5. Switch to page “Host Virtual Network Mapping” and make sure both VMnet1 and VMnet2 adapters are attached properly:
   
   
6. Click the row of VMNet1 “>” button to configure subnet of VMNet1.  Set subnet IP address to “192.168.101.0”:
   
   
7. Click the row of VMNet2 “>” button to configure subnet of VMNet2.  Set subnet IP address to “192.168.102.0”:
   
   
8. Click “Apply” button to commit changes.
9. You may also need to check both VMNet1 and VMNet2 DHCP setting to see if the setting is correct:
   
    
10. Switch to page “DHCP” and make sure both DHCP service for VMNet1 and VMNet2 are started:
     
11. Switch to “Summary” page to check the status of both VMNet1 and VMNet2:
    
     
12. You may continue to next steps once done

Install a Fedora Core Virtual Machine

1. Create a new virtual machine with the following hardware configuration:
1. Memory: 1GB for installation.  You may change to 128MB after finish installation.
2. Hard Disk: 1GB
3. Network adapter 1 connect as “Custom” VMNet1 (Host-Only)
4. Network adapter 2 connect as “Custom” VMNet2 (Host-Only)
   
   
2. Start the machine and install Fedora Core via any installation methods you prefer. (The following example use Fedora Core 11)
3. The machine is only use for masqueraded network simulation, you may uncheck all software package to install a minimal bare-bone Fedora Core.

Configure Fedora Core Virtual Machine

1. Configure network adapter to start after machine boot. 
   
   $ system-config-network
   
2. In “Select Action” screen, select “Edit a device params” and press Enter to enter “Select A Device” screen.
3. Select “eth0” and press Enter to enter “Network Configuration” screen. Press OK to commit changes.
4. Select “eth1” and press Enter to enter “Network Configuration” screen. Press OK to commit changes and return to “Select A Device” screen.
5. Press “Save” button to commit changes and return to “Select Action” screen.
6. Press “Save&Quit” button to commit changes.
7. Use “vi” editor to set both devices “ONBOOT” to “Yes”:

$ cat /etc/sysconfig/networking/profiles/default/ifcfg-eth0
DEVICE=eth0
HWADDR=00:0c:29:2d:96:b6
ONBOOT=yes
BOOTPROTO=dhcp
TYPE=Ethernet
USERCTL=no
PEERDNS=yes
IPV6INIT=no



$ cat /etc/sysconfig/networking/profiles/default/ifcfg-eth1
DEVICE=eth1
HWADDR=00:0c:29:2d:96:c0
ONBOOT=yes
BOOTPROTO=dhcp
TYPE=Ethernet
USERCTL=no
PEERDNS=yes
IPV6INIT=no



8. Configure Network Service to make it get started after machine boot
   
   $ chkconfig –-list network
   network         0:off   1:off   2:off   3:off   4:off   5:off   6:off
   
   $ chkconfig network on
   
   $ chkconfig –-list network
   network         0:off   1:off   2:on    3:on    4:on    5:on    6:off


9. Restart network service

$ service network restart


10. You may then use ifconfig to check both eth0 and eth1 device is up
    
     
11. Change “net.ipv4.ip_forward” to 1 in file “/etc/sysctl.conf” to enable port forwarding:

$ cat /etc/sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled.  See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

12. Run the following command to disable Firewall to reduce unnecessary troubles in later stage:
    
    $ chkconfig iptables off
    
13. Restart the machine and double to make sure the following are working
1. Both etc0 and et1 is up with assigned IP Address
2. port forwarding is working

Troubleshoot: I encounter “Device eth0 does not seem to be present, delaying initialization.” after I clone the virtual machine

From time to time, you may encounter a situation where both the eth0 or eth1 doesn’t up.  Run the following command explicitly may yields:

The cause of this problem could be the device has been changed or swap.  The new device names for the network adapter could be eth2 or eth3.  A quick solution to this problem is:

$ rm /etc/udev/rules.d/70-persistent-net.rules
$ reboot

After restart the machine, the device names would restored back to eth0 and eth1.

Troubleshoot: I encounter “Device eth0 has different MAC address than expected, ignoring.” after I clone the virtual machine

This is most probably the HWAddr (Mac Address) of both network adapter (/etc/sysconfig/network-scripts/ifcfg-ethN) doesn’t match with VMWare  virtual machine configuration file (.vmx)

To solve the problem, remove a line “HWADDR” in

1. /etc/sysconfig/network-scripts/ifcfg-eth0
2. /etc/sysconfig/network-scripts/ifcfg-eth1

For example,

$ cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=00:0c:29:2d:96:b6
ONBOOT=yes
BOOTPROTO=dhcp
TYPE=Ethernet
USERCTL=no
PEERDNS=yes
IPV6INIT=no

Restart network service start network devices:

$ service network restart

Quick Test Connection

You may now use ping command to test if both the VMNet 1 and VMNet 2 network are working in your own host PC (not Fedora in VMWare):

C:\>ping 192.168.101.128

Pinging 192.168.101.128 with 32 bytes of data:
Reply from 192.168.101.128: bytes=32 time<1ms TTL=64
Reply from 192.168.101.128: bytes=32 time<1ms TTL=64
Reply from 192.168.101.128: bytes=32 time<1ms TTL=64
Reply from 192.168.101.128: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.101.128:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\>ping 192.168.102.128

Pinging 192.168.102.128 with 32 bytes of data:
Reply from 192.168.102.128: bytes=32 time<1ms TTL=64
Reply from 192.168.102.128: bytes=32 time<1ms TTL=64
Reply from 192.168.102.128: bytes=32 time<1ms TTL=64
Reply from 192.168.102.128: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.102.128:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

This test only ensure you may reach both networks from your host but it doesn’t confirm if you may reach VMNet1 from VMNet2 and vice versa.

Advance Test Connection

In your host PC, there should be 2 virtual network adapters (VMNet1 and VMNet2) configured:

C:\>ipconfig

Windows IP Configuration

Ethernet adapter VMware Network Adapter VMnet1:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::f935:c046:af36:32c4%17
   IPv4 Address. . . . . . . . . . . : 192.168.101.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 

Ethernet adapter VMware Network Adapter VMnet2:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::d5a7:81c4:e2c9:de44%19
   IPv4 Address. . . . . . . . . . . : 192.168.102.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :

To make sure both network may reach each other, set the default gateway for both VMNet1 and VMNet2 connection to the Fedora VMware machine’s network adapter.  Please note that the Fedora machine’s network adapters obtained IP addresses via DHCP dynamically.  You may use ifconfig to check the IP addresses:

$ ifconfig eth0

$ ifconfig eth1

Once you get the IP addresses, set it to VMNet1 and VMNet2 adapter of your host PC respectively:

C:\>ipconfig

Windows IP Configuration

Ethernet adapter VMware Network Adapter VMnet1:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::f935:c046:af36:32c4%17
   IPv4 Address. . . . . . . . . . . : 192.168.101.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.101.128

Ethernet adapter VMware Network Adapter VMnet2:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::d5a7:81c4:e2c9:de44%19
   IPv4 Address. . . . . . . . . . . : 192.168.102.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.102.128

You may need to check if your Firewall allow ICMP Echo Request connection.  In Windows Vista or Windows 7, Start “Windows Firewall and with Advanced Security” in privilege account and enable a rule “File and Printer Sharing (Echo Request – ICMPv4-In)”

You may use the following ping command to test if both networks may reach others:

C:\>ping -S 192.168.102.1 192.168.101.1

Pinging 192.168.101.1 from 192.168.102.1 with 32 bytes of data:
Reply from 192.168.101.1: bytes=32 time<1ms TTL=127
Reply from 192.168.101.1: bytes=32 time<1ms TTL=127
Reply from 192.168.101.1: bytes=32 time<1ms TTL=127
Reply from 192.168.101.1: bytes=32 time<1ms TTL=127

Ping statistics for 192.168.101.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\>ping -S 192.168.101.1 192.168.102.1

Pinging 192.168.102.1 from 192.168.101.1 with 32 bytes of data:
Reply from 192.168.102.1: bytes=32 time<1ms TTL=127
Reply from 192.168.102.1: bytes=32 time<1ms TTL=127
Reply from 192.168.102.1: bytes=32 time<1ms TTL=127
Reply from 192.168.102.1: bytes=32 time<1ms TTL=127

Ping statistics for 192.168.102.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

Now it shows that both 192.168.101.0 and 192.168.102.0 networks may reach others.  We may proceed to traffic control.

Setup TC to simulate broadband connection

Example 1: Delay

# add the delay to device
$ tc qdisc add dev eth0 root netem delay 25ms 10ms

# change the delay of device
$ tc qdisc change dev eth0 root netem delay 35ms 10ms

# show the information of device
$ tc qdisc show dev eth0

# delete the device traffic control
$ tc qdisc delete dev eth0 root

Example 2: Rate 

# add rate control to eth1 
$ tc qdisc add dev eth1 root tbf rate 384kbit buffer 1600 limit 3000

# change the delay of device
$ tc qdisc change dev eth1 root tbf rate 5kbit buffer 1600 limit 3000

# show the information of device
$ tc qdisc show dev eth1

# delete the device traffic control
$ tc qdisc delete dev eth1 root

Reference:

1. http://www.linuxfoundation.org/en/Net:Netem