A good friend of mine is to thank for this excellent pair of one liners. One is to be executed on source, and the other on destination target.

Testing forward route

# Machine Source
root@iup2-web01:/mnt/www# dd if=/dev/zero bs=1024K count=1024 | nc -v 10.181.164.100 23
Connection to 10.181.164.100 23 port [tcp/telnet] succeeded!
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 11.0009 s, 97.6 MB/s
---

# Machine Destination
root@iup2-nfs:~# nc -v -l 23 > /dev/null
Listening on [0.0.0.0] (family 0, port 23)
Connection from [10.181.162.15] port 23 [tcp/telnet] accepted (family 2, sport 54373)

However when we try this in reverse, we see a major degradation in the network speed. Here is test with 20MB transfer instead of 1000MB:

Testing Reverse Route

---

# Machine Source
root@iup2-nfs:~# dd if=/dev/zero bs=1024K count=20 | nc -v 10.181.162.15 23
Connection to 10.181.162.15 23 port [tcp/telnet] succeeded!
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 144.327 s, 145 kB/s
---
# Machine Destination
root@iup2-web01:/mnt/www#  nc -v -l 23 > /dev/null 
Listening on [0.0.0.0] (family 0, port 23)
Connection from [10.181.164.100] port 23 [tcp/telnet] accepted (family 2, sport 56072)

As we can see one of the machines has some difficulty. The issue at hand was that there was some problems with the virtual switch daemon on the hypervisor. Thanks to my friend Gospodin for documenting this one and sharing with me how he tested it,