[Asterisk-Users] OT: HOWTO: Create a 90mbit bonded link 600 metre s away with Cat 3 or telco wire [long]

[Colin Anderson]

I was given the challenge recently of creating a LAN-LAN bridge between two
buildings several
hundred metres from each other, using only existing Cat 3 wiring and without
having to resort
to an expensive and finicky 5 Ghz wireless link. I was able to create a 90
megabit link for 
about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable Black
Box VDSL Ethernet 
Extender, which supports 30 megabits over a single twisted pair. 

This is relevant to the list because I have seen many posts with people
facing the same 
kind of challenge deploying Asterisk in remote locations. In my case, I am
running 
~40 Snom 360's from the remote building to where my Asterisk server is, and
it's working
fine. 

Hope this helps someone. Yay Linux. 

----------------------------------------------------------------------------
---------------
HOWTO: Create a bonded Ethernet link over common Cat 3 or telco-grade cable
up to 1.9 km 
----------------------------------------------------------------------------
---------------

Abstract: Using the newly-available Black Box LB300A VDSL Ethernet Extenders
and Linux bonding,
it is possible to create a protocol-independent, redundant, high speed
Ethernet bridge using common
Cat 3 or telco-grade cabling at ranges from 600 metres (1,970 feet) to 1.9
km (6,233 feet), 
with bandwidth ranging from 90 megabits to 3 megabits. The characteristic of
the link is that of
a regular bridged LAN, and is suitable for high speed or latency sensitive
applications such as Voice
over IP.

HARDWARE:

2 X 4-PCI slot PC's
6 X Black Box LB300A VDSL Ethernet Extenders 
8 X PCI Ethernet cards
http://www.blackbox.com/Catalog/Detail.aspx?cid=425,1423,1424&mid=4946
RJ-12 crimper
RJ-12 6-conductor plugs (will work with RJ-11)

SOFTWARE:

CentOS 4.1 or any Linux distro that supports bonding and bridging


1. Install hardware on both machines

In my case, I used an ASUS A8V with an Athlon XP 3000+ with 4) 3C905CX
cards. The reason I selected 
this board was because there is 5 avaliable PCI slots. Install the cards 0-3
in slots 0-3. 

2. Disable onboard hardware

Because the NIC's will require an IRQ each, disable the following resources:

	a) Onboard NIC
	b) Onboard sound
	c) Onboard USB
	d) Onboard legacy ports 
	e) Set "F1 on error" or "halt on errors" to off in the BIOS so you
can boot headless
	
3. Install your OS

I chose CentOS 4.1 as my distro, I really like it. I installed CentOS as a
"server" with everything 
disabled except "development". I also set the detected NIC's to
"unconfigured" - no DHCP, 
no start at boot.

4. Boot. Do an ifconfig eth0, eth1, eth2, eth3 to make sure all of the NIC's
are running OK

5. Determine which NIC is eth0, eth1, eth2, eth3. eth0 will be the interface
to the LAN and 
eth1,2,3 will be the bonded link. Both the bonded link and the LAN interface
will be bridged together.

Easy enough - plug in a network cable into a NIC and run dhclient. Once you
get an IP from your 
DHCP server you can use ifconfig to determine whihc one got the IP. Repeat
for each NIC. Once you 
have detemined which physical device is which, I used a sharpie to label
each port so I wouldn't 
get confused.

6. Install bridging support, if your distro does not support it OOB.
Fortunately, CentOS does so 
no problem there.

7. Install bridge-utils to configure the bridge. Cake in CentOS: "yum
install bridge-utils"

8. Configure the bonding config

In /etc/sysconfig/network-scripts make valid entries for your bonded device
and each slave.

ifcfg-bond0:

DEVICE=bond0
ONBOOT=no
BOOTPROTO=none

ifcfg-eth1/2/3:

DEVICE=eth1/2/3 (replace as nessisary)
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

9. Startup script /etc/rc.d/rc.local:

modprobe bonding miimon=1*
brctl addbr br0
brctl addif br0 eth0
sleep 10s**
brctl addif br0 bond0

*miimon=1 means disable the port after one second on link down. This is the
redundant part. If
one of your links fail, the other two will keep working. 

** The 'sleep' is to allow the bridge to stabilize. In testing, the bridge
did not work if each 
brctl statement immediately followed each other.

10. Test without VDSL extenders

Obtain or make 3 X Ethernet crossover cables. Plug 1 PC's eth0 port into the
source LAN. Plug the 
3 X cross over cables into eth1,2,3 on both PC's. Plug a device into the 2nd
PC's eth0 with a 
crossover cable, or plug the second PC into a switch then plug your devices
into the same switch. 
Make sure both PC's have a keyboard and monitor so you can see what's going
on. Finally, plug
crossover cables into each PC's eth1, eth2 & eth3

On both PC's run tcpdump -i br0 from the console . In my case, I saw traffic
right away. Do some 
test pings from and to your devices, and try to get an IP address (dhcp)
from the "remote" LAN. 

Troubleshooting:

*Link up at 100base-T FD?
*ifconfig eth1, eth2, eth3 shows as SLAVE?
*same ifconfig shows TX and RX values > 0
*try mii-tool -F 100baseT-FD

11. Rack up and plug in the PC's in the source and destination LAN's.
Terminate the Cat 3 with the
RJ-11 or 12 plugs on both LAN's. The pinout is undocumented in the extremely
brief manual that comes 
with the LB-300 so straight-through on all pins was used. Undoubtedly, it
follows the telco standard of
the middle pins, but this is untested. Plug the freshly-terminated plugs
into the RJ12 port on
the LB-300. Once you are done, use 1 metre patch cables to plug the RJ45
ports on the LB-300 into
eth1, eth2, and eth3. Make sure the switch on the LB-300 is set to "Loc"
(LOCAL) on the source LAN's 
LB-3090's and "Rmt" (REMOTE) on the destination LAN's LB-300. 

12. Power up and watch for the blinkenlights

Troubleshooting:

*Same as troubleshooting above?
*Look at the lights on the top of the LB-300's. They should give you an
indication as to what's
happening to the LB-300 according to the troubleshooting chart on the LB-300
manual.

13. Enjoy your link!