[Asterisk-Users] OT: HOWTO: Create a 90mbit bonded link 600 metre s away with Cat 3 or telco wire [long]

[BJ Weschke]

On 4/6/06, Colin Anderson <ColinA at landmarkmasterbuilder.com> wrote:
> I was given the challenge recently of creating a LAN-LAN bridge between two
> buildings several
> hundred metres from each other, using only existing Cat 3 wiring and without
> having to resort
> to an expensive and finicky 5 Ghz wireless link. I was able to create a 90
> megabit link for
> about $3,000 Cdn with new PC's, CentOS 4.1, and the newly avaliable Black
> Box VDSL Ethernet
> Extender, which supports 30 megabits over a single twisted pair.
>
> This is relevant to the list because I have seen many posts with people
> facing the same
> kind of challenge deploying Asterisk in remote locations. In my case, I am
> running
> ~40 Snom 360's from the remote building to where my Asterisk server is, and
> it's working
> fine.
>
> Hope this helps someone. Yay Linux.
>
> ----------------------------------------------------------------------------
> ---------------
> HOWTO: Create a bonded Ethernet link over common Cat 3 or telco-grade cable
> up to 1.9 km
> ----------------------------------------------------------------------------
> ---------------
>
> Abstract: Using the newly-available Black Box LB300A VDSL Ethernet Extenders
> and Linux bonding,
> it is possible to create a protocol-independent, redundant, high speed
> Ethernet bridge using common
> Cat 3 or telco-grade cabling at ranges from 600 metres (1,970 feet) to 1.9
> km (6,233 feet),
> with bandwidth ranging from 90 megabits to 3 megabits. The characteristic of
> the link is that of
> a regular bridged LAN, and is suitable for high speed or latency sensitive
> applications such as Voice
> over IP.
>
> HARDWARE:
>
> 2 X 4-PCI slot PC's
> 6 X Black Box LB300A VDSL Ethernet Extenders
> 8 X PCI Ethernet cards
> http://www.blackbox.com/Catalog/Detail.aspx?cid=425,1423,1424&mid=4946
> RJ-12 crimper
> RJ-12 6-conductor plugs (will work with RJ-11)
>
> SOFTWARE:
>
> CentOS 4.1 or any Linux distro that supports bonding and bridging
>
>
> 1. Install hardware on both machines
>
> In my case, I used an ASUS A8V with an Athlon XP 3000+ with 4) 3C905CX
> cards. The reason I selected
> this board was because there is 5 avaliable PCI slots. Install the cards 0-3
> in slots 0-3.
>
> 2. Disable onboard hardware
>
> Because the NIC's will require an IRQ each, disable the following resources:
>
>        a) Onboard NIC
>        b) Onboard sound
>        c) Onboard USB
>        d) Onboard legacy ports
>        e) Set "F1 on error" or "halt on errors" to off in the BIOS so you
> can boot headless
>
> 3. Install your OS
>
> I chose CentOS 4.1 as my distro, I really like it. I installed CentOS as a
> "server" with everything
> disabled except "development". I also set the detected NIC's to
> "unconfigured" - no DHCP,
> no start at boot.
>
> 4. Boot. Do an ifconfig eth0, eth1, eth2, eth3 to make sure all of the NIC's
> are running OK
>
> 5. Determine which NIC is eth0, eth1, eth2, eth3. eth0 will be the interface
> to the LAN and
> eth1,2,3 will be the bonded link. Both the bonded link and the LAN interface
> will be bridged together.
>
> Easy enough - plug in a network cable into a NIC and run dhclient. Once you
> get an IP from your
> DHCP server you can use ifconfig to determine whihc one got the IP. Repeat
> for each NIC. Once you
> have detemined which physical device is which, I used a sharpie to label
> each port so I wouldn't
> get confused.
>
> 6. Install bridging support, if your distro does not support it OOB.
> Fortunately, CentOS does so
> no problem there.
>
> 7. Install bridge-utils to configure the bridge. Cake in CentOS: "yum
> install bridge-utils"
>
> 8. Configure the bonding config
>
> In /etc/sysconfig/network-scripts make valid entries for your bonded device
> and each slave.
>
> ifcfg-bond0:
>
> DEVICE=bond0
> ONBOOT=no
> BOOTPROTO=none
>
> ifcfg-eth1/2/3:
>
> DEVICE=eth1/2/3 (replace as nessisary)
> USERCTL=no
> ONBOOT=yes
> MASTER=bond0
> SLAVE=yes
> BOOTPROTO=none
>
> 9. Startup script /etc/rc.d/rc.local:
>
> modprobe bonding miimon=1*
> brctl addbr br0
> brctl addif br0 eth0
> sleep 10s**
> brctl addif br0 bond0
>
> *miimon=1 means disable the port after one second on link down. This is the
> redundant part. If
> one of your links fail, the other two will keep working.
>
> ** The 'sleep' is to allow the bridge to stabilize. In testing, the bridge
> did not work if each
> brctl statement immediately followed each other.
>
> 10. Test without VDSL extenders
>
> Obtain or make 3 X Ethernet crossover cables. Plug 1 PC's eth0 port into the
> source LAN. Plug the
> 3 X cross over cables into eth1,2,3 on both PC's. Plug a device into the 2nd
> PC's eth0 with a
> crossover cable, or plug the second PC into a switch then plug your devices
> into the same switch.
> Make sure both PC's have a keyboard and monitor so you can see what's going
> on. Finally, plug
> crossover cables into each PC's eth1, eth2 & eth3
>
> On both PC's run tcpdump -i br0 from the console . In my case, I saw traffic
> right away. Do some
> test pings from and to your devices, and try to get an IP address (dhcp)
> from the "remote" LAN.
>
> Troubleshooting:
>
> *Link up at 100base-T FD?
> *ifconfig eth1, eth2, eth3 shows as SLAVE?
> *same ifconfig shows TX and RX values > 0
> *try mii-tool -F 100baseT-FD
>
> 11. Rack up and plug in the PC's in the source and destination LAN's.
> Terminate the Cat 3 with the
> RJ-11 or 12 plugs on both LAN's. The pinout is undocumented in the extremely
> brief manual that comes
> with the LB-300 so straight-through on all pins was used. Undoubtedly, it
> follows the telco standard of
> the middle pins, but this is untested. Plug the freshly-terminated plugs
> into the RJ12 port on
> the LB-300. Once you are done, use 1 metre patch cables to plug the RJ45
> ports on the LB-300 into
> eth1, eth2, and eth3. Make sure the switch on the LB-300 is set to "Loc"
> (LOCAL) on the source LAN's
> LB-3090's and "Rmt" (REMOTE) on the destination LAN's LB-300.
>
> 12. Power up and watch for the blinkenlights
>
> Troubleshooting:
>
> *Same as troubleshooting above?
> *Look at the lights on the top of the LB-300's. They should give you an
> indication as to what's
> happening to the LB-300 according to the troubleshooting chart on the LB-300
> manual.
>
> 13. Enjoy your link!
>


 Along the same lines, we've had good success at one client where the
local ILEC leased dry pair at this property management client and we
were able to apply HDSL adapters from http://www.rad-direct.com/ to
these lines and get about 3Mb/s in a distance of over 2 miles on some
of the links. We've got IP501's hung off these connections and the
really nice thing (that we really hadn't even planned on until we saw
it happening) was that these adapters passed 802.1q tagged packets
right through so we can use the QoS on the managed switches as if it
were just part of the rest of the LAN.

--
Bird's The Word Technologies, Inc.
http://www.btwtech.com/