[asterisk-users] Strange network issue

[Dave Platt]

> They've got a bunch of Grandstreams that seem to be rock solid... until 
> 7:00pm.  At 7:00, some of the phones become unavailable, and stay down.  Call 
> quality is solid almost all the time.  But right at 7:00, things go bad.  Only 
> some of the phone lines go down and they stay down until the phone is 
> rebooted.
> 
> I'm not even sure what to look for when I go to the site.  Any ideas?

I'd look to see if there are any electrical circuits (lights,
fans, etc.) which are on a timer of some sort, and are automatically
powered off at 7 PM.

If somebody mistakenly plugged a piece of network kit into such a
circuit, it would lose power at that time, and your network might
end up being partitioned, or routing (switch or IP-level) might
change abruptly.

If (for example) the phones were being DHCP-provisioned with
network numbers and a "here is your default gateway" configuration,
and that gateway were to lose power, the phones would lose connectivity,
and might not recover until they discarded their DHCP credentials
and routing information, and broadcast for a new configuration...
which would happen if they were power-cycled, or (if not then)
many hours later.

Similar things could happen if (for example) a janitor were to
plug a floor polisher into a power circuit shared with servers
or network equipment... the turn-on / turn-off power sags and
spikes might knock networking gear off-line.  [This is not
a hypothetical example... numerous cases of this sort of thing
have been reported over the years.]

If these phones are being DHCP-provisioned, you might want to
check each phone and see what configuration has been acquired...
i.e. if it got its information from the "real" DHCP server,
or from some other source.  I've had network problems in the past
result from people plugging some sort of "all-in-one" appliance
or server into an existing net... the appliance starts trying to
provide DHCP service and routing on its own.  This can seriously
disrupt the network... either immediately (if the appliance's
configuration is incompatible with the network) or at a later time
(if e.g. the appliance is acting as a router, and is then powered
off).