[asterisk-users] Upgrading to Asterisk 13 - Strange Music On Hold Issue - Banging my head on this one

[Mitch Sharp]

I sent this last night but it never showed up in the thread list so
I'm trying again. Please pardon me if it duplicates.

So I've been banging my head against the rack on this one and am now
turning to the group for help.

I'm in the process of bringing five Asterisk servers (all originally
built from source code by myself) from various versions
(1.6.2.x,11.6-cert13, and 13.1-cert2) up to the latest Asterisk
13.8-cert3. This has happened across multiple operating systems and
compiler versions: an old CentOS 5.9 server (GCC 4.1.2) that is
running 11.6-cert13 just fine and wanting to do an in place upgrade to
13.8-cert3, an Ubuntu 14.04.5 LTS (GCC 4.8.4) that was built from
scratch to replace an old server, and multiple Ubuntu 16.04.1 LTS (GCC
4.8, 4.9 and 5.4.0) built from scratch to replace old servers. The
servers themselves range from old to new, fast and faster, Dell and
Supermicro, production and lab. With one exception, I have had the
same Music On Hold problem on each one of the servers.

The problem is this: when a caller is placed on hold or parked,
initially the music on hold 100% of the time is garbled. It sounds
like what it did back in Asterisk 1.2 when zaptel/dahdi was screwed up
and timing was off. Within 5-20 seconds it will clear up. Once its
cleared up, you can take that caller off hold and put them back on
hold, or un-park and re-park them as many times as you want and the
Music On Hold will not glitch again.

I've tried timing with res_timing_dahdi and res_timing_timerfd. I've
tried quietmp3 and files (wav, gsm, ulaw, sln, mp3). I've tried
different codec: ULAW and G729. There is absolutely no load on any of
these machines when I'm testing... just the one phone call that I'm
testing on. I've tested with various SIP endpoints (Polycom
SoundPointIP 550, Polycom VVX 500 and 600, Zoiper, Linphone). It
doesn't matter if its two SIP endpoints or a SIP endpoint to PSTN (via
SIP of course), the behavior is always the same. Absolutely nothing
I've done has made a dent in the problem.

The only thing that is consistent is I don't have the problem on
11.6-cert13 or -cert15 (regardless of OS or processor). I have not
tried Asterisk 12 to see if the issue is present. I have tried
compiling 13.8-cert3 with GCC 4.8 and 4.9 and no change there either.
I've also tested Asterisk 14.1.1 and the behavior is the same This
behavior exists already on the in production 13.1-cert2 server, but
doesn't impact anything because nothing ever uses Music On Hold on
that box.

The two exceptions are a new Supermicro server and a Dell server that
are not having the problem at all running Asterisk 13.8-cert3. The
Supermicro is running Ubuntu 16.04.1 LTS (GCC 5.4.0) and the Dell is
running Ubuntu 14.04.5 LTS (GCC 4.8.4). Using res_timing_timerfd and
no dahdi kernel modules loaded, Music On Hold is pristine. The only
thing I can see is that both of these machines have Xeon E5
processors.

For anyone interested, my build instructions/script is posted at the link below.

Below are the processors running in the five servers. This is the only
difference I can really see in the machines. If it's a CPU issue, if I
knew what specific feature was causing it work I'd just replace the
three problem children with servers that knowingly fixed the problem.

Working fine on these processors:
Intel Xeon CPU E5-2650 @ 2.00Ghz (DELL PowerEdge R620, Currently
Ubuntu 14.04.5 LTS)
Intel Xeon CPU E5-2637 v4 @ 3.50Ghz (Supermicro, Currently Ubuntu 16.04.1 LTS)

Not working on these processors:
Intel Atom CPU 330 @ 1.60Ghz (Supermicro, Currently CentOS 5.9,
retiring but not until I figure this out!)
Intel Xeon CPU X3470 @ 2.93Ghz (Supermicro Currently Ubuntu 16.04.1 LTS)
Intel Xeon CPU E3-1230 v2 @ 3.30Ghz (DELL PowerEdge R310, Currently
Ubuntu 16.04.1 LTS)

I feel like I'm spinning my wheels on this one. Any help would be
GREATLY appreciated!!!

-- Mitch Sharp (bluecrow76)
http://files.bluecrow.net/asterisk/13/