[Asterisk-Users] G729 vs. gsm

[Steve Underwood]

Marie wrote:

>Anyone have the time and webspace to post a quick recording of a
>sample conversation in both codecs? If you want to get even more
>tricky, perhaps samples of music on hold in both as well? Or noisy
>environments?
>  
>
This kind of quickie test is worthless. In doing serious codec 
evaluation we do things called MOS scores. They cost a fortune, as they 
involve lots of people over a long period. Basically they involve asking 
a large number of people to rate quality over a large number of varied 
voice samples. As an example of why a single test is useless, most 
codecs are better at coding male voices than female voices - i.e. they 
favour lower pitched voices. A couple of codecs are the other way 
around. If you ever tried the original US digital AMPS cellular system 
you will have heard this effect very clearly. The absolute meaningful 
comparison is with a few people having widely varying voices.

>Obviously not very scientific and prone to a wide margin of error
>(background noise, etc) -- but I think if there was a comparison sound
>file posted on the Wiki or something it would help people feel better
>informed before jumping on the Digium licenses.
>  
>
Even the MOS approach is not that scientific. Its still a subjective 
assessment, but at least it is averaged over a large number of people. A 
real problem with MOS, is how the layman is supposed to interpret the 
results.

CD audio scores 5 - that is the maximum.

G.711 gets about 4.5. The subjective assessment of most people is that 
is G.711 is quite a lot worse than CD audio. Actually, it is more than 
just subjective. The narrow bandwidth of G.711 looses the difference 
between, say, and "s" and an "f". All unvoiced sounds come out much the 
same over a normal PSTN phone.

G.729 scores about 4. That might not sound too much lower than G.711. 
However, if you have worked with MOS you will know that 0.5 is actually 
rather a big drop. Subjective comments from people in the MOS tests says 
the difference is substantial. If people cannot actually tell the 
difference, this can be traced back to a problem with their hearing.

G.723.1 scores about 3.8. Again, that might not sound too much worse 
than G.729, but it doesn't tell the whole story. Have you ever tried 
identifying speakers over G.723.1? It does a good job of avoiding 
robotic sound, but tends to make everyone sound alike. Issues like that 
can be important, but don't always get highlighted by MOS.

One very bad thing about MOS, and most of the playoffs for standards 
bodies (e.g. picking the GSM codecs from amongst the submitted 
contenders) is these tests rarely introduce background noise, or 
anything other than a single human voice talking. Codecs in common use 
have not, therefore, been selected for any tolerance to what is commonly 
sent down a telephone line (this is a particular problem with cellular 
standards, as these phones are heavily used in noisy places). For VoIP 
users, there is also the problem that few people perform any meaningful 
assessments of how badly packet loss degrades things. This varies quite 
a lot between codecs. G.729 and G.723.1 are rather bad at tolerating 
packet loss.

>Just a quick test for "humm, that doesn't sound too bad, maybe it's
>worth spending $10 for each license and fiddling with everything to
>give it a try".
>
>I can't be the only one that isn't quite sure where to place G729 on
>the scale of GSM - ULaw.
>  
>
It isn't a one way scale. Music on hold over G.729 is often 
unrecognisable. Over GSM 06.10 it is usually just poor. Is that a big 
issue for you, or totally irrelevant? GSM 06.10 and G.729 at 8kbps offer 
fairly similar quality for clean voice. GSM wins when there is 
background noise. G.729 wins on bit rate.

The real world gets so complicated. :-)

Regards,
Steve