[asterisk-dev] res_fax_spandsp segfaults during fax detection - FIXED?

[Michal Rybárik]

Hi folks,

I probably found what causes random segfaults during fax detection with 
spandsp... Many thanks for your contributions and brainstorming about 
this, it helped a lot. If you have such issue, please test this possible 
fix and let me know...

At res_fax.c, somewhere around line 3070, you'll find (at least for 
Asterisk 11)
      case AST_FORMAT_ALAW:
      case AST_FORMAT_ULAW:
Comment out these two lines, recompile, reload and test.

These two lines are in framehook code, which is responsible for fax 
detection for T38gateway. These lines are causing, that ALAW and ULAW 
audio samples are not transcoded to slinear before they are passed to 
res_fax_spandsp function spandsp_v21_detect(), and later to libspandsp 
function modem_connect_tones_rx().  But this libspandsp function doesn't 
expect ulaw or alaw (8bit) samples, it will read and process samples as 
they are 16bit slinear.

If we received 160 alaw/ulaw samples (typical 20ms frame) from network, 
each sample is 1 byte long (sorry for my disinformation about this topic 
in previous email).  So res_rtp_asterisk allocated 160bytes in the 
memory for storing this frame (its data part) and gave us a pointer to 
this memory. If we pass this pointer to function 
modem_connect_tones_rx() (its second argument) along with information 
that 160 samples are stored from this position (its third argument), 
this function is going to read 320 bytes of memory (160 * 16bit) and 
analyze, if 1100 Hz fax tone is there. But if we allocated and stored 
160 bytes in memory, and we are now reading 320bytes, we will get at 
least bad data or even a segfault too - it depends on position of these 
data in the memory, which is quite unpredictable, or one may say - 
random.. Exactly as these segfaults. Elementary my dear Watson.... :o))

--
Michal Rybarik