[Asterisk-Dev] Occasional yet persistant Seg fault, memory corruption? Memory overwritten

Tue Jun 10 13:09:28 MST 2003

I have a persistant, occasional segfault that appears to be caused by memory corruption but it
doesn't seem like SEU or single bit errors ( typical of hardware ).

I have crashes approximately once per 2 days often in small clusters then it works perfectly for a
while.

Each crash produces a backtrace full of invalid pointers and strange values.

The crashes are always in different places but have some things in common.
1.  It's quite often in int_free()
2.  Usually accompanied by junk pointers.
3.  ast_spawn_extension has been called with the same bogus value ( each time it differs ) for all
parameters

Here is today's GDB dig into this crash in case this jogs somebody's memory:
----------------------------------------------------------------------------
(gdb) bt
#0  0x42074c21 in _int_free () from /lib/i686/libc.so.6
#1  0x42075a5c in free () from /lib/i686/libc.so.6
#2  0x0805305d in ast_verbose (fmt=0x42138c80 "") at logger.c:317
#3  0x41d4d988 in dial_exec (chan=0x479f1878, data=0x8dbb140) at app_dial.c:550
#4  0x08062089 in pbx_exec (c=0x479f1878, app=0x80cf8a0, data=0x481ff72c,
     newstack=1) at pbx.c:393
#5  0x08064079 in pbx_extension_helper (c=0x479f1878,
     context=0x479f199c "castlefax", exten=0x479f1a3c "s", priority=1,
     callerid=0x47987660 "4122208473", action=0) at pbx.c:1124
#6  0x08064fc5 in ast_spawn_extension (c=0x2e726463,
     context=0x2e726463 <Address 0x2e726463 out of bounds>,
     exten=0x2e726463 <Address 0x2e726463 out of bounds>, priority=779248739,
     callerid=0x2e726463 <Address 0x2e726463 out of bounds>) at pbx.c:1608
#7  0x080651a2 in ast_pbx_run (c=0x4213820c) at pbx.c:1669
#8  0x08065ab1 in pbx_thread (data=0x2e726463) at pbx.c:1821
#9  0x4002b881 in pthread_start_thread () from /lib/i686/libpthread.so.0
(gdb) info frame 2
Stack frame at 0x481fefb4:
  eip = 0x805305d in ast_verbose (logger.c:317); saved eip 0x41d4d988
  called by frame at 0x481ff504, caller of frame at 0x481fef84
  source language c.
  Arglist at 0x481fefb4, args: fmt=0x42138c80 ""
  Locals at 0x481fefb4, Previous frame's sp is 0x0
  Saved registers:
   ebp at 0x481fefb4, esi at 0x481fefac, edi at 0x481fefb0, eip at 0x481fefb8
(gdb) frame 2
#2  0x0805305d in ast_verbose (fmt=0x42138c80 "") at logger.c:317
317                             free(m->msg);
(gdb) info locals
stuff = "    -- Called 48\n\0all from '4122208473' to '1287' on channel 3, span
4\n\0\0n macro 'stdext'\n\0\n\0urgh;\n\0cago Evanston,\n\0 -> 31000\n\01999-2001
  Linux Support Services, Inc.\n", '\0' <repeats 3929 times>
pos = 17
opos = 0
replacelast = 0
complete = 1
m = (struct msglist *) 0x4213820c
v = (struct verb *) 0x4213820c
----------------------------------------------------------------------------

Here's another crash that just happened Note the crap pointers in ast_spawn_extension that get
passed down to lower functions:

(gdb) bt
#0  __ast_free_region (ptr=0x8554c60, file=0x80a50c3 "config.c", lineno=94,
     func=0x80a50cc "ast_destroy") at astmm.c:132
#1  0x080566b4 in ast_destroy (ast=0x8258a88) at config.c:94
#2  0x41d57980 in leave_voicemail (chan=0x846f940, ext=0x464fe99d "2638",
     silent=0, busy=1, unavail=0) at app_voicemail.c:838
#3  0x41d5e0d1 in vm_exec (chan=0x0, data=0x464fecdc) at app_voicemail.c:2275
#4  0x08062089 in pbx_exec (c=0x846f940, app=0x80dc650, data=0x464fecdc,
     newstack=1) at pbx.c:393
#5  0x08064079 in pbx_extension_helper (c=0x846f940,
     context=0x846fa64 "macro-stdext", exten=0x846fb04 "s", priority=102,
     callerid=0x0, action=0) at pbx.c:1124
#6  0x08064fc5 in ast_spawn_extension (c=0xa1,
     context=0xa1 <Address 0xa1 out of bounds>,
     exten=0xa1 <Address 0xa1 out of bounds>, priority=161,
     callerid=0xa1 <Address 0xa1 out of bounds>) at pbx.c:1608
#7  0x41d943aa in macro_exec (chan=0x846f940, data=0x464ff72c)
     at app_macro.c:138
#8  0x08062089 in pbx_exec (c=0x846f940, app=0x81e4d90, data=0x464ff72c,
     newstack=1) at pbx.c:393
#9  0x08064079 in pbx_extension_helper (c=0x846f940,
     context=0x846fa64 "macro-stdext", exten=0x846fb04 "s", priority=1,
     callerid=0x0, action=1) at pbx.c:1124
#10 0x08064fc5 in ast_spawn_extension (c=0xa1,
---Type <return> to continue, or q <return> to quit---
     context=0xa1 <Address 0xa1 out of bounds>,
     exten=0xa1 <Address 0xa1 out of bounds>, priority=161,
     callerid=0xa1 <Address 0xa1 out of bounds>) at pbx.c:1608
#11 0x080651a2 in ast_pbx_run (c=0x41d61268) at pbx.c:1669
#12 0x08065ab1 in pbx_thread (data=0xa1) at pbx.c:1821
#13 0x4002b881 in pthread_start_thread () from /lib/i686/libpthread.so.0
(gdb) frame 0
#0  __ast_free_region (ptr=0x8554c60, file=0x80a50c3 "config.c", lineno=94,
     func=0x80a50cc "ast_destroy") at astmm.c:132
132                     reg = reg->next;
(gdb) info locals
hash = 302
reg = (struct ast_region *) 0x1
prev = (struct ast_region *) 0x1
(gdb) frame 1
#1  0x080566b4 in ast_destroy (ast=0x8258a88) at config.c:94
94                              free(v->name);
(gdb) info locals
cat = (struct ast_category *) 0x8553b08
catn = (struct ast_category *) 0xa1
v = (struct ast_variable *) 0x8554b58
vn = (struct ast_variable *) 0x1
(gdb)

--------------------------------------------------------------------------------
I added fencepost checking code to ast_mm and it finds several instances of overwritten memory while
running *.  These pointers into the code are pointers to the free call that ast_mm found problems
during.  It is not where the overwriting happens.

Note the overwrite found in logger.c where the first crash occurred ( why this one isn't using
ast_mm I don't know )

1054045028 - WARNING: Lower fencepost overwritten (found in free) at 0x8b6e200,
in ast_channel_free of channel.c, line 540
1054045989 - WARNING: Lower fencepost overwritten (found in free) at 0x8e433b0,
in ast_cdr_free of cdr.c, line 112
1054140518 - WARNING: Lower fencepost overwritten (found in free) at 0x459db4f0,
  in ast_dsp_free of dsp.c, line 1267
1054142790 - WARNING: Lower fencepost overwritten (found in free) at 0x459d77b0,
  in ast_channel_free of channel.c, line 570
1054210146 - New session
1054224843 - WARNING: Upper fencepost overwritten (found in free) at 0x9b02d50,
in ast_var_delete of chanvars.c, line 68
1054321393 - WARNING: Lower fencepost overwritten (found in free) at 0x478f1fa8,
  in ast_verbose of logger.c, line 317
1054569431 - WARNING: Lower fencepost overwritten (found in free) at 0x8be3308,
in ast_cdr_free of cdr.c, line 112
1054569600 - WARNING: Lower fencepost overwritten (found in free) at 0x8ce3e30,
in ast_var_delete of chanvars.c, line 65
1054752774 - New session
1054755011 - WARNING: Lower fencepost overwritten (found in free) at 0x861c718,
in ast_verbose of logger.c, line 317