[Asterisk-Dev] Occasional yet persistant Seg fault, memory
corruption? Memory overwritten
Mark Spencer
markster at digium.com
Fri Jun 13 00:16:53 MST 2003
Does valgrind give any more useful information ( don't run valgrind and
asterisk's memory debugging at the same time)
Mark
On Tue, 10 Jun 2003 asterisk at billheckel.com wrote:
> I have a persistant, occasional segfault that appears to be caused by memory corruption but it
> doesn't seem like SEU or single bit errors ( typical of hardware ).
>
> I have crashes approximately once per 2 days often in small clusters then it works perfectly for a
> while.
>
> Each crash produces a backtrace full of invalid pointers and strange values.
>
> The crashes are always in different places but have some things in common.
> 1. It's quite often in int_free()
> 2. Usually accompanied by junk pointers.
> 3. ast_spawn_extension has been called with the same bogus value ( each time it differs ) for all
> parameters
>
> Here is today's GDB dig into this crash in case this jogs somebody's memory:
> ----------------------------------------------------------------------------
> (gdb) bt
> #0 0x42074c21 in _int_free () from /lib/i686/libc.so.6
> #1 0x42075a5c in free () from /lib/i686/libc.so.6
> #2 0x0805305d in ast_verbose (fmt=0x42138c80 "") at logger.c:317
> #3 0x41d4d988 in dial_exec (chan=0x479f1878, data=0x8dbb140) at app_dial.c:550
> #4 0x08062089 in pbx_exec (c=0x479f1878, app=0x80cf8a0, data=0x481ff72c,
> newstack=1) at pbx.c:393
> #5 0x08064079 in pbx_extension_helper (c=0x479f1878,
> context=0x479f199c "castlefax", exten=0x479f1a3c "s", priority=1,
> callerid=0x47987660 "4122208473", action=0) at pbx.c:1124
> #6 0x08064fc5 in ast_spawn_extension (c=0x2e726463,
> context=0x2e726463 <Address 0x2e726463 out of bounds>,
> exten=0x2e726463 <Address 0x2e726463 out of bounds>, priority=779248739,
> callerid=0x2e726463 <Address 0x2e726463 out of bounds>) at pbx.c:1608
> #7 0x080651a2 in ast_pbx_run (c=0x4213820c) at pbx.c:1669
> #8 0x08065ab1 in pbx_thread (data=0x2e726463) at pbx.c:1821
> #9 0x4002b881 in pthread_start_thread () from /lib/i686/libpthread.so.0
> (gdb) info frame 2
> Stack frame at 0x481fefb4:
> eip = 0x805305d in ast_verbose (logger.c:317); saved eip 0x41d4d988
> called by frame at 0x481ff504, caller of frame at 0x481fef84
> source language c.
> Arglist at 0x481fefb4, args: fmt=0x42138c80 ""
> Locals at 0x481fefb4, Previous frame's sp is 0x0
> Saved registers:
> ebp at 0x481fefb4, esi at 0x481fefac, edi at 0x481fefb0, eip at 0x481fefb8
> (gdb) frame 2
> #2 0x0805305d in ast_verbose (fmt=0x42138c80 "") at logger.c:317
> 317 free(m->msg);
> (gdb) info locals
> stuff = " -- Called 48\n\0all from '4122208473' to '1287' on channel 3, span
> 4\n\0\0n macro 'stdext'\n\0\n\0urgh;\n\0cago Evanston,\n\0 -> 31000\n\01999-2001
> Linux Support Services, Inc.\n", '\0' <repeats 3929 times>
> pos = 17
> opos = 0
> replacelast = 0
> complete = 1
> m = (struct msglist *) 0x4213820c
> v = (struct verb *) 0x4213820c
> ----------------------------------------------------------------------------
>
> Here's another crash that just happened Note the crap pointers in ast_spawn_extension that get
> passed down to lower functions:
>
> (gdb) bt
> #0 __ast_free_region (ptr=0x8554c60, file=0x80a50c3 "config.c", lineno=94,
> func=0x80a50cc "ast_destroy") at astmm.c:132
> #1 0x080566b4 in ast_destroy (ast=0x8258a88) at config.c:94
> #2 0x41d57980 in leave_voicemail (chan=0x846f940, ext=0x464fe99d "2638",
> silent=0, busy=1, unavail=0) at app_voicemail.c:838
> #3 0x41d5e0d1 in vm_exec (chan=0x0, data=0x464fecdc) at app_voicemail.c:2275
> #4 0x08062089 in pbx_exec (c=0x846f940, app=0x80dc650, data=0x464fecdc,
> newstack=1) at pbx.c:393
> #5 0x08064079 in pbx_extension_helper (c=0x846f940,
> context=0x846fa64 "macro-stdext", exten=0x846fb04 "s", priority=102,
> callerid=0x0, action=0) at pbx.c:1124
> #6 0x08064fc5 in ast_spawn_extension (c=0xa1,
> context=0xa1 <Address 0xa1 out of bounds>,
> exten=0xa1 <Address 0xa1 out of bounds>, priority=161,
> callerid=0xa1 <Address 0xa1 out of bounds>) at pbx.c:1608
> #7 0x41d943aa in macro_exec (chan=0x846f940, data=0x464ff72c)
> at app_macro.c:138
> #8 0x08062089 in pbx_exec (c=0x846f940, app=0x81e4d90, data=0x464ff72c,
> newstack=1) at pbx.c:393
> #9 0x08064079 in pbx_extension_helper (c=0x846f940,
> context=0x846fa64 "macro-stdext", exten=0x846fb04 "s", priority=1,
> callerid=0x0, action=1) at pbx.c:1124
> #10 0x08064fc5 in ast_spawn_extension (c=0xa1,
> ---Type <return> to continue, or q <return> to quit---
> context=0xa1 <Address 0xa1 out of bounds>,
> exten=0xa1 <Address 0xa1 out of bounds>, priority=161,
> callerid=0xa1 <Address 0xa1 out of bounds>) at pbx.c:1608
> #11 0x080651a2 in ast_pbx_run (c=0x41d61268) at pbx.c:1669
> #12 0x08065ab1 in pbx_thread (data=0xa1) at pbx.c:1821
> #13 0x4002b881 in pthread_start_thread () from /lib/i686/libpthread.so.0
> (gdb) frame 0
> #0 __ast_free_region (ptr=0x8554c60, file=0x80a50c3 "config.c", lineno=94,
> func=0x80a50cc "ast_destroy") at astmm.c:132
> 132 reg = reg->next;
> (gdb) info locals
> hash = 302
> reg = (struct ast_region *) 0x1
> prev = (struct ast_region *) 0x1
> (gdb) frame 1
> #1 0x080566b4 in ast_destroy (ast=0x8258a88) at config.c:94
> 94 free(v->name);
> (gdb) info locals
> cat = (struct ast_category *) 0x8553b08
> catn = (struct ast_category *) 0xa1
> v = (struct ast_variable *) 0x8554b58
> vn = (struct ast_variable *) 0x1
> (gdb)
>
>
>
>
>
> --------------------------------------------------------------------------------
> I added fencepost checking code to ast_mm and it finds several instances of overwritten memory while
> running *. These pointers into the code are pointers to the free call that ast_mm found problems
> during. It is not where the overwriting happens.
>
> Note the overwrite found in logger.c where the first crash occurred ( why this one isn't using
> ast_mm I don't know )
>
> 1054045028 - WARNING: Lower fencepost overwritten (found in free) at 0x8b6e200,
> in ast_channel_free of channel.c, line 540
> 1054045989 - WARNING: Lower fencepost overwritten (found in free) at 0x8e433b0,
> in ast_cdr_free of cdr.c, line 112
> 1054140518 - WARNING: Lower fencepost overwritten (found in free) at 0x459db4f0,
> in ast_dsp_free of dsp.c, line 1267
> 1054142790 - WARNING: Lower fencepost overwritten (found in free) at 0x459d77b0,
> in ast_channel_free of channel.c, line 570
> 1054210146 - New session
> 1054224843 - WARNING: Upper fencepost overwritten (found in free) at 0x9b02d50,
> in ast_var_delete of chanvars.c, line 68
> 1054321393 - WARNING: Lower fencepost overwritten (found in free) at 0x478f1fa8,
> in ast_verbose of logger.c, line 317
> 1054569431 - WARNING: Lower fencepost overwritten (found in free) at 0x8be3308,
> in ast_cdr_free of cdr.c, line 112
> 1054569600 - WARNING: Lower fencepost overwritten (found in free) at 0x8ce3e30,
> in ast_var_delete of chanvars.c, line 65
> 1054752774 - New session
> 1054755011 - WARNING: Lower fencepost overwritten (found in free) at 0x861c718,
> in ast_verbose of logger.c, line 317
>
>
>
> _______________________________________________
> Asterisk-Dev mailing list
> Asterisk-Dev at lists.digium.com
> http://lists.digium.com/mailman/listinfo/asterisk-dev
>
More information about the asterisk-dev
mailing list