[asterisk-dev] Deadlock between msglist_lock and loglock

[Tony Mountifield]

I have Asterisk 1.2 branch running on a dual-Xeon box, and for two weeks
running (after a recent update) I have had Asterisk lock up at the same
time. After the first time I recompiled with DEBUG_THREADS so that if it
happened again I could see where the mutexes had been claimed.

It turns out that what triggers it is my logrotate job that does
asterisk -rx "logger reload".

I'd be interested to hear if anyone else has noticed the same thing.
It might possible only get triggered on a dual-CPU box, I'm not sure.

My initial investigation shows the following:

ast_verbose() claims msglist_lock and then while holding that mutex
it calls ast_log(). ast_log() then claims loglock.

reload_logger() claims loglock, and then when processing logger.conf
it calls ast_verbose() to say it is parsing it.

In my deadlock situation I have one process trying to log
"remote unix connection", which has claimed msglist_lock and is now
sitting in ast_log waiting for loglock. I have another process
severals stack frames down from reload_logger(), holding locklog and
sitting inside ast_verbose() waiting for msglist_lock.

I haven't worked out a fix as yet, and I'd be grateful for anyone's
input. It might be as simple as having reload_logger() claim msglist_lock
before it claims loglock. Or not :-)

I was hoping to have a fix to suggest before logging it in Mantis, but
I think I'll log it there anyway.

For now I'll have to avoid rotating the logs.

Cheers
Tony
-- 
Tony Mountifield
Work: tony at softins.co.uk - http://www.softins.co.uk
Play: tony at mountifield.org - http://tony.mountifield.org