[asterisk-bugs] [JIRA] (ASTERISK-26186) res_pjsip_multihomed: Deadlock when finding message header

Mateusz Kowalski (JIRA) noreply at issues.asterisk.org
Mon Jul 11 09:41:57 CDT 2016


     [ https://issues.asterisk.org/jira/browse/ASTERISK-26186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mateusz Kowalski updated ASTERISK-26186:
----------------------------------------

    Description: 
Hello,

I encountered a deadlock in PJSIP module. Two backtraces available as
* http://pastebin.com/raw/kZtc5df3 (Thread 43)
* http://pastebin.com/raw/93H9qe4x (Thread 42)

Output from {{core show locks}} is empty:
{code}
tone-0058-re-1*CLI> core show locks

=======================================================================
=== 13.9.1
=== Currently Held Locks
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=======================================================================
{code}

Any packets arriving to the machine on port 5060 are left without any response. They are not visible in Asterisk console even with {{pjsip set logger on}}

Output from {{netstat}} shows big RX queue for the process:
{code}
udp   214912      0 0.0.0.0:5060            0.0.0.0:*                           46787/asterisk
{code}

If you check {{netstat}} per port, it happens also on RTP port (I'm using 10000-20000):
{code}
udp        0      0 0.0.0.0:2727            0.0.0.0:*                           46787/asterisk
udp      896      0 0.0.0.0:35859           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:37171           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:4520            0.0.0.0:*                           46787/asterisk
udp   214912      0 0.0.0.0:5060            0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:10520           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:10521           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:11356           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:11357           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:12048           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:12049           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:12062           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:12063           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:13890           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:13891           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:14608           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:14609           0.0.0.0:*                           46787/asterisk
udp   213248      0 0.0.0.0:14678           0.0.0.0:*                           46787/asterisk
udp    34048      0 0.0.0.0:14679           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:14752           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:14753           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:15106           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:15107           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18100           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18101           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18392           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18393           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18918           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18919           0.0.0.0:*                           46787/asterisk
{code}

Initially I reported issue on #asterisk but later on got redirected to open a ticket in here.

Snaps of my configuration in here:

{{asterisk.conf}}
{code}
[directories](!)
astetcdir => /etc/asterisk
astmoddir => /usr/lib/asterisk/modules
astvarlibdir => /var/lib/asterisk
astdbdir => /var/lib/asterisk
astkeydir => /var/lib/asterisk
astdatadir => /var/lib/asterisk
astagidir => /var/lib/asterisk/agi-bin
astspooldir => /var/spool/asterisk
astrundir => /var/run/asterisk
astlogdir => /var/log/asterisk
astsbindir => /usr/sbin

[options]
;verbose = 5
;debug = 5
;alwaysfork = yes		; Same as -F at startup.
;nofork = yes			; Same as -f at startup.
;quiet = yes			; Same as -q at startup.
;timestamp = yes		; Same as -T at startup.
;execincludes = yes		; Support #exec in config files.
;console = yes			; Run as console (same as -c at startup).
;highpriority = yes		; Run realtime priority (same as -p at
				; startup).
;initcrypto = yes		; Initialize crypto keys (same as -i at
				; startup).
;nocolor = yes			; Disable console colors.
;dontwarn = yes			; Disable some warnings.
;dumpcore = yes			; Dump core on crash (same as -g at startup).
;languageprefix = yes		; Use the new sound prefix path syntax.
;systemname = my_system_name	; Prefix uniqueid with a system name for
				; Global uniqueness issues.
;autosystemname = yes		; Automatically set systemname to hostname,
				; uses 'localhost' on failure, or systemname if
				; set.
;mindtmfduration = 80		; Set minimum DTMF duration in ms (default 80 ms)
				; If we get shorter DTMF messages, these will be
				; changed to the minimum duration
;maxcalls = 10			; Maximum amount of calls allowed.
;maxload = 0.9			; Asterisk stops accepting new calls if the
				; load average exceed this limit.
;maxfiles = 1000		; Maximum amount of openfiles.
;minmemfree = 1			; In MBs, Asterisk stops accepting new calls if
				; the amount of free memory falls below this
				; watermark.
;cache_record_files = yes	; Cache recorded sound files to another
				; directory during recording.
;record_cache_dir = /tmp	; Specify cache directory (used in conjunction
				; with cache_record_files).
;transmit_silence = yes		; Transmit silence while a channel is in a
				; waiting state, a recording only state, or
				; when DTMF is being generated.  Note that the
				; silence internally is generated in raw signed
				; linear format. This means that it must be
				; transcoded into the native format of the
				; channel before it can be sent to the device.
				; It is for this reason that this is optional,
				; as it may result in requiring a temporary
				; codec translation path for a channel that may
				; not otherwise require one.
;transcode_via_sln = yes	; Build transcode paths via SLINEAR, instead of
				; directly.
;runuser = asterisk		; The user to run as.
;rungroup = asterisk		; The group to run as.
;lightbackground = yes		; If your terminal is set for a light-colored
				; background.
;forceblackbackground = yes     ; Force the background of the terminal to be
                                ; black, in order for terminal colors to show
                                ; up properly.
;defaultlanguage = en           ; Default language
documentation_language = en_US	; Set the language you want documentation
				; displayed in. Value is in the same format as
				; locale names.
;hideconnect = yes		; Hide messages displayed when a remote console
				; connects and disconnects.
;lockconfdir = no		; Protect the directory containing the
				; configuration files (/etc/asterisk) with a
				; lock.
;stdexten = gosub		; How to invoke the extensions.conf stdexten.
				; macro - Invoke the stdexten using a macro as
				;         done by legacy Asterisk versions.
				; gosub - Invoke the stdexten using a gosub as
				;         documented in extensions.conf.sample.
				; Default gosub.
;live_dangerously = no		; Enable the execution of 'dangerous' dialplan
				; functions from external sources (AMI,
				; etc.) These functions (such as SHELL) are
				; considered dangerous because they can allow
				; privilege escalation.
				; Default no

; Changing the following lines may compromise your security.
;[files]
;astctlpermissions = 0660
;astctlowner = root
;astctlgroup = apache
;astctl = asterisk.ctl
{code}

{{extensions.conf}}
{code}
[professional]
switch => Realtime/professional at extensions

[private]
switch => Realtime/private at extensions

[sigos]
switch => Realtime/sigos at extensions
{code}

{{hep.conf}}
{code}
[general]
enabled = yes
capture_address = 188.184.65.170:9060
;capture_password = foo
capture_id = 1234
{code}

{{modules.conf}}
{code}
[modules]
autoload=yes

preload => res_odbc.so
preload => res_config_odbc.so
{code}

{{pjsip.conf}}
{code}
[transport-udp]
type=transport
protocol=udp
bind=0.0.0.0
{code}

{{sorcery.conf}}
{code}
[res_pjsip]
endpoint=realtime,ps_endpoints
;endpoint=config,pjsip.conf,criteria=type=endpoint
;auth=realtime,ps_auths
aor=realtime,ps_aors
;aor=config,pjsip.conf,criteria=type=aor
;domain_alias=realtime,ps_domain_aliases
;contact=realtime,ps_contacts

[res_pjsip_endpoint_identifier_ip]
identify=realtime,ps_endpoint_id_ips
;identify=config,pjsip.conf,criteria=type=identify
{code}

Thanks for any help,
Mateusz

  was:
Hello,

I encountered a deadlock in PJSIP module. Two backtraces available as
* http://pastebin.com/raw/kZtc5df3 (Thread 43)
* http://pastebin.com/raw/93H9qe4x (Thread 42)

Output from {{core show locks}} is empty:
{code}
tone-0058-re-1*CLI> core show locks

=======================================================================
=== 13.9.1
=== Currently Held Locks
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=======================================================================
{code}

Any packets arriving to the machine on port 5060 are left without any response. They are not visible in Asterisk console even with {{pjsip set logger on}}

Output from {{netstat}} shows big RX queue for the process:
{code}
udp   214912      0 0.0.0.0:5060            0.0.0.0:*                           46787/asterisk
{code}

If you check {{netstat}} per port, it happens also on RTP port (I'm using 10000-20000):
{code}
udp        0      0 0.0.0.0:2727            0.0.0.0:*                           46787/asterisk
udp      896      0 0.0.0.0:35859           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:37171           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:4520            0.0.0.0:*                           46787/asterisk
udp   214912      0 0.0.0.0:5060            0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:10520           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:10521           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:11356           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:11357           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:12048           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:12049           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:12062           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:12063           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:13890           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:13891           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:14608           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:14609           0.0.0.0:*                           46787/asterisk
udp   213248      0 0.0.0.0:14678           0.0.0.0:*                           46787/asterisk
udp    34048      0 0.0.0.0:14679           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:14752           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:14753           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:15106           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:15107           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18100           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18101           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18392           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18393           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18918           0.0.0.0:*                           46787/asterisk
udp        0      0 0.0.0.0:18919           0.0.0.0:*                           46787/asterisk
{code}

Initially I reported issue on #asterisk but later on got redirected to open a ticket in here.

Thanks for any help,
Mateusz


> res_pjsip_multihomed: Deadlock when finding message header
> ----------------------------------------------------------
>
>                 Key: ASTERISK-26186
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-26186
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: pjproject/pjsip
>    Affects Versions: 13.9.1
>         Environment: CentOS 7.2 (3.10.0-327.22.2.el7.x86_64)
> Asterisk 13.9.1
> PJSIP 2.5.1
>            Reporter: Mateusz Kowalski
>            Assignee: Unassigned
>         Attachments: trace1.txt, trace2.txt
>
>
> Hello,
> I encountered a deadlock in PJSIP module. Two backtraces available as
> * http://pastebin.com/raw/kZtc5df3 (Thread 43)
> * http://pastebin.com/raw/93H9qe4x (Thread 42)
> Output from {{core show locks}} is empty:
> {code}
> tone-0058-re-1*CLI> core show locks
> =======================================================================
> === 13.9.1
> === Currently Held Locks
> =======================================================================
> ===
> === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
> ===
> =======================================================================
> {code}
> Any packets arriving to the machine on port 5060 are left without any response. They are not visible in Asterisk console even with {{pjsip set logger on}}
> Output from {{netstat}} shows big RX queue for the process:
> {code}
> udp   214912      0 0.0.0.0:5060            0.0.0.0:*                           46787/asterisk
> {code}
> If you check {{netstat}} per port, it happens also on RTP port (I'm using 10000-20000):
> {code}
> udp        0      0 0.0.0.0:2727            0.0.0.0:*                           46787/asterisk
> udp      896      0 0.0.0.0:35859           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:37171           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:4520            0.0.0.0:*                           46787/asterisk
> udp   214912      0 0.0.0.0:5060            0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:10520           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:10521           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:11356           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:11357           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:12048           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:12049           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:12062           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:12063           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:13890           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:13891           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:14608           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:14609           0.0.0.0:*                           46787/asterisk
> udp   213248      0 0.0.0.0:14678           0.0.0.0:*                           46787/asterisk
> udp    34048      0 0.0.0.0:14679           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:14752           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:14753           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:15106           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:15107           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:18100           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:18101           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:18392           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:18393           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:18918           0.0.0.0:*                           46787/asterisk
> udp        0      0 0.0.0.0:18919           0.0.0.0:*                           46787/asterisk
> {code}
> Initially I reported issue on #asterisk but later on got redirected to open a ticket in here.
> Snaps of my configuration in here:
> {{asterisk.conf}}
> {code}
> [directories](!)
> astetcdir => /etc/asterisk
> astmoddir => /usr/lib/asterisk/modules
> astvarlibdir => /var/lib/asterisk
> astdbdir => /var/lib/asterisk
> astkeydir => /var/lib/asterisk
> astdatadir => /var/lib/asterisk
> astagidir => /var/lib/asterisk/agi-bin
> astspooldir => /var/spool/asterisk
> astrundir => /var/run/asterisk
> astlogdir => /var/log/asterisk
> astsbindir => /usr/sbin
> [options]
> ;verbose = 5
> ;debug = 5
> ;alwaysfork = yes		; Same as -F at startup.
> ;nofork = yes			; Same as -f at startup.
> ;quiet = yes			; Same as -q at startup.
> ;timestamp = yes		; Same as -T at startup.
> ;execincludes = yes		; Support #exec in config files.
> ;console = yes			; Run as console (same as -c at startup).
> ;highpriority = yes		; Run realtime priority (same as -p at
> 				; startup).
> ;initcrypto = yes		; Initialize crypto keys (same as -i at
> 				; startup).
> ;nocolor = yes			; Disable console colors.
> ;dontwarn = yes			; Disable some warnings.
> ;dumpcore = yes			; Dump core on crash (same as -g at startup).
> ;languageprefix = yes		; Use the new sound prefix path syntax.
> ;systemname = my_system_name	; Prefix uniqueid with a system name for
> 				; Global uniqueness issues.
> ;autosystemname = yes		; Automatically set systemname to hostname,
> 				; uses 'localhost' on failure, or systemname if
> 				; set.
> ;mindtmfduration = 80		; Set minimum DTMF duration in ms (default 80 ms)
> 				; If we get shorter DTMF messages, these will be
> 				; changed to the minimum duration
> ;maxcalls = 10			; Maximum amount of calls allowed.
> ;maxload = 0.9			; Asterisk stops accepting new calls if the
> 				; load average exceed this limit.
> ;maxfiles = 1000		; Maximum amount of openfiles.
> ;minmemfree = 1			; In MBs, Asterisk stops accepting new calls if
> 				; the amount of free memory falls below this
> 				; watermark.
> ;cache_record_files = yes	; Cache recorded sound files to another
> 				; directory during recording.
> ;record_cache_dir = /tmp	; Specify cache directory (used in conjunction
> 				; with cache_record_files).
> ;transmit_silence = yes		; Transmit silence while a channel is in a
> 				; waiting state, a recording only state, or
> 				; when DTMF is being generated.  Note that the
> 				; silence internally is generated in raw signed
> 				; linear format. This means that it must be
> 				; transcoded into the native format of the
> 				; channel before it can be sent to the device.
> 				; It is for this reason that this is optional,
> 				; as it may result in requiring a temporary
> 				; codec translation path for a channel that may
> 				; not otherwise require one.
> ;transcode_via_sln = yes	; Build transcode paths via SLINEAR, instead of
> 				; directly.
> ;runuser = asterisk		; The user to run as.
> ;rungroup = asterisk		; The group to run as.
> ;lightbackground = yes		; If your terminal is set for a light-colored
> 				; background.
> ;forceblackbackground = yes     ; Force the background of the terminal to be
>                                 ; black, in order for terminal colors to show
>                                 ; up properly.
> ;defaultlanguage = en           ; Default language
> documentation_language = en_US	; Set the language you want documentation
> 				; displayed in. Value is in the same format as
> 				; locale names.
> ;hideconnect = yes		; Hide messages displayed when a remote console
> 				; connects and disconnects.
> ;lockconfdir = no		; Protect the directory containing the
> 				; configuration files (/etc/asterisk) with a
> 				; lock.
> ;stdexten = gosub		; How to invoke the extensions.conf stdexten.
> 				; macro - Invoke the stdexten using a macro as
> 				;         done by legacy Asterisk versions.
> 				; gosub - Invoke the stdexten using a gosub as
> 				;         documented in extensions.conf.sample.
> 				; Default gosub.
> ;live_dangerously = no		; Enable the execution of 'dangerous' dialplan
> 				; functions from external sources (AMI,
> 				; etc.) These functions (such as SHELL) are
> 				; considered dangerous because they can allow
> 				; privilege escalation.
> 				; Default no
> ; Changing the following lines may compromise your security.
> ;[files]
> ;astctlpermissions = 0660
> ;astctlowner = root
> ;astctlgroup = apache
> ;astctl = asterisk.ctl
> {code}
> {{extensions.conf}}
> {code}
> [professional]
> switch => Realtime/professional at extensions
> [private]
> switch => Realtime/private at extensions
> [sigos]
> switch => Realtime/sigos at extensions
> {code}
> {{hep.conf}}
> {code}
> [general]
> enabled = yes
> capture_address = 188.184.65.170:9060
> ;capture_password = foo
> capture_id = 1234
> {code}
> {{modules.conf}}
> {code}
> [modules]
> autoload=yes
> preload => res_odbc.so
> preload => res_config_odbc.so
> {code}
> {{pjsip.conf}}
> {code}
> [transport-udp]
> type=transport
> protocol=udp
> bind=0.0.0.0
> {code}
> {{sorcery.conf}}
> {code}
> [res_pjsip]
> endpoint=realtime,ps_endpoints
> ;endpoint=config,pjsip.conf,criteria=type=endpoint
> ;auth=realtime,ps_auths
> aor=realtime,ps_aors
> ;aor=config,pjsip.conf,criteria=type=aor
> ;domain_alias=realtime,ps_domain_aliases
> ;contact=realtime,ps_contacts
> [res_pjsip_endpoint_identifier_ip]
> identify=realtime,ps_endpoint_id_ips
> ;identify=config,pjsip.conf,criteria=type=identify
> {code}
> Thanks for any help,
> Mateusz



--
This message was sent by Atlassian JIRA
(v6.2#6252)



More information about the asterisk-bugs mailing list