[asterisk-bugs] [JIRA] (ASTERISK-28830) Incorrect UTF-8 handling when using function FILTER

Wed Apr 22 04:07:25 CDT 2020

    [ https://issues.asterisk.org/jira/browse/ASTERISK-28830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=250500#comment-250500 ] 

Dmitriy commented on ASTERISK-28830:
------------------------------------

I Have similar problem. It is not problem of FILTER function - it is global problem

Example1
Dialplan
same => n,NoOp("Это не конференция, значит включаем на канале запись разговора и фичю для возможности создания конференции налету")

CLI Result 1:
-- Executing [s at AFTER_ANSWER:14] GotoIf("SIP/AVAYA-00003334", "0?conf:record") in new stack
-- Goto (AFTER_ANSWER,s,18)
-- Executing [s at AFTER_ANSWER:18] NoOp("SIP/AVAYA-00003334", "Это не конференция, значит включаем на канале запись разговора и фичю для возможности создания конфе    -- Executing [s at AFTER_ANSWER:19] ExecIf("SIP/AVAYA-00003334", "0?SET(DYNAMIC_FEATURES=conference_callee)") in new stack
-- Executing [s at AFTER_ANSWER:20] Set("SIP/AVAYA-00003334", "recordDir=2020-04-22 13:03:20") in new stack

CLI Result 2:
-- Executing [s at AFTER_ANSWER:14] GotoIf("SIP/4253-00003353", "0?conf:record") in new stack
-- Goto (AFTER_ANSWER,s,18)
-- Executing [s at AFTER_ANSWER:18] NoOp("SIP/4253-00003353", "Это не конференция, значит включаем на канале запись разговора и фичю для возможности создания конференции налету") in new stack
-- Executing [s at AFTER_ANSWER:19] ExecIf("SIP/4253-00003353", "1?SET(DYNAMIC_FEATURES=conference_callee)") in new stack

CLI Result 3:
-- Executing [s at AFTER_ANSWER:14] GotoIf("SIP/AVAYA-00003356", "0?conf:record") in new stack
-- Goto (AFTER_ANSWER,s,18)
-- Executing [s at AFTER_ANSWER:18] NoOp("SIP/AVAYA-00003356", "Это не конференция, значит включаем на канале запись разговора и фичю для возможности создания конфере▒    -- Executing [s at AFTER_ANSWER:19] ExecIf("SIP/AVAYA-00003356", "0?SET(DYNAMIC_FEATURES=conference_callee)") in new stack
-- Executing [s at AFTER_ANSWER:20] Set("SIP/AVAYA-00003356", "recordDir=2020-04-22 13:07:26") in new stack

Example2
Crash CDR recording
[Apr 16 05:08:57] WARNING[1301]: res_odbc.c:538 ast_odbc_print_errors: SQL Execute returned an error: HY000: 
[MySQL][ODBC 8.0(a) Driver][mysqld-5.7.29-0ubuntu0.18.04.1]Incorrect string value: '\xD0' for column 'lastdata' at row 1
[Apr 16 05:08:57] WARNING[1301]: res_odbc.c:433 ast_odbc_prepare_and_execute: SQL Execute error -1!
[Apr 16 05:08:57] WARNING[1301]: cdr_adaptive_odbc.c:765 odbc_log: cdr_adaptive_odbc: Insert failed on 'asteriskcdrdb:cdr'.  
CDR failed: INSERT INTO cdr (calldate, clid, src, dst, dcontext, channel, dstchannel, lastapp, lastdata, duration, billsec, disposition, amaflags, accountcode, uniqueid, recordingfile, cnum, cnam, linkedid, peeraccount, sequence) VALUES ({ ts '2020-04-16 05:06:36' }, '"Оператор 03" <xxxx>', 'xxxx', 'xxxx', 'FROM_PHONE_03', 'SIP/101-00000095', 'SIP/TRUNK_E1_GATE-00000096', 'Dial', 'SIP/TRUNK_E1_GATE/xxxx,,gU(ABONENT_ANSWER,101,Максимовская ▒', 140, 132, 'ANSWERED', 3, '955', '1586995596.156', '2020/04/16/2020-04-16 05:06:36 - 1586995596.157 (101[Максимовская С.С] -> xxxx).mp3', '101', 'Максимовская С.С', '1586995596.156', '955', 179)

Example3
CLI>database show
/agent/0131/name                                  : Демина АВ
/agent/0132/name                                  : Домра▒       > ева АА

I have this bug on other OS and other Asterisk versions
Ubuntu 12.04, 16.04, 18.04
CentOS 8
Asterisk 13.18-cert2
Asterisk 13.21-cert6
Asterisk 16.3-cert6
Asterisk 16.9.0

> Incorrect UTF-8 handling when using function FILTER
> ---------------------------------------------------
>
>                 Key: ASTERISK-28830
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-28830
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Functions/General
>    Affects Versions: 13.31.0
>            Reporter: newborn
>
> As you can see, after using FILTER we're getting wrong symbols (Коп▒▒а")
> This causes dialplan errors and crash 
> How to filter UTF-8 (Russian alphabet) correctly?
> Code
> {noformat}
> NoOp(.. Processing call from ${CALLERID(all)} to ${EXTEN}@${CONTEXT}-GLOBAL ..);
> Set(CALLERID(name)=${FILTER(А-Яа-яA-Za-z\040\050\051\055\056,${CALLERID(name)})});
> {noformat}
> Result
> {noformat}
> [Apr 14 18:26:08]     -- Executing [0999 at inbound-trunk:1] NoOp("IAX2/02asterisk03-2864", ".. Processing call from "ООО Рога и Копыта" <7***> to 0299 at inbound-trunk-GLOBAL ..") in new stack
> [Apr 14 18:26:08]     -- Executing [0299 at inbound-trunk:2] Set("IAX2/msk02asterisk03-11656", "CALLERID(name)=ООО Рога и Коп▒▒а") in new stack
> [Apr 14 18:26:08] WARNING[1322]: res_odbc.c:550 ast_odbc_print_errors: SQL Execute returned an error: 22021: ERROR: invalid byte sequence for encoding "UTF8": 0xd1 0xd1;
> Error while preparing parameters
> [Apr 14 18:26:08] WARNING[1322]: res_odbc.c:445 ast_odbc_prepare_and_execute: SQL Execute error -1!
> [Apr 14 18:26:08] WARNING[1322]: cel_odbc.c:797 odbc_log: Insert failed on 'asterisk-local:cel'.  CEL failed: INSERT INTO cel 
> [Apr 14 18:26:16]     -- Executing [s at macro-inboundcall:5] Set("SIP/00001a28", "CALLFILENAME=in-2020-04-14-Tuesday--18h26m--from--ООО Рога и Коп▒▒а-7***--to-999--1586877968.66779") in new stack
> [Apr 14 18:26:16] ERROR[21790][C-00000df1]: json.c:870 ast_json_vpack: Error building JSON from '{s: s, s: s}': Invalid UTF-8 string.
> [Apr 14 18:26:16] ERROR[21790][C-00000df1]:   Got 19 backtrace records
> # 0: /usr/sbin/asterisk(ast_json_pack+0x99) [0x4ff6e9]
> # 1: /usr/sbin/asterisk(ast_channel_publish_varset+0x25) [0x5874a5]
> # 2: /usr/sbin/asterisk(pbx_builtin_setvar_helper+0x123) [0x54b143]
> # 3: /usr/sbin/asterisk(pbx_builtin_setvar+0xc6) [0x54b3f6]
> # 4: /usr/sbin/asterisk(pbx_exec+0xb9) [0x5432d9]
> # 5: /usr/sbin/asterisk() [0x53b92e]
> # 6: /usr/sbin/asterisk(ast_spawn_extension+0x30) [0x53c410]
> # 7: /usr/lib64/asterisk/modules/app_macro.so(+0x2d17) [0x7f89f309dd17]
> # 8: /usr/sbin/asterisk(pbx_exec+0xb9) [0x5432d9]
> # 9: /usr/sbin/asterisk(ast_app_exec_macro+0x5e) [0x443f6e]
> #10: /usr/lib64/asterisk/modules/app_dial.so(+0xbd0d) [0x7f89f6118d0d]
> #11: /usr/lib64/asterisk/modules/app_dial.so(+0xd2d6) [0x7f89f611a2d6]
> #12: /usr/sbin/asterisk(pbx_exec+0xb9) [0x5432d9]
> #13: /usr/sbin/asterisk() [0x53b92e]
> #14: /usr/sbin/asterisk() [0x53c95e]
> #15: /usr/sbin/asterisk() [0x53de1b]
> #16: /usr/sbin/asterisk() [0x5a3f4e]
> #17: /lib64/libpthread.so.0(+0x7e65) [0x7f8a2ec4fe65]
> #18: /lib64/libc.so.6(clone+0x6d) [0x7f8a2e22088d]
> {noformat}

--
This message was sent by Atlassian JIRA
(v6.2#6252)