[asterisk-dev] Journald support for Asterisk

Tue May 12 00:24:42 CDT 2015

CentOS/RHEL 6 are under production support until Nov 2020 [1] and journald
is not available.  Even if journald could improve Asterisk logging, that's
not an acceptable reason to force Asterisk users to upgrade from a
supported OS.  If there was a module that allowed Asterisk logging to talk
to journald I wouldn't care, but it should not replace the logging system
or be required to run Asterisk.

[1]
http://wiki.centos.org/FAQ/General#head-fe8a0be91ee3e7dea812e8694491e1dde5b75e6d

On Sun, May 10, 2015 at 5:22 PM, Ludovic Gasc <gmludo at gmail.com> wrote:

> 2015-05-10 20:46 GMT+02:00 Bruce Ferrell <bferrell at baywinds.org>:
>
>> On 05/09/2015 01:53 PM, Ludovic Gasc wrote:
>> > 2015-05-09 16:15 GMT+02:00 Bruce Ferrell <bferrell at baywinds.org
>> <mailto:bferrell at baywinds.org>>:
>> >
>> >     On 05/09/2015 01:17 AM, Ludovic Gasc wrote:
>> >     > Hi,
>> >     >
>> >     > Systemd and Journald is now by default on Debian Jessie and
>> Ubuntu 15.04, as on RHEL/CentOS.
>> >     > Journald supports syslog format, nevertheless, at least for us,
>> the structured log system provided with journald helps us to debug the
>> production.
>> >     >
>> >     > The idea behind that is to attach metadata with a log line to
>> facilitate the search with journalctl, you can write queries to find the
>> errors.
>> >     >
>> >     > For example, with Apache:
>> http://httpd.apache.org/docs/trunk/mod/mod_journald.html
>> >     >
>> >     > For our Python daemons, for each log line, we store account_id,
>> request_id, endpoint and method used to be easy to retrieve quickly
>> interesting logs.
>> >     >
>> >     > Moreover, not yet used for us, but you can generate statistics
>> about your source code real usage based
>> >     > on:
>> http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html#CODE_FILE=
>> >     >
>> >     > For Asterisk, for example with dialplan logging, you should
>> attach the context, the extension and the channel.
>> >     >
>> >     > You can simulate this journald feature with a specific format
>> message for your logs, nevertheless, journalctl is more user-friendly to
>> retrieve pertinent logs compare to the
>> >     > classical grep usage.
>> >     > ave no idea if i
>> >     > I'm permitted to raise the question about an eventual journald
>> support in Asterisk because I don't find anything about that in issues
>> tracker nor mailing list.
>> >     >
>> >     > Moreover, I understand that even if you add the journald support,
>> it's certainly necessary to change logging everywhere in Asterisk, however,
>> I should help to do that.
>> >     >
>> >     > Regards
>> >     > --
>> >     > Ludovic Gasc (GMLudo)
>> >     > http://www.gmludo.eu/
>> >     >
>> >     >
>> >     systemd and journald SUCK to high heaven.
>> >
>> >
>> > I've no problems to believe you, nevertheless, could you be more
>> precise ?
>> > What are the facts that you arrive to this conclusion ?
>> >
>> >
>> >       I have no idea if the issues I've had with them (OpenSUSE) are
>> distro related or inherent.
>> >
>> >
>> > It's issues on your production with several servers ? Or on your laptop
>> ?
>> > Could give us links with unresolved bugs you have ?
>> > For now, to my experience, with 6 VMs on Debian Jessie on production
>> with small clients and Asterisk 13, I've no issues with systemd nor
>> journald.
>> >
>> >
>> >     I have seen that the implementation of apache with a static
>> systemd/journald module will no longer correctly serve content.
>> >
>> >
>> > Could you give me a link about that ?
>> I would suggest you try it for yourself.  Procedure to reproduce:
>>
>> Perform fresh install of OpenSUSE 13.2
>> assure apache/php/MySQL support are installed
>> install wordpress from wordpress.org
>> Observe CSS is NOT correctly served.
>>
>> I did this several times to assure my findings.
>>
>> I rebuilt the opensuse rpms from the source rpms to omit systemd/journald
>> from the apache build.  Correct operation was now observed
>>
>
> Interesting, it seems like a regression bug.
> The question now is the patchs from opensuse has broken this module, or
> the bug is in Apache itself.
> The bug seems to be too obvious to be present in Apache source code, but
> who knows ?
>
>
>> The response to the bug report was "Use Nginex"
>>
>
> To be really honest with you, I've thought that when you explain your
> problem, I've dropped Apache on our production a long time ago for
> efficiency issues.
>
>
>>
>> >     Please do NOT do this
>> >
>>
>> > Why ? If it works for me, what's the problem for you ?
>>
>>   Systemd integration has been insidious.  Should you wish to discuss
>> further, please off this list.
>>
>> >
>> >
>> >       It also breaks fail2ban site security.
>> >
>> >
>> > I don't understand the issue here, I see at least two solutions:
>> > 1. You can continue to use rsyslog even if you have journald, BTW, it's
>> the default setup of Debian Jessie, where journald forwards everything to
>> rsyslog to avoid to perturb
>> > sysadmins with log files.
>> > 2. Apparently, fail2ban supports journald:
>> https://github.com/fail2ban/fail2ban/pull/82
>>
>> 1.) I don't know of any asterisk systems that use any form of system
>> logging.  It can, and the fact that you keep bringing syslog logging into
>> this indicates a lack of
>> understanding of asterisk usage on your part.
>> 2.) Not released code and still under test.  This is disingenuous, at
>> best.
>> 3.) Just because everyone else is jumping off of bridges isn't a good
>> enough reason to do it.  It's NOT broken and this is NOT an improvement.
>>
>
> Concretely, we have an infrastructure that for now handles more than 3
> millions of SIP calls by month with a dedicated telco team for that, and
> we're continuing to grow.
>
> For now, even with voipmonitor and homer, it's a nightmare for our telco
> team to debug that.
> One part of the problem is because we have an heterogeneous architecture
> with kamailio, freeswitch and asterisk, but also, as most telco solutions
> in the cloud, we're trying to stack the most clients as possible with the
> less VMs: It means that each node is in tenant mode => no nodes are
> dedicated for one client.
>
> To improve this situation, I'm looking for a solution to invert the
> classical UNIX model: to split logs not by daemon, but by client => instead
> of a telco guy becomes crazy to parse with grep each daemon log and track
> one call with different ids, telco guys want to track easily all history to
> understand why the system has taken this decision to do that. One of a
> concrete example is Asterisk and FastAGI daemons interactions.
>
> journald itself isn't clearly a silver bullet to handle all problematic,
> nevertheless, the first step is to have a better categorization of logs for
> after build a tool behind that.
> In the past, we've tried a little bit ELK but it was very complicated,
> heavy process for a few ROI from our experience.
>
> We'd already made a small tool to parse Asterisk logs and categorize them,
> but under high load, we had efficiency issues: it didn't parse enough
> fastly to be useable for a direct live debugging.
>
> It's why I've made a little labo with some clients to validate my ideas
> before a bigger production, because I don't trust the mainstream nor
> hipster architectures promoted in IT world.
> I "believe" only in concrete values I can measure on concrete use cases
> useful for my users: clients or telco guys.
>
> Ok, while Asterisk doesn't support structured logs, the workaround is to
> implement a more efficient parser for Asterisk logs to categorize that.
>
> Thanks for your attention.
>
>
>> >
>> >
>> >     Asterisk works well now.  It integrates easily.
>> >
>> >
>> > For this point, I'm agree with you, it's very important to continue to
>> be integrated easily, I use that a lot.
>> >
>> >
>> >
>> >
>> >
>> >     --
>> >
>>  _____________________________________________________________________
>> >     -- Bandwidth and Colocation Provided by http://www.api-digital.com
>> --
>> >
>> >     asterisk-dev mailing list
>> >     To UNSUBSCRIBE or update options visit:
>> >        http://lists.digium.com/mailman/listinfo/asterisk-dev
>> >
>> >
>> >
>> >
>>
>>
>> --
>> _____________________________________________________________________
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> asterisk-dev mailing list
>> To UNSUBSCRIBE or update options visit:
>>    http://lists.digium.com/mailman/listinfo/asterisk-dev
>>
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20150512/7573d3e7/attachment.html>