[asterisk-dev] Journald support for Asterisk

Sun May 10 16:22:00 CDT 2015

2015-05-10 20:46 GMT+02:00 Bruce Ferrell <bferrell at baywinds.org>:

> On 05/09/2015 01:53 PM, Ludovic Gasc wrote:
> > 2015-05-09 16:15 GMT+02:00 Bruce Ferrell <bferrell at baywinds.org <mailto:
> bferrell at baywinds.org>>:
> >
> >     On 05/09/2015 01:17 AM, Ludovic Gasc wrote:
> >     > Hi,
> >     >
> >     > Systemd and Journald is now by default on Debian Jessie and Ubuntu
> 15.04, as on RHEL/CentOS.
> >     > Journald supports syslog format, nevertheless, at least for us,
> the structured log system provided with journald helps us to debug the
> production.
> >     >
> >     > The idea behind that is to attach metadata with a log line to
> facilitate the search with journalctl, you can write queries to find the
> errors.
> >     >
> >     > For example, with Apache:
> http://httpd.apache.org/docs/trunk/mod/mod_journald.html
> >     >
> >     > For our Python daemons, for each log line, we store account_id,
> request_id, endpoint and method used to be easy to retrieve quickly
> interesting logs.
> >     >
> >     > Moreover, not yet used for us, but you can generate statistics
> about your source code real usage based
> >     > on:
> http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html#CODE_FILE=
> >     >
> >     > For Asterisk, for example with dialplan logging, you should attach
> the context, the extension and the channel.
> >     >
> >     > You can simulate this journald feature with a specific format
> message for your logs, nevertheless, journalctl is more user-friendly to
> retrieve pertinent logs compare to the
> >     > classical grep usage.
> >     > ave no idea if i
> >     > I'm permitted to raise the question about an eventual journald
> support in Asterisk because I don't find anything about that in issues
> tracker nor mailing list.
> >     >
> >     > Moreover, I understand that even if you add the journald support,
> it's certainly necessary to change logging everywhere in Asterisk, however,
> I should help to do that.
> >     >
> >     > Regards
> >     > --
> >     > Ludovic Gasc (GMLudo)
> >     > http://www.gmludo.eu/
> >     >
> >     >
> >     systemd and journald SUCK to high heaven.
> >
> >
> > I've no problems to believe you, nevertheless, could you be more precise
> ?
> > What are the facts that you arrive to this conclusion ?
> >
> >
> >       I have no idea if the issues I've had with them (OpenSUSE) are
> distro related or inherent.
> >
> >
> > It's issues on your production with several servers ? Or on your laptop ?
> > Could give us links with unresolved bugs you have ?
> > For now, to my experience, with 6 VMs on Debian Jessie on production
> with small clients and Asterisk 13, I've no issues with systemd nor
> journald.
> >
> >
> >     I have seen that the implementation of apache with a static
> systemd/journald module will no longer correctly serve content.
> >
> >
> > Could you give me a link about that ?
> I would suggest you try it for yourself.  Procedure to reproduce:
>
> Perform fresh install of OpenSUSE 13.2
> assure apache/php/MySQL support are installed
> install wordpress from wordpress.org
> Observe CSS is NOT correctly served.
>
> I did this several times to assure my findings.
>
> I rebuilt the opensuse rpms from the source rpms to omit systemd/journald
> from the apache build.  Correct operation was now observed
>

Interesting, it seems like a regression bug.
The question now is the patchs from opensuse has broken this module, or the
bug is in Apache itself.
The bug seems to be too obvious to be present in Apache source code, but
who knows ?

> The response to the bug report was "Use Nginex"
>

To be really honest with you, I've thought that when you explain your
problem, I've dropped Apache on our production a long time ago for
efficiency issues.

>
> >     Please do NOT do this
> >
>
> > Why ? If it works for me, what's the problem for you ?
>
>   Systemd integration has been insidious.  Should you wish to discuss
> further, please off this list.
>
> >
> >
> >       It also breaks fail2ban site security.
> >
> >
> > I don't understand the issue here, I see at least two solutions:
> > 1. You can continue to use rsyslog even if you have journald, BTW, it's
> the default setup of Debian Jessie, where journald forwards everything to
> rsyslog to avoid to perturb
> > sysadmins with log files.
> > 2. Apparently, fail2ban supports journald:
> https://github.com/fail2ban/fail2ban/pull/82
>
> 1.) I don't know of any asterisk systems that use any form of system
> logging.  It can, and the fact that you keep bringing syslog logging into
> this indicates a lack of
> understanding of asterisk usage on your part.
> 2.) Not released code and still under test.  This is disingenuous, at best.
> 3.) Just because everyone else is jumping off of bridges isn't a good
> enough reason to do it.  It's NOT broken and this is NOT an improvement.
>

Concretely, we have an infrastructure that for now handles more than 3
millions of SIP calls by month with a dedicated telco team for that, and
we're continuing to grow.

For now, even with voipmonitor and homer, it's a nightmare for our telco
team to debug that.
One part of the problem is because we have an heterogeneous architecture
with kamailio, freeswitch and asterisk, but also, as most telco solutions
in the cloud, we're trying to stack the most clients as possible with the
less VMs: It means that each node is in tenant mode => no nodes are
dedicated for one client.

To improve this situation, I'm looking for a solution to invert the
classical UNIX model: to split logs not by daemon, but by client => instead
of a telco guy becomes crazy to parse with grep each daemon log and track
one call with different ids, telco guys want to track easily all history to
understand why the system has taken this decision to do that. One of a
concrete example is Asterisk and FastAGI daemons interactions.

journald itself isn't clearly a silver bullet to handle all problematic,
nevertheless, the first step is to have a better categorization of logs for
after build a tool behind that.
In the past, we've tried a little bit ELK but it was very complicated,
heavy process for a few ROI from our experience.

We'd already made a small tool to parse Asterisk logs and categorize them,
but under high load, we had efficiency issues: it didn't parse enough
fastly to be useable for a direct live debugging.

It's why I've made a little labo with some clients to validate my ideas
before a bigger production, because I don't trust the mainstream nor
hipster architectures promoted in IT world.
I "believe" only in concrete values I can measure on concrete use cases
useful for my users: clients or telco guys.

Ok, while Asterisk doesn't support structured logs, the workaround is to
implement a more efficient parser for Asterisk logs to categorize that.

Thanks for your attention.

> >
> >
> >     Asterisk works well now.  It integrates easily.
> >
> >
> > For this point, I'm agree with you, it's very important to continue to
> be integrated easily, I use that a lot.
> >
> >
> >
> >
> >
> >     --
> >     _____________________________________________________________________
> >     -- Bandwidth and Colocation Provided by http://www.api-digital.com
> --
> >
> >     asterisk-dev mailing list
> >     To UNSUBSCRIBE or update options visit:
> >        http://lists.digium.com/mailman/listinfo/asterisk-dev
> >
> >
> >
> >
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20150510/2e01b12b/attachment.html>