[asterisk-dev] Journald support for Asterisk

Sat May 9 16:35:05 CDT 2015

2015-05-09 16:52 GMT+02:00 Matthew Jordan <mjordan at digium.com>:

> On Sat, May 9, 2015 at 9:15 AM, Bruce Ferrell <bferrell at baywinds.org>
> wrote:
> > On 05/09/2015 01:17 AM, Ludovic Gasc wrote:
> >> Hi,
> >>
> >> Systemd and Journald is now by default on Debian Jessie and Ubuntu
> 15.04, as on RHEL/CentOS.
> >> Journald supports syslog format, nevertheless, at least for us, the
> structured log system provided with journald helps us to debug the
> production.
> >>
> >> The idea behind that is to attach metadata with a log line to
> facilitate the search with journalctl, you can write queries to find the
> errors.
> >>
> >> For example, with Apache:
> http://httpd.apache.org/docs/trunk/mod/mod_journald.html
> >>
> >> For our Python daemons, for each log line, we store account_id,
> request_id, endpoint and method used to be easy to retrieve quickly
> interesting logs.
> >>
> >> Moreover, not yet used for us, but you can generate statistics about
> your source code real usage based
> >> on:
> http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html#CODE_FILE=
> >>
> >> For Asterisk, for example with dialplan logging, you should attach the
> context, the extension and the channel.
> >>
> >> You can simulate this journald feature with a specific format message
> for your logs, nevertheless, journalctl is more user-friendly to retrieve
> pertinent logs compare to the
> >> classical grep usage.
> >> ave no idea if i
> >> I'm permitted to raise the question about an eventual journald support
> in Asterisk because I don't find anything about that in issues tracker nor
> mailing list.
> >>
> >> Moreover, I understand that even if you add the journald support, it's
> certainly necessary to change logging everywhere in Asterisk, however, I
> should help to do that.
> >>
> >> Regards
> >> --
> >> Ludovic Gasc (GMLudo)
> >> http://www.gmludo.eu/
> >>
> >>
> > systemd and journald SUCK to high heaven.  I have no idea if the issues
> I've had with them (OpenSUSE) are distro related or inherent.
> >
> > I have seen that the implementation of apache with a static
> systemd/journald module will no longer correctly serve content.
> >
> > Please do NOT do this.  It also breaks fail2ban site security.  Asterisk
> works well now.  It integrates easily.
>
> Being very honest, my knowledge of systemd and journald is not
> extensive. I have no real position on systemd/journald one way or the
> other. I will say that I do not view it as an inevitable conclusion,
> nor a necessary requirement for Asterisk. As such, I'm viewing this
> only as a proposal for "we'd like to change the vast majority of log
> statements in Asterisk."
>
> First, as Bruno pointed out, a lot of external systems rely on parsing
> log messages. Asterisk does not exist in a vacuum, and is often a
> component in a much larger system. As such, any modification to
> Asterisk's logging would have to be configurable.
>

The idea is to add metadata to logs, not to change logs mechanism, like
with Python logging where you have an "extra" parameter to give more
details.
If your handler like syslog doesn't handle metadata, it isn't used, example
with Python:
http://www.freedesktop.org/software/systemd/python-systemd/journal.html#journalhandler-class
I can have a journalhandler, consolehandler and sysloghandler,
only journalhandler will handle "extra" parameter without disturb others
handlers.

Even if you wish drop syslog and files support in asterisk, with journald,
it's possible to repush logs in a rsyslog, to continue to use log parsers.

> Second, the logging system in Asterisk has a measurable impact on
> performance. Unfortunately, adding a configuration option that is
> evaluated on every log statement will further degrade performance when
> logging verbosity is increased. A compile time option is not a
> pleasant alternative either. Note that we experienced a similar
> situation in Asterisk 11 with the remote console independent verbosity
> setting, and spent a lot of time over many point releases getting the
> performance back in line.
>

Yes, it's also my "fair", I need to do a macro-benchmark to measure
globally the real impact to provide more data in logs.
It should appear catastrophic on a micro-benchmark, but on a
macro-benchmark with all stack, it should be a detail compare to the rest
of processing.

>
> It is also telling that Apache's mod_journald states:
> {quote}
> Performance warning
>
> Currently, systemd-journald is not designed for high-throughput
> logging and logging access_log to systemd-journald could decrease the
> performance a lot.
> {quote}
>

Very interesting, I'll benchmark that to have a number rate behind "not
designed for high-throughput
logging" sentence.

Third, journald seems to be providing structured log data. While some
> logging statements in Asterisk may benefit from such a system, many
> may not. Asterisk can be *extremely* chatty.

I'm a little bit aware of that: with my colleagues, we're managing between
400 and 500 Asterisks (no, it isn't a joke) for our infrastructure and at
our clients ;-)

> I'm not sure that every
> log statement in Asterisk should be converted or support such a
> system.

Add more metadata it isn't mandatory with journald, you can send only the
minimal piece of information, more or less like with syslog protocol.

A more concrete proposal over what messages are appropriate
> for journald would be nice.
>

I'll provide more concrete use cases with a fake logging to show you that.

> I'd be curious if another option would be to treat journald as we
> currently treat syslog - not as a formatting option, but as a
> destination. If that were possible, support could be added for it
> without necessarily impacting the existing log statements. Asterisk
> supports the concept of having dynamic log channels, and support for
> journald could then also be added as an external module without
> impacting the core - or impacting users who don't care for it.
>

Thanks for the suggestion, but at least to me, change the logging protocol
only for the pleasure it isn't really interesting, moreover because
journald supports syslog protocol. I'm interested by metadata, I don't care
that I retrieve that via syslog, journald or HTTP protocol.
Unless if you explain me that in a logging channel, it's possible to
inspect the function log caller to know his origin and slurp context to
send, I don't think that your suggestion should change something to my use
case.

In fact, the idea should be to benchmark a small PoC with a patched version
of Asterisk for journald only for some few things, to measure the overhead
with journald, but I don't have yet enough knowledge to do that in Asterisk.
I could provide us the benchmarks numbers for Python when I'll do, but not
really sure that Python results should proportionally apply on Asterisk.

By the way, I'll be present to KamailioWorld this year at Berlin, if you
wish to discuss face-to-face, it would be interesting.

Have a nice week-end.

> --
> Matthew Jordan
> Digium, Inc. | Director of Technology
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> Check us out at: http://digium.com & http://asterisk.org
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-dev
>

--
Ludovic Gasc (GMLudo)
http://www.gmludo.eu/

2015-05-09 16:52 GMT+02:00 Matthew Jordan <mjordan at digium.com>:

> On Sat, May 9, 2015 at 9:15 AM, Bruce Ferrell <bferrell at baywinds.org>
> wrote:
> > On 05/09/2015 01:17 AM, Ludovic Gasc wrote:
> >> Hi,
> >>
> >> Systemd and Journald is now by default on Debian Jessie and Ubuntu
> 15.04, as on RHEL/CentOS.
> >> Journald supports syslog format, nevertheless, at least for us, the
> structured log system provided with journald helps us to debug the
> production.
> >>
> >> The idea behind that is to attach metadata with a log line to
> facilitate the search with journalctl, you can write queries to find the
> errors.
> >>
> >> For example, with Apache:
> http://httpd.apache.org/docs/trunk/mod/mod_journald.html
> >>
> >> For our Python daemons, for each log line, we store account_id,
> request_id, endpoint and method used to be easy to retrieve quickly
> interesting logs.
> >>
> >> Moreover, not yet used for us, but you can generate statistics about
> your source code real usage based
> >> on:
> http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html#CODE_FILE=
> >>
> >> For Asterisk, for example with dialplan logging, you should attach the
> context, the extension and the channel.
> >>
> >> You can simulate this journald feature with a specific format message
> for your logs, nevertheless, journalctl is more user-friendly to retrieve
> pertinent logs compare to the
> >> classical grep usage.
> >> ave no idea if i
> >> I'm permitted to raise the question about an eventual journald support
> in Asterisk because I don't find anything about that in issues tracker nor
> mailing list.
> >>
> >> Moreover, I understand that even if you add the journald support, it's
> certainly necessary to change logging everywhere in Asterisk, however, I
> should help to do that.
> >>
> >> Regards
> >> --
> >> Ludovic Gasc (GMLudo)
> >> http://www.gmludo.eu/
> >>
> >>
> > systemd and journald SUCK to high heaven.  I have no idea if the issues
> I've had with them (OpenSUSE) are distro related or inherent.
> >
> > I have seen that the implementation of apache with a static
> systemd/journald module will no longer correctly serve content.
> >
> > Please do NOT do this.  It also breaks fail2ban site security.  Asterisk
> works well now.  It integrates easily.
>
> Being very honest, my knowledge of systemd and journald is not
> extensive. I have no real position on systemd/journald one way or the
> other. I will say that I do not view it as an inevitable conclusion,
> nor a necessary requirement for Asterisk. As such, I'm viewing this
> only as a proposal for "we'd like to change the vast majority of log
> statements in Asterisk."
>
> First, as Bruno pointed out, a lot of external systems rely on parsing
> log messages. Asterisk does not exist in a vacuum, and is often a
> component in a much larger system. As such, any modification to
> Asterisk's logging would have to be configurable.
>
> Second, the logging system in Asterisk has a measurable impact on
> performance. Unfortunately, adding a configuration option that is
> evaluated on every log statement will further degrade performance when
> logging verbosity is increased. A compile time option is not a
> pleasant alternative either. Note that we experienced a similar
> situation in Asterisk 11 with the remote console independent verbosity
> setting, and spent a lot of time over many point releases getting the
> performance back in line.
>
> It is also telling that Apache's mod_journald states:
> {quote}
> Performance warning
>
> Currently, systemd-journald is not designed for high-throughput
> logging and logging access_log to systemd-journald could decrease the
> performance a lot.
> {quote}
>
> Third, journald seems to be providing structured log data. While some
> logging statements in Asterisk may benefit from such a system, many
> may not. Asterisk can be *extremely* chatty. I'm not sure that every
> log statement in Asterisk should be converted or support such a
> system. A more concrete proposal over what messages are appropriate
> for journald would be nice.
>
> I'd be curious if another option would be to treat journald as we
> currently treat syslog - not as a formatting option, but as a
> destination. If that were possible, support could be added for it
> without necessarily impacting the existing log statements. Asterisk
> supports the concept of having dynamic log channels, and support for
> journald could then also be added as an external module without
> impacting the core - or impacting users who don't care for it.
>
> --
> Matthew Jordan
> Digium, Inc. | Director of Technology
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> Check us out at: http://digium.com & http://asterisk.org
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20150509/e194e86e/attachment-0001.html>