[asterisk-dev] Journald support for Asterisk
Bruce Ferrell
bferrell at baywinds.org
Sun May 10 17:17:20 CDT 2015
On 05/10/2015 02:22 PM, Ludovic Gasc wrote:
> 2015-05-10 20:46 GMT+02:00 Bruce Ferrell <bferrell at baywinds.org <mailto:bferrell at baywinds.org>>:
>
> On 05/09/2015 01:53 PM, Ludovic Gasc wrote:
> > 2015-05-09 16:15 GMT+02:00 Bruce Ferrell <bferrell at baywinds.org <mailto:bferrell at baywinds.org> <mailto:bferrell at baywinds.org <mailto:bferrell at baywinds.org>>>:
> >
> > On 05/09/2015 01:17 AM, Ludovic Gasc wrote:
> > > Hi,
> > >
> > > Systemd and Journald is now by default on Debian Jessie and Ubuntu 15.04, as on RHEL/CentOS.
> > > Journald supports syslog format, nevertheless, at least for us, the structured log system provided with journald helps us to debug the production.
> > >
> > > The idea behind that is to attach metadata with a log line to facilitate the search with journalctl, you can write queries to find the errors.
> > >
> > > For example, with Apache: http://httpd.apache.org/docs/trunk/mod/mod_journald.html
> > >
> > > For our Python daemons, for each log line, we store account_id, request_id, endpoint and method used to be easy to retrieve quickly interesting logs.
> > >
> > > Moreover, not yet used for us, but you can generate statistics about your source code real usage based
> > > on: http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html#CODE_FILE=
> > >
> > > For Asterisk, for example with dialplan logging, you should attach the context, the extension and the channel.
> > >
> > > You can simulate this journald feature with a specific format message for your logs, nevertheless, journalctl is more user-friendly to retrieve pertinent logs compare
> to the
> > > classical grep usage.
> > > ave no idea if i
> > > I'm permitted to raise the question about an eventual journald support in Asterisk because I don't find anything about that in issues tracker nor mailing list.
> > >
> > > Moreover, I understand that even if you add the journald support, it's certainly necessary to change logging everywhere in Asterisk, however, I should help to do that.
> > >
> > > Regards
> > > --
> > > Ludovic Gasc (GMLudo)
> > > http://www.gmludo.eu/
> > >
> > >
> > systemd and journald SUCK to high heaven.
> >
> >
> > I've no problems to believe you, nevertheless, could you be more precise ?
> > What are the facts that you arrive to this conclusion ?
> >
> >
> > I have no idea if the issues I've had with them (OpenSUSE) are distro related or inherent.
> >
> >
> > It's issues on your production with several servers ? Or on your laptop ?
> > Could give us links with unresolved bugs you have ?
> > For now, to my experience, with 6 VMs on Debian Jessie on production with small clients and Asterisk 13, I've no issues with systemd nor journald.
> >
> >
> > I have seen that the implementation of apache with a static systemd/journald module will no longer correctly serve content.
> >
> >
> > Could you give me a link about that ?
> I would suggest you try it for yourself. Procedure to reproduce:
>
> Perform fresh install of OpenSUSE 13.2
> assure apache/php/MySQL support are installed
> install wordpress from wordpress.org <http://wordpress.org>
> Observe CSS is NOT correctly served.
>
> I did this several times to assure my findings.
>
> I rebuilt the opensuse rpms from the source rpms to omit systemd/journald from the apache build. Correct operation was now observed
>
>
> Interesting, it seems like a regression bug.
> The question now is the patchs from opensuse has broken this module, or the bug is in Apache itself.
> The bug seems to be too obvious to be present in Apache source code, but who knows ?
>
>
> The response to the bug report was "Use Nginex"
>
>
> To be really honest with you, I've thought that when you explain your problem, I've dropped Apache on our production a long time ago for efficiency issues.
>
>
>
> > Please do NOT do this
> >
>
> > Why ? If it works for me, what's the problem for you ?
>
> Systemd integration has been insidious. Should you wish to discuss further, please off this list.
>
> >
> >
> > It also breaks fail2ban site security.
> >
> >
> > I don't understand the issue here, I see at least two solutions:
> > 1. You can continue to use rsyslog even if you have journald, BTW, it's the default setup of Debian Jessie, where journald forwards everything to rsyslog to avoid to perturb
> > sysadmins with log files.
> > 2. Apparently, fail2ban supports journald: https://github.com/fail2ban/fail2ban/pull/82
>
> 1.) I don't know of any asterisk systems that use any form of system logging. It can, and the fact that you keep bringing syslog logging into this indicates a lack of
> understanding of asterisk usage on your part.
> 2.) Not released code and still under test. This is disingenuous, at best.
> 3.) Just because everyone else is jumping off of bridges isn't a good enough reason to do it. It's NOT broken and this is NOT an improvement.
>
>
> Concretely, we have an infrastructure that for now handles more than 3 millions of SIP calls by month with a dedicated telco team for that, and we're continuing to grow.
>
> For now, even with voipmonitor and homer, it's a nightmare for our telco team to debug that.
> One part of the problem is because we have an heterogeneous architecture with kamailio, freeswitch and asterisk, but also, as most telco solutions in the cloud, we're trying to
> stack the most clients as possible with the less VMs: It means that each node is in tenant mode => no nodes are dedicated for one client.
> To improve this situation, I'm looking for a solution to invert the classical UNIX model: to split logs not by daemon, but by client => instead of a telco guy becomes crazy to
> parse with grep each daemon log and track one call with different ids, telco guys want to track easily all history to understand why the system has taken this decision to do
> that. One of a concrete example is Asterisk and FastAGI daemons interactions.
>
This entire argument is a straw man and the proposal breaks just about every existing monitoring system.
I AM a "telco" guy. I go back to the days of analog echo cancellation and 2600hz signalling, went through the digital conversion and now IP telephony. I've worked with regional
tandem systems (Nortel DMS250/500) that handled 3 million calls an HOUR, for an individual switch. Operational analysis is what I do.
SIP call records are easy to correlate and trouble shoot. Asterisk logs are equally easy to trace. Asterisk does not have multiple daemons and if it makes calls to external
processes, the is the functional equivalent of sending out to SS7 signalling transfer point or other external process and evaluating the return. In the circumstance, what did I
send, what did I get back? Is that a correct response?
Trouble shooting is ALWAYS a case of breaking a problem into it's component parts and diagnosing each.... "This input works, it send's the right output, it's passed to the next
component... Oops, why did THAT fail... On to the next".
When I have to do real-time troubleshooting, I fall back on tried and true methods... protocol analyzers (tcpdump and wireshark)... And I don't have to do much real-time trouble
shooting once I get something working and it bloody well better stay that way, unless someone changes something. That goes back to good systems management practice.
Log structure comes from structured system design... Not patching formatting onto messages. Asterisk has well designed log messages and used correctly in conjunction with log
analysis are invaluable.
>From all you've had to say, and what you say you've tried (ELK most notably... It's a bitch, but extremely powerful seems to be the consensus) what you really seem to want is a
complete rewrite of the logging subsystem or log analysis to "just happen" at little to no "cost". Filtering logs is always after the fact... Extremely useful for long term trend
analysis, but not so much for real time. Splunk, solarwinds and graylog are terrific at this... and journald is neither supported nor needed to implement them. Even then, a human
being has to determine what a logged event means and assign a meaning and weight.
There are no magic bullets... It "just happens" is a myth for lazy managers and programmers, in the same category as unicorns and flying kittens.
> journald itself isn't clearly a silver bullet to handle all problematic, nevertheless, the first step is to have a better categorization of logs for after build a tool behind that.
> In the past, we've tried a little bit ELK but it was very complicated, heavy process for a few ROI from our experience.
>
> We'd already made a small tool to parse Asterisk logs and categorize them, but under high load, we had efficiency issues: it didn't parse enough fastly to be useable for a direct
> live debugging.
>
> It's why I've made a little labo with some clients to validate my ideas before a bigger production, because I don't trust the mainstream nor hipster architectures promoted in IT
> world.
> I "believe" only in concrete values I can measure on concrete use cases useful for my users: clients or telco guys.
>
> Ok, while Asterisk doesn't support structured logs, the workaround is to implement a more efficient parser for Asterisk logs to categorize that.
>
> Thanks for your attention.
>
>
> >
> >
> > Asterisk works well now. It integrates easily.
> >
> >
> > For this point, I'm agree with you, it's very important to continue to be integrated easily, I use that a lot.
> >
> >
> >
> >
> >
> > --
> > _____________________________________________________________________
> > -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> >
> > asterisk-dev mailing list
> > To UNSUBSCRIBE or update options visit:
> > http://lists.digium.com/mailman/listinfo/asterisk-dev
> >
> >
> >
> >
>
>
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
> http://lists.digium.com/mailman/listinfo/asterisk-dev
>
>
>
>
More information about the asterisk-dev
mailing list