<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">2015-05-11 0:17 GMT+02:00 Bruce Ferrell <span dir="ltr"><<a href="mailto:bferrell@baywinds.org" target="_blank">bferrell@baywinds.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">On 05/10/2015 02:22 PM, Ludovic Gasc wrote:<br>
> 2015-05-10 20:46 GMT+02:00 Bruce Ferrell <<a href="mailto:bferrell@baywinds.org">bferrell@baywinds.org</a> <mailto:<a href="mailto:bferrell@baywinds.org">bferrell@baywinds.org</a>>>:<br>
<span class="">><br>
> On 05/09/2015 01:53 PM, Ludovic Gasc wrote:<br>
</span>> > 2015-05-09 16:15 GMT+02:00 Bruce Ferrell <<a href="mailto:bferrell@baywinds.org">bferrell@baywinds.org</a> <mailto:<a href="mailto:bferrell@baywinds.org">bferrell@baywinds.org</a>> <mailto:<a href="mailto:bferrell@baywinds.org">bferrell@baywinds.org</a> <mailto:<a href="mailto:bferrell@baywinds.org">bferrell@baywinds.org</a>>>>:<br>
<div><div class="h5">> ><br>
> > On 05/09/2015 01:17 AM, Ludovic Gasc wrote:<br>
> > > Hi,<br>
> > ><br>
> > > Systemd and Journald is now by default on Debian Jessie and Ubuntu 15.04, as on RHEL/CentOS.<br>
> > > Journald supports syslog format, nevertheless, at least for us, the structured log system provided with journald helps us to debug the production.<br>
> > ><br>
> > > The idea behind that is to attach metadata with a log line to facilitate the search with journalctl, you can write queries to find the errors.<br>
> > ><br>
> > > For example, with Apache: <a href="http://httpd.apache.org/docs/trunk/mod/mod_journald.html" target="_blank">http://httpd.apache.org/docs/trunk/mod/mod_journald.html</a><br>
> > ><br>
> > > For our Python daemons, for each log line, we store account_id, request_id, endpoint and method used to be easy to retrieve quickly interesting logs.<br>
> > ><br>
> > > Moreover, not yet used for us, but you can generate statistics about your source code real usage based<br>
> > > on: <a href="http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html#CODE_FILE=" target="_blank">http://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html#CODE_FILE=</a><br>
> > ><br>
> > > For Asterisk, for example with dialplan logging, you should attach the context, the extension and the channel.<br>
> > ><br>
> > > You can simulate this journald feature with a specific format message for your logs, nevertheless, journalctl is more user-friendly to retrieve pertinent logs compare<br>
> to the<br>
> > > classical grep usage.<br>
> > > ave no idea if i<br>
> > > I'm permitted to raise the question about an eventual journald support in Asterisk because I don't find anything about that in issues tracker nor mailing list.<br>
> > ><br>
> > > Moreover, I understand that even if you add the journald support, it's certainly necessary to change logging everywhere in Asterisk, however, I should help to do that.<br>
> > ><br>
> > > Regards<br>
> > > --<br>
> > > Ludovic Gasc (GMLudo)<br>
> > > <a href="http://www.gmludo.eu/" target="_blank">http://www.gmludo.eu/</a><br>
> > ><br>
> > ><br>
> > systemd and journald SUCK to high heaven.<br>
> ><br>
> ><br>
> > I've no problems to believe you, nevertheless, could you be more precise ?<br>
> > What are the facts that you arrive to this conclusion ?<br>
> ><br>
> ><br>
> > I have no idea if the issues I've had with them (OpenSUSE) are distro related or inherent.<br>
> ><br>
> ><br>
> > It's issues on your production with several servers ? Or on your laptop ?<br>
> > Could give us links with unresolved bugs you have ?<br>
> > For now, to my experience, with 6 VMs on Debian Jessie on production with small clients and Asterisk 13, I've no issues with systemd nor journald.<br>
> ><br>
> ><br>
> > I have seen that the implementation of apache with a static systemd/journald module will no longer correctly serve content.<br>
> ><br>
> ><br>
> > Could you give me a link about that ?<br>
> I would suggest you try it for yourself. Procedure to reproduce:<br>
><br>
> Perform fresh install of OpenSUSE 13.2<br>
> assure apache/php/MySQL support are installed<br>
</div></div>> install wordpress from <a href="http://wordpress.org" target="_blank">wordpress.org</a> <<a href="http://wordpress.org" target="_blank">http://wordpress.org</a>><br>
<div><div class="h5">> Observe CSS is NOT correctly served.<br>
><br>
> I did this several times to assure my findings.<br>
><br>
> I rebuilt the opensuse rpms from the source rpms to omit systemd/journald from the apache build. Correct operation was now observed<br>
><br>
><br>
> Interesting, it seems like a regression bug.<br>
> The question now is the patchs from opensuse has broken this module, or the bug is in Apache itself.<br>
> The bug seems to be too obvious to be present in Apache source code, but who knows ?<br>
><br>
><br>
> The response to the bug report was "Use Nginex"<br>
><br>
><br>
> To be really honest with you, I've thought that when you explain your problem, I've dropped Apache on our production a long time ago for efficiency issues.<br>
><br>
><br>
><br>
> > Please do NOT do this<br>
> ><br>
><br>
> > Why ? If it works for me, what's the problem for you ?<br>
><br>
> Systemd integration has been insidious. Should you wish to discuss further, please off this list.<br>
><br>
> ><br>
> ><br>
> > It also breaks fail2ban site security.<br>
> ><br>
> ><br>
> > I don't understand the issue here, I see at least two solutions:<br>
> > 1. You can continue to use rsyslog even if you have journald, BTW, it's the default setup of Debian Jessie, where journald forwards everything to rsyslog to avoid to perturb<br>
> > sysadmins with log files.<br>
> > 2. Apparently, fail2ban supports journald: <a href="https://github.com/fail2ban/fail2ban/pull/82" target="_blank">https://github.com/fail2ban/fail2ban/pull/82</a><br>
><br>
> 1.) I don't know of any asterisk systems that use any form of system logging. It can, and the fact that you keep bringing syslog logging into this indicates a lack of<br>
> understanding of asterisk usage on your part.<br>
> 2.) Not released code and still under test. This is disingenuous, at best.<br>
> 3.) Just because everyone else is jumping off of bridges isn't a good enough reason to do it. It's NOT broken and this is NOT an improvement.<br>
><br>
><br>
> Concretely, we have an infrastructure that for now handles more than 3 millions of SIP calls by month with a dedicated telco team for that, and we're continuing to grow.<br>
><br>
> For now, even with voipmonitor and homer, it's a nightmare for our telco team to debug that.<br>
> One part of the problem is because we have an heterogeneous architecture with kamailio, freeswitch and asterisk, but also, as most telco solutions in the cloud, we're trying to<br>
> stack the most clients as possible with the less VMs: It means that each node is in tenant mode => no nodes are dedicated for one client.<br>
<br>
> To improve this situation, I'm looking for a solution to invert the classical UNIX model: to split logs not by daemon, but by client => instead of a telco guy becomes crazy to<br>
> parse with grep each daemon log and track one call with different ids, telco guys want to track easily all history to understand why the system has taken this decision to do<br>
> that. One of a concrete example is Asterisk and FastAGI daemons interactions.<br>
><br>
<br>
</div></div>This entire argument is a straw man and the proposal breaks just about every existing monitoring system.<br></blockquote><div><br></div><div>Hum, you may right, or I've maybe another context as you, who knows ?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I AM a "telco" guy. I go back to the days of analog echo cancellation and 2600hz signalling, went through the digital conversion and now IP telephony. </blockquote><div><br></div><div>Sincerely, congratulations, I don't have 10% of your skills in the telco world.</div><div>Nevertheless, one day, maybe, it could change ;-)</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I've worked with regional<br>
tandem systems (Nortel DMS250/500) that handled 3 million calls an HOUR, for an individual switch. Operational analysis is what I do.<br></blockquote><div><br></div><div>I like to discuss with the persons from production field: Even if we aren't always agree, the bullshit level is often reduced compare to a higher level of management.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
SIP call records are easy to correlate and trouble shoot. Asterisk logs are equally easy to trace. Asterisk does not have multiple daemons and if it makes calls to external<br>
processes, the is the functional equivalent of sending out to SS7 signalling transfer point or other external process and evaluating the return. In the circumstance, what did I<br>
send, what did I get back? Is that a correct response?<br>
<br>
Trouble shooting is ALWAYS a case of breaking a problem into it's component parts and diagnosing each.... "This input works, it send's the right output, it's passed to the next<br>
component... Oops, why did THAT fail... On to the next".<br>
<br>
When I have to do real-time troubleshooting, I fall back on tried and true methods... protocol analyzers (tcpdump and wireshark)... And I don't have to do much real-time trouble<br>
shooting once I get something working and it bloody well better stay that way, unless someone changes something. That goes back to good systems management practice.<br></blockquote><div><br></div><div>Sorry Bruce, but I can't be agree with you:</div><div>I completely believe you that you're a telco rock star and you have the skills to use tcpdump and you know SIP RFCs by heart.</div><div>Nevertheless, as least in my world, I'm working with guys, in the support team, who don't understand all refinements of debugging and some person don't understand a lot SIP signalling.</div><div>The issue isn't that we're incompetent to recruit people nor we have bad salary conditions nor they are impostors, but, at least in Belgium, it's really hard to find somebody with the minimal brain hardware to handle correctly IT jobs.</div><div><br></div><div>Moreover, yes, sometimes we have a pure SIP/Asterisk problem, but often, you need to check more information: Asterisk is an important but a small part of our added value: With have several daemons that interact with Asterisk, use AMI or FastAGI, receives commands from HTTP or WebSockets...</div><div><br></div><div>Here, most of Asterisk behaviour is remotely piloted by several daemons.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Log structure comes from structured system design... Not patching formatting onto messages. Asterisk has well designed log messages and used correctly in conjunction with log<br>
analysis are invaluable.<br>
<br>
>From all you've had to say, and what you say you've tried (ELK most notably... It's a bitch, but extremely powerful seems to be the consensus) what you really seem to want is a<br>
complete rewrite of the logging subsystem or log analysis to "just happen" at little to no "cost". Filtering logs is always after the fact... Extremely useful for long term trend<br>
analysis, but not so much for real time.</blockquote><div><br></div><div>Don't worry, by the door, the window, the fireplace or via a tunnel I've built myself, I always find a solution to have what I want ;-)</div><div>If my guys need that, I'll find them a solution.</div><div>To illustrate my sentence, one day, I'd an issue to send a special NOTIFY via Asterisk via AMI for a special hardphone model and I couldn't send directly SIP packet because phones are behind a NAT. I've resolved that to inject directly packets on the wire via a raw socket with Scapy. It's a really crappy solution, but it works.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> Splunk, solarwinds and graylog are terrific at this... and journald is neither supported nor needed to implement them. </blockquote><div><br></div><div>The idea behind structured log is to retrieve easily a context with a log to reduce the effort of categorization after, because you don't need to parse and recognize patterns in log message.</div><div>Technically, instead of to have a String, you have a Dict of Strings.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Even then, a human<br>
being has to determine what a logged event means and assign a meaning and weight.<br></blockquote><div><br></div><div>If the context is enough detailed, it's the job of sysadmin to decide how to query in logs.</div><div>The main issue to do that is that, as pointed by Matt, you should have efficiency impacts, but must be measured.</div><div>It shouldn't be the first time that I've seen a catastrophic micro-benchmark where in fact it's negligible on a macro-benchmark.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">There are no magic bullets...</blockquote><div><br></div><div>Completely agree with you, silver bullets don't exist, especially in the opensource telco world.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> It "just happens" is a myth for lazy managers</blockquote><div><br></div><div>I hope you don't think that I'm a manager, or maybe in your world managers write code also ;-)</div><div>For example, I've made a very small SIP stack with Python and AsyncIO.</div><div>It isn't a joke, it's on our production since 6 months to send SIP messages on endpoints and monitor BLFs.</div><div>I'll opensource that soon.</div><div>It isn't as prestigious as your echo cancellation implementation, but I hope it's enough for you to believe me that I'm a developer :-)</div><div>I don't think all developers have the level to implement a subset of SIP.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> and programmers,</blockquote><div><br></div><div>A very good programmer must be lazy first, isn't he ? ;-)<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> in the same category as unicorns and flying kittens.<br></blockquote><div><br></div><div>I'll reuse your expression, I like a lot.</div><div><br></div><div>Have a nice day.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><div class="h5"><br>
<br>
> journald itself isn't clearly a silver bullet to handle all problematic, nevertheless, the first step is to have a better categorization of logs for after build a tool behind that.<br>
> In the past, we've tried a little bit ELK but it was very complicated, heavy process for a few ROI from our experience.<br>
><br>
> We'd already made a small tool to parse Asterisk logs and categorize them, but under high load, we had efficiency issues: it didn't parse enough fastly to be useable for a direct<br>
> live debugging.<br>
><br>
> It's why I've made a little labo with some clients to validate my ideas before a bigger production, because I don't trust the mainstream nor hipster architectures promoted in IT<br>
> world.<br>
> I "believe" only in concrete values I can measure on concrete use cases useful for my users: clients or telco guys.<br>
><br>
> Ok, while Asterisk doesn't support structured logs, the workaround is to implement a more efficient parser for Asterisk logs to categorize that.<br>
><br>
> Thanks for your attention.<br>
><br>
><br>
> ><br>
> ><br>
> > Asterisk works well now. It integrates easily.<br>
> ><br>
> ><br>
> > For this point, I'm agree with you, it's very important to continue to be integrated easily, I use that a lot.<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > _____________________________________________________________________<br>
> > -- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" target="_blank">http://www.api-digital.com</a> --<br>
> ><br>
> > asterisk-dev mailing list<br>
> > To UNSUBSCRIBE or update options visit:<br>
> > <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br>
> ><br>
> ><br>
> ><br>
> ><br>
><br>
><br>
> --<br>
> _____________________________________________________________________<br>
> -- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" target="_blank">http://www.api-digital.com</a> --<br>
><br>
> asterisk-dev mailing list<br>
> To UNSUBSCRIBE or update options visit:<br>
> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br>
><br>
><br>
><br>
><br>
<br>
<br>
--<br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" target="_blank">http://www.api-digital.com</a> --<br>
<br>
asterisk-dev mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br>
</div></div></blockquote></div><br></div></div>