[asterisk-dev] Recommendations for using a SIP stack with Asterisk

[Daniel Pocock]

On 13/11/12 17:57, Mark Michelson wrote:
> You may want to jump to the bottom of this reply first so you understand
> my position a bit more clearly. I've only answered the portions of your
> latest reply where you asked a direct question since I think the
> argument has become counterproductive and I'd like to move towards a
> solution.
> 
> On 11/12/2012 09:13 PM, Faidon Liambotis wrote:
>> On 11/12/12 18:40, Mark Michelson wrote:
>>> As a point of order, I mentioned that resiprocate had limited
>>> package support. The reason I said this is because, for instance,
>>> anyone using the current LTS of Ubuntu (12.04) does not have
>>> lib-resiprocate-dev available to them by default. It was not added
>>> until Ubuntu 12.10.
>> As far as I know, you're looking at an Asterisk 12 release at
>> tentatively October 2013 and an Asterisk LTS release a year after that.
>> Several Ubuntu releases are going to be released until then and the next
>> Ubuntu LTS is going to be released in April 2014, not that far away from
>> your LTS release (esp. if you drift).
>>
>>> The issue is that probably every single one of those projects has had
>>> to make some modification to PJSIP that would not be beneficial to
>>> anyone outside of their own project. If PJLIB were to be packaged,
>>> then how could any of those projects actually use it?
>> Why? How do you know that? How do you know that one of the fixes that
>> one of these projects has made, isn't fixing a bug someone will be
>> having with Asterisk in two years time and one you're going to end up
>> debugging again?
> 
> I certainly don't know. That's why I used the word "probably". If they
> are harboring general purpose fixes for themselves, they're doing it
> [even more] wrong.
>>
>> There's no way to tell! You'd have to manually review all of what is
>> effectively active forks of that project in that large list of yours and
>> look carefully for fixes relevant to your use cases. Doesn't that suck
>> for you? Why would you want to prolong and extend this monstrosity?
> 
> Oh it would suck horribly for us to try to be proactive in this regard.
> We wouldn't be prolonging though, since as I stated in my previous reply
> to you that we would not harbor general-purpose bug fixes or
> improvements to ourselves. These would be pushed upstream in the hopes
> that everyone would benefit. I know that not all projects would go to
> the same trouble, but we definitely would.
>>
>> Do we really have to argue that merging back changes to the upstream
>> project is a good idea?
> 
> Nope. It's not only a good idea, it's the best idea.
>>> However, not all modifications made would be beneficial to anyone
>>> that is not using Asterisk. These modifications would have no
>>> business being pushed upstream.
>> Again, how do you know that? You haven't even picked the stack yet, and
>> you know how beneficial to others your future patches will be?
> 
> You're right that I certainly don't have the clairvoyance to know all
> the patches the Asterisk project will make to the SIP stack of choice in
> the future. The type of patch I was thinking of was something like
> adding calls to Asterisk's event system, adding calls to send AMI
> events, and in other ways do very Asterisk-specific things. Do I know
> such patches will be added? No. Should the Asterisk project reserve the
> right to be able to add such changes? I certainly think so.
>>> Expanding the argument to every library is irrelevant here. There
>>> are many general-purpose libraries that people use and will *never*
>>> need to modify. SIP stacks, by their very nature, are designed to be
>>> modified by those that use them.
>> I don't think it's irrelevant. I don't understand how a SIP stack is
>> different from every other library/protocol stack out there. I think
>> it's exactly the same.
>>
>> This might actually be the case for some of them, and if that's the
>> case, I believe they're very badly designed as generic stacks. Do the
>> upstreams of all of the three candidate (or possibly other) stacks feel
>> like that?
> 
> I can't accurately answer that question, since my experience lies only
> in using PJSIP, and I have not talked with the developers of the other
> projects. The opinion I express here is mine based on my understanding
> of the nature of SIP.
>>
>>> Libpri is slightly different. If you want PRI support at all, you
>>> need libpri (or something similar from a third party). You have no
>>> alternative. Tentatively, the plan for SIP support in Asterisk 12
>>> would be to provide both the "classic" chan_sip that Asterisk already
>>> includes and the new SIP channel driver. Priority would go towards
>>> using the new channel driver, but the old would be provided as a
>>> backup just in case you encountered issues.If users get told they can
>>> either use no external library and have SIP support or install an
>>> external library from source and have SIP support, they're likely to
>>> choose the easier path. This is bad since we want as much adoption as
>>> possible of the new SIP channel driver.
>> I also don't see how keeping an alternative to the new chan_sip makes
>> the situation regarding having a SIP library embedded and forked within
>> Asterisk any different. This is completely besides the point.
>>
>>> However, as has already been established, distro support for SIP
>>> stacks is low to non-existent depending on the SIP stack in use. This
>>> would mean having to install a new library from source to use the new
>>> SIP stack.
>> As said earlier, I'm fairly certain distributions will catch up long
>> before you actually release Asterisk 12, as long as there's a way for
>> them to.
>>
>> BTW, it's nice how you're arguing that SIP stacks are "designed to be
>> modified by those that use them" and at the same time complaining about
>> distro support. Of course pjsip wouldn't be in distros, that's because
>> it's currently impossible! Fix that, and you'll soon find it packaged.
>>
>>> The typical Asterisk user will not do this. They'll either continue
>>> using a previous version of Asterisk where things mostly work, or
>>> they'll upgrade to Asterisk 12 and continue to use the old SIP stack
>>> instead of the new one.
>> I think this is a red herring. Not that I'd recommend it, but you could
>> always embed an *unmodified*, released version of the library in
>> question and never patch it in your tree. Or modify your toolchain to
>> fetch it and build it at build time, like you've done in other cases in
>> the past.
>>
>>> In all these cases, the Asterisk developers are not the only barrier
>>> to the user having a fixed issue.
>> I think that's the core of the issue, as I mentioned in my previous
>> mail. You want to have control; working with the community means giving
>> up control, so you're essentially in an impasse that you want to resolve
>> by forking and embedding.
>>
>>> We absolutely want to use a third-party SIP stack because writing
>>> our own would be a complete waste of time. 95% of what we would need
>>> in a SIP stack would be provided by whatever stack we choose to use,
>>> and the SIP stacks out there are well-tested.
>> This is a bit besides the point but: if they're well-tested and they've
>> proven to be better than you at this, then why do you argue that you
>> should be patching their code, bypassing upstream's review & testing
>> process? :-)
>>
>> You're effectively admitting failure in building a SIP stack and you're
>> throwing away years of work on chan_sip only to replace it with
>> something someone else built and that you (rightfully) want to re-use.
>>
>> At the same time, you're arguing that you already "know better" from the
>> upstream (the one that you haven't even picked up yet!), and you'd
>> prefer to patch their code unhindered. How's that going to be better
>> than the current chan_sip in the long run?
> 
> We don't know better from the upstream. We simply want to reserve the
> right to be able to add to or modify the stack in order to suit our
> needs as a project.
> 
> To give a concrete example, Asterisk 11 has websocket support for SIP.
> None of the SIP stacks under consideration have this yet. If we desire
> feature parity with previous versions of Asterisk, then we'd need to
> select a SIP stack and add Websocket support to it. We'd add the
> necessary code, as well as tests, and push the patch upstream. In the
> meantime though, we would want our users to be able to use the feature
> even if the patch has not been addressed upstream or put into a new
> release of their library. Yes, this means that for a period, a user
> could be using something that has not been approved by the upstream.
> 
>>
>>> Unfortunately, I'm much more pessimistic in endeavours such as
>>> these.
>>>
>>> Asterisk, the SIP stacks, and distributions all have wildly
>>> different release cycles. Narrowing it down a bit,  the different
>>> distributions have wildly different release cycles. And heck, even
>>> the different distributions that Red Hat provides have different
>>> release cycles. If we're going to have people rely on distro-provided
>>> packages for the SIP stack, then that would mean having to compile a
>>> table somewhere that explains, "If you have issue X, then this is
>>> fixed in Asterisk version Y, but only if you are either using
>>> pjsip/resiprocate/sofia version Z or using libpjlib-dev on Ubuntu
>>> version A or higher or libpjlib-devel in Fedora version B or higher,
>>> etc. This is simply unmaintainable and will cause mass confusion.
>> You're confusing two different matters for the third time in your mail:
>> forking/embedding vs. pushing your changes to upstream is completely
>> orthogonal to what distributions, their packages and release schedules
>> do.
>>
>>> All of the SIP stacks have a well-defined API and ABI as it stands.
>>> The problem is that this only covers 90-95% of peoples' needs in most
>>> cases, so they add new API calls and change the ABI to suit what they
>>> need to do.
>> That's an oxymoron. This 5-10% of people's needs can be pushed back into
>> upstream's well-defined API/ABI. If everyone's doing it differently,
>> then it's not a well-defined API/ABI anymore.
>>
>> Or, let me put it this way: let's say you pick pjsip; you fork it, add
>> the 10% API calls that you're missing and other bugfixes of yours. What
>> stops you from releasing that as a separate tarball, that builds a
>> separate shared library called, say, libfoosip? Would it have a
>> well-defined API/ABI?
> 
> In my mind, there's a difference between a well-defined API and a
> complete API. A well-defined one is one that is documented properly and
> does what it claims to do. As such, it may not cover 100% of the needs
> of any given project. A complete API is one that is defined in such a
> way as to provide 100% of a domain's capability. For cases where a
> library's domain is small, it's quite possible to write a well-defined
> and complete API. For SIP, I argue that one could only write a
> well-defined API. A complete one would not be possible.
> 
> So to answer the question, yes, the API would be well-defined whether
> the changes were added or not.
>>
>>> In addition, SIP is rapidly evolving, meaning that new
>>> APIs are likely to be added to SIP stacks as new versions are
>>> released. Most libraries function in a domain that has a complete
>>> definition and has not changed from that initial definition. SIP, on
>>> the other hand, is constantly undergoing change, and its core RFCs
>>> are interpreted differently by different people. Trying to define a
>>> SIP library that is complete, suits everyone's needs, and is easy to
>>> use is just not possible.
>> I don't see how these changes can't be implemented in a shared library
>> as new API calls for everyone to use and profit from. More features in
>> SIP stacks, there for the next project to use, test and contribute
>> back to!
>>
>> And I think you're vastly overestimating SIP; SIP is indeed a complex
>> protocool but it's not more so than *everything else out there*. Just
>> look at the thousands of libraries present in your average distribution
>> and the protocols that they implement.
>>
>> Regards,
>> Faidon
> So here's the thing. I, personally, really don't want to embed a SIP
> stack with Asterisk if we don't have to. It makes the job of packagers
> difficult and it creates walled gardens rather than an open community as
> is the intent of open source software. I very much understand where you
> and Russell and others are coming from. As I stated at the beginning of
> my previous reply, we want to do this the "right" way.
> 
> What I posted in my previous reply were counterarguments for relying on
> upstream releases of a SIP stack and new releases of distro packages of
> the SIP stack (should the distro packages even exist) from our
> perspective as Asterisk developers. While my arguments in some cases
> were speculative, it's difficult to invalidate them since the situations
> I put forth are feasible and some have even arisen during Digium's
> dealings with certain third-party software providers. My intent with my
> previous reply was to show how relying solely on either of these
> channels is unworkable from the Asterisk project's perspective.
> 
> I don't want to argue semantics on the Internet because it doesn't move
> us any closer to choosing a SIP stack or determining how we use it.What
> I am interested in hearing is a third option. What I want is to find
> something that gives the Asterisk project the necessary control to be
> able to get fixes to users as quickly as possible, that encourages
> contribution from the Asterisk project to the upstream SIP stack, that
> does not place unnecessary burden on packagers of Asterisk, and that
> does not require users to go to unnecessary lengths in order to use the
> new SIP stack.
> Does anybody have ideas that will satisfy these requirements?


You need to look at the pattern that has emerged with pjsip not
absorbing patches and ask `will this pattern repeat for Digium patches?'

If you really want pjsip, but if you are not confident that they will
accept patches, then maybe round up some of the other people who wrote
patches (like for python-sipsimple) and make a fork together with all
the patches you want.  This new project, with a new name, could be
packaged up independently.

I don't think you will face this problem with resiprocate.  If Digium
was really interested, I think we would have a good hard look at moving
the project to github so you can have a git workflow for forking,
branching and submitting patches, and eventually you'd be committing
directly.

Another option that comes to mind is Sofia SIP.  That has been forked
heavily for another open source PBX.  Unfortunately, they are keeping
that forked code in their source tree and anyone who wants to build
official packages of their code is probably going to be stuck on that.
This seems like the very situation that Asterisk should avoid.  That
said, if Sofia SIP is a candidate, then you could also look at the same
strategy that I've suggested for pjsip: to fork the most up-to-date
version under a new name, as a standalone project, and hopefully
collaborate with other users to advance that project and have it
packaged independently.