[asterisk-dev] Asterisk 14 - Remote URI Playback

[Matthew Jordan]

On Tue, Nov 4, 2014 at 12:57 PM, BJ Weschke <bweschke at btwtech.com> wrote:
>  Matt -
>
>  This is a pretty neat idea, indeed, but I've got some questions/thoughts on
> implementation. :-)   Apologies if all of this was already
> considered/accounted for already..
>
>  1) Does the entire file need to be downloaded and in place on the HTTP
> Media Cache before you can call an ast_openstream on it? This could cause
> some problems with larger files not sitting on a fat pipe local to the
> Asterisk instance.

It does need to be completely on the local file system, which would be
a problem for extremely large files and/or slow network connections.

The ability to do an 'asynchronous' version of this is not really
present. The filestream code in the core of Asterisk doesn't have
anything present that would allow it to buffer the file partially
before playing back with some expected max size. If we went down that
road, it'd almost be a completely separate filestream concept from
what we have today, which is pretty non-trivial.

I don't think I have a good solution for really large files just yet.
There's some ways to do this using cURL (where we get back a chunk of
binary data, buffer it, and immediately start turning it into frames
for a channel) - but that feels like it would need a lot of work,
since we'd be essentially creating a new remote filestream type.

>  2) What kind of locking is in place on the design to prevent HTTP Media
> Cache from trying to update an expired resource that's already in the middle
> of being streamed to a channel?

Items in the cache are reference counted, so if something is using an
item in the cache while the cache is being purged, that is safely
handled. The buckets API (which is based on sorcery) assumes a 'if
you're using it, you can hold it safely while something else swaps it
out' model of management - so it is safe to update the entry in the
cache with something new while something else uses the old cached
entry. The 'local file name' associated with the URI would be created
with mkstemp, so the risk of collision with local file names is low.

In the same fashion, a local file that is currently open and being
streamed has a reference associated with it in the OS. Calling unlink
on it will not cause the file to be disposed of until it is released.

>  3) I think you need to also introduce a PUT method on HTTP Media Cache
> because I can think of a bunch of scenarios where having a write operation
> on func_curl may be lacking in the file needing to be retrieved (eg - trying
> to pull ACL'd media from an S3 volume where you need custom HTTP request
> headers, etc). We shouldn't try to architect/design for all of these
> scenarios in Asterisk via a write operation on func_curl and a PUT to HTTP
> Media Cache seems like a reasonable approach to handle that.
>

I had thought about this, but didn't have a strong use case for it -
thanks for providing one!

How about something like:

GET /media_cache - retrieve a List of [Sound] in the cache
PUT /media_cache (note: would need to have parameters passed in the body)
    uri=URI to retrieve the media from
    headers=JSON list of key/value pairs to pass with the uri
DELETE /media_cache?uri
    uri=URI to remove from the cache

Sounds data model would be updated with something like the following:
  "uri": {
       "required": false,
       "description": "If retrieved from a remote source, the
originating URI of the sound",
       "type": "string"
   },
   "local_timestamp": {
       "required": false,
       "description": "Creation timestamp of the sound on the local system",
       "type": "datetime"
   },
   "remote_timestamp": {
        "required": false,
        "description": "Creation timestamp of the sound as known by
the remote system (if remote)",
        "type": "datetime"
   }


-- 
Matthew Jordan
Digium, Inc. | Engineering Manager
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org