[Asterisk-video] RE: AVTF - What's going on?
John Martin
John.Martin at AuPix.com
Tue May 16 14:53:28 MST 2006
Hi All,
I know you're all a long way into video for Asterisk in some way or
other but I thought I'd outline what we're trying to do, what we've got
going and what we're still playing with. If everyone is prepared to show
and tell then perhaps we can figure out where we should be going and how
to get there... sorry if this ends up being a rather long winded mail.
So what are AuPix trying to achieve?
We have a number of customers that would like to use our video phones in
Asterisk but have been missing a way of doing the sort of things that
you would normally expect of a phone but with video tacked on. Our
phones support SIP and H.323 so that's what we've majored on. We have
customers doing regular telephony stuff and ones doing contact center
things, so we needed to address all of this in Asterisk.
So what'a working?:
Well we've been adding video things to SIP and H.323 since Asterisk
1.0.7. Initially it was simple stuff like assuming H.263, CIF, MPI=1,
that sort of thing, and building from there. In various Asterisk
releases we've found things get broken, (i.e since around 1.2.something
we found that chan->nativeformats got a bit more intelligent and so
broke a number of channel cap exchanges) so we fix and patch as we go.
As of today the following have been changed, tweaked, hacked or
generally man handled by us:
Chan_sip:
- maxcallbitrate: a basic way of telling an endpoint what bitrate it
should use (our videophones can do 2Mbps and so assuming such a rate for
a DSL connection was never going to make for great call quality, see
RTCP below). This should be superseded by a better capability mechanism.
- per peer videosupport: some carriers really barf when you send them a
video cap so we needed a way of allowing some endpoints to have video
support and some not (i.e. internal extensions have video but some
external trunks only audio).
- per peer INFO or RTCP fast update requests: again some switches don't
appreciate SIP info messages with XML in them as the Fast Update Request
mechanism, so we added a per peer support way of selecting between
sending SIP INFO with XML or by doing RTCP 192 (which is an old H.261
way of doing FUR) again see RTCP below.
- Process SDP: we have a simple way of extracting video fmtp lines from
SDP. It's a bit ugly and has little awareness of SDP scope at the moment
but it does extract fmtp for H.261, H.263 and H.264 into a common
ast_capabilities (see channel.c below).
- Build SDP: having extracted the fmtp lines and put them in a struct we
pass the struct around to the called endpoints (changes to the bridging
functions in channel.c) to limit their cap. We do a min() on confcap,
peercap and chancap to determine a maximum jointcap that an endpoint
will be allowed to generate once it gets an ack with SDP - don't get us
going on early media responses because it breaks some switches that
should know better.
- Video capability in SIP.conf: we use the h.261/3 fmtp definitions to
specify video capabilities in the conf file. i.e. you can specify lines
like:
H261=CIF=1 QCIF=2 CIF16=4 MaxBR=2024 framerate=25
H263=CIF=1 QCIF=2 CIF16=4 QVGA MaxBR=2024 A/B/C
Etc.
I know you wouldn't normally do an MPI and framerate but we support it.
We also allow for incorrectly formatted fmtp like some vendors are want
to do.
Chan_h323:
- Well we added video for a start, sounds like others have been having
similar fun too :-) . We also added the same maxcallbitrate and per peer
videosupport as for SIP.
- H.261, H.263 and nearly H.264 (I've got H.264 doing all the cap
exchanges and getting the media channels running, I'm just patching in
all the dynamic RTP stuff needed by Asterisk, h.323 is doing dynamicRTP
- I just need to let Asterisk know about it. If anyone else has got
H.264 going I'd love to see their approach).
- There's also the beginnings of the same fmtp like conf options for
H.323 in the conf file.
- Fast update requests received and transmitted.
- Fixes in H.323 for various Tandberg endpoints that don't like the way
chan_h323 used to do port reflection (this is a pain for NAT support but
most of our customers with H.323 seem to have stuck with DMZ or public
IP address as the work around for NAT on H.323).
- Video codecs are now broken out into their own files: ast_h26x.cpp and
the makefile fixed to allow the links to *.cxx to work.
- Logger support is also broken out into it's own file.
- We're currently using RFC2190 support for H.263 but I think we'll do
our own capability for that in the next few days.
Chan_local:
- it just needs a .write_video in the channel tech to get this working
(I think that was all, I'd have to do a diff to see if we fixed anything
else).
Chan_agent:
- similar to chan_local (.write_video) but I think there may have been a
bit more required.
App_echo:
- simply added a ast_indicate at the front of the app so that if you do
a playback or IVR before hitting app_echo then you don't get any
breakthrough on some endpoints.
Func_stringtoip:
- this is one of our own creations that allows a user to be registered
with chan_sip, or with a gatekeeper, and enter some arbitrary IP address
like 10*1*1*1, or 010001001001, to make a call to 10.1.1.1 (I know that
should be an internal IP address, replace with the public IP address of
your choice). This is useful when you're supporting local extensions
that want to talk to, for instance, lots of D-Link i2eyes that are all
essentially sitting in the DMZ of a CPE router and not registered. I
guess this could have been done in a set of dialplan apps but I'd rather
do in in C.
RTP.c:
Hmmmm.... lots of very good work was done on this before we shoved our
nose into it. We only added a few tweaks to RTP.c for RTCP but I think
it works now when video is involved too.
- RTP.c was seg faulting when the channel drivers shut down the RTP
layer but the RTCP timers were still going - no prizes for guessing that
the timer routine barfed when it tried to access the non-existent rtp
structs.
- Tidied up the parsing of the RTCP compound packets so that it worked.
- Generate compound RTCP packets (you should always send at least two
packets so we added a dummy SDES - it should really be filled in).
- FUR now generated from the channel drivers if they want it. Our
chan_sip decides whether to do it this way or with sip info. Some sort
of auto refelction should be possible but it may not always be reliable.
- FUR reported back to the channel drivers, thanks to Neil Stratford for
pointing out a blindingly obvious way of doing it that us mere mortals
had completely missed.
- Fixed up some report errors on seqno wraparound etc.
- A bit more of a troublesome problem for us has been seqno
resequencing. Asterisk puts it's own sequence numbers into outbound RTP
packets and ignores the inbound ones. This is generally fine for audio
but we found that the lumpy nature of high bandwidth video calls could
produce packet races (packets get swapped in transit to Asterisk)
through the internet but Ast would renumber the outbound packets. This,
of course, means that when the packet hits the target endpoint (having
passed through Asterisk) it's actually out of sequence but the endpoint
has no way of knowing that because all the seqno's are sequential
(unless there's been an Ast to target swap aswell!). We've added support
for seqno being added to ast_frame so that the RTP layer can use the
inbound seqno if it's available. This gets complex with IVR and other
file playback functions. Our solution can also cause large seqno jumps
when going from file playback to live video, which some endpoints will
crash on... the RFC says they shouldn't ;-) This fix isn't in Olle's
RTCP branch as I thought it might be a little too dangerous.
- RTCP report frequency in conf: The rtp.conf now supports an
rtcpinterval=x ms setting to set how often rtcp reports are generated.
The default 5000msecs isn't really enough to be handling video rate
control. I think the RFC recommends that 5% of packets should be RTCP
reports - that seemed a bit high to us, but sending one every 5 seconds
when you've sent over 3000 video packets in that time (0.03%) seemed too
low too :-)
- DTMF tones (not really a video one this, but I'm on a roll so bear
with me): I think there might be something very wrong with the DTMF tone
generation in Asterisk. We get a lot of complaints about it and have
tweaked what's there to improve things. We're waiting for Kevins's
variable length DTMF support 'cos I'm sure that'll be much better.
Channel.c/h
- We've come up with a struct to store all the new video caps in. It
looks like this but I'm sure there'll be lots of views on the way it
should be done and propagated around Asterisk (the codecnegociation
branch would maybe be a good start)
#define AST_VIDEO_SQCIF (1)
#define AST_VIDEO_QSIF (2)
#define AST_VIDEO_QCIF (3)
#define AST_VIDEO_QVGA (4)
#define AST_VIDEO_SIF (5)
#define AST_VIDEO_CIF (6)
#define AST_VIDEO_VGA (7)
#define AST_VIDEO_4CIF (8)
#define AST_VIDEO_SVGA (9)
#define AST_VIDEO_XGA (10)
#define AST_VIDEO_16CIF (11)
#define AST_VIDEO_LEVEL_BASELINE (1 << 0)
#define AST_VIDEO_LEVEL_MAIN (1 << 1)
#define AST_VIDEO_LEVEL_EXTENDED (1 << 2)
#define MAX_VIDEO_SIZES 11
/* The following array defines the video sizes we know about */
struct videoSize{
char* type;
unsigned int num_mbs;
unsigned int maxframterate;
};
static struct videoSize videoSizes[MAX_VIDEO_SIZES] = {
{"SQCIF", 48, 0},
{"QSIF", 75, 0},
{"QCIF", 99, 0},
{"QVGA", 300, 0},
{"SIF", 300, 0},
{"CIF", 396, 0},
{"VGA", 1200, 0},
{"CIF4", 1584, 0},
{"SVGA", 1875, 0},
{"XGA", 2304, 0},
{"CIF16", 6336, 0},
};
/*! \brief ast_video_cap: Struct holding video capabilities */ struct
ast_h2613_video_cap {
int valid;
/*!< 1 = This codec is specified, 0 = not specified */
unsigned int annexes;
/*!< Video annexes - stored as bitfield 2^(annex-ascii(A))*/
unsigned int maxframerates[MAX_VIDEO_SIZES]; /*!< Maximum
framerate (framerate = 30/mpi) */
unsigned int maxbr;
/*!< Maximum supported bitrate for this capability */
};
/*! \brief ast_h264_video_cap: Struct holding video capabilities */
struct ast_h264_video_cap {
unsigned int flags; /*!< Video flags
*/
unsigned int level; /*!< Profile
level */
unsigned int maxmbps; /*!< Maximum number of
macroblocks per second */
unsigned int minfs; /*!< Minimum
frame size in macroblocks */
unsigned int maxfs; /*!< Maximum
frame size in macroblocks */
unsigned int maxcpb; /*!< Maximum coded
picture buffer size in units of 1000 bits (1200 NAL HRD)*/
unsigned int maxdpb; /*!< Maximum decoded
picture buffer size in units of 1024 bytes */
unsigned int packet_mode; /*!< Packetisation mode */
unsigned int maxbr; /*!< Maximum
supported bitrate for this capability */
};
/*! \brief ast_video_capabilities: Struct holding all video capabilities
*/ struct ast_video_capabilities {
unsigned int maxbitrate; /*!< Max video bitrate */
struct ast_h2613_video_cap h261; /*!< H261 video capabilities
*/
struct ast_h2613_video_cap h263; /*!< H263 video capabilities
*/
struct ast_h264_video_cap h264; /*!< H264 video capabilities
*/ };
/*! \brief ast_video_capabilities: Struct holding all video capabilities
*/ struct ast_capabilities {
int maxcallbitrate;
/*!< Max allowed bitrate for this channel */
struct ast_video_capabilities videocaps; /*!< video
capabilities */ };
I think that's enough for now... if I've still got any of you with me
then perhaps the future of what we're up to can come another time....
John
John Martin
John.Martin at AuPix.com
http://www.AuPix.com
-----Original Message-----
From: Olle E Johansson [mailto:registry at webway.se]
Sent: 16 May 2006 10:13
To: asterisk-video at edvina.net
Subject: AVTF - What's going on?
This is stuff going on that I'm aware of:
* I am working on fixes for SIP/SDP in the sdpcleanup branch.
Grandstream and Foniris generously
donated video phones for me to test with. I am implementing some
checks if we really have
video on the incoming call before we offer it on the outbound
call. Seems stupid to offer
video without a stream. If we can fix this stuff and test it
quickly, I can get it into 1.4.
* John Martin contributed some patches for FMTP support that is
needed. Will open a branch
for this hopefully this week. This is too late for 1.4, but will
have to be part of 1.6. It will require
some architectural changes, and we're beyond freeze for that.
* John Martin is also doing work on H.323 and video
What else is going on codewise or should be added to the wish list?
/Olle
More information about the asterisk-video
mailing list