[Asterisk-video] lip sync issues with mp4_play

[Sergio Garcia Murillo]

Hi Dan,

Lip sync?? jeje.. good one.. ;)

Let's start with the bad news there is no audio/video sync mechanism in 
asterisk (as we comment in a previous mail), the good news are that if 
you have two synced media streams input in asterisk, the two media 
streams outputs are going to be synced.

So let's go for the source, app_mp4. As Emmanuel says in another mail, 
the hint track marks the timing for the rtp packets to be sent, so if 
the source i synced, app_mp4 will play it synced.
The bad news is that mp4 files are prepared for streaming, not for been 
transmitted on a fixed rate channel and played on a live, non-buffered 
player. So, what's the problem?
First, app_h324m does nothing with the sync, it just receives the 
packets from asterisk, extract the media and send it to the h324m 
library, so far so good.

The h324m library has a small jitter buffer, and it does not start 
sending until it has a minimum of data on it (3 packets if I remember 
correctly), so it introduces a small delay. Why? because if audio data 
is not sent with an almost perfect timing the mobile phone will suffer 
an audio under run and the audio will be crappy. But 3 packets are 60ms, 
so not much to worry.
When the jitter has enough data it starts sending audio at a fixed rate, 
and audio data will have preference over video, so the audio will be 
constantly played, no variable delays introduced.

Video is another issue, the biggest problem are the encoders. Most of 
them, ffmpeg at least, are not constant bitrate encoders. So they will 
produce a very big IFrame followed by small PFrames. What's the problem 
with that? The ISDN can't send faster, so a variable delay will be 
introduced on video depending on the size of the frame.

What happens then? Did you remember that we said that audio and video 
were RECEIVED synced on the app_h324m application? So when you sent the 
audio and video frame to the h324m library they are still synced, and 
even the h324m library starts sending it almost synced. But the amr 
arrives much more faster than the video, that depending on the size 
could take almost a second to be sent depending on the video bitrate.

What do mobile phones do? Usually a best effort approach, as soon as 
they receive audio, they play it. As soon as they receive the a video 
frame they decode and display it. That is the reason why sometimes a 
video plays faster than it should be for a while (several frames have 
been enqueued, been the latest very small, and they are received by the 
mobile phone almost at the same time).

What can be done? not much, buffering will introduce an unacceptable 
delay in a person to person communication, so the only good solution is 
to be ale to have a CBR h263 encoder that outputs always the same amount 
of data for each frame. That way a fixed delay will be introduced in 
video and by adjusting the jitter the synchronization we could be almost 
perfect.

Best regards!
Sergio

Dan Julius escribió:
> Hi,
>  
> Is there a good reference to better understand how audio/video 
> synchronization is handled in h324m?
>  
> I'm using mp4_play to play a file I created following the mp4 mini 
> tutorial. I combined an h263 track with a g711 track,
> I've noticed that by modifying the bitrate of the video channel I can 
> get different results, but all of them have lip-sync issues.
>  
> where is the av sync supposed to be handled, in the mp4 file, by 
> libh324m, or the 3g handset?
>  
> Thanks for the advice,
> Dan
>  
> ------------------------------------------------------------------------
>
> _______________________________________________
> --Bandwidth and Colocation Provided by http://www.api-digital.com--
>
> asterisk-video mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-video

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.digium.com/pipermail/asterisk-video/attachments/20081111/8098107b/attachment-0001.htm