[asterisk-dev] GMail Support

[Mark Michelson]

Marc Smith wrote:
> Hi,
> 

Hi!

Thanks for the detailed e-mail. My responses are in-line below

> Using imap.gmail.com (GMail / Google Apps) for the IMAP server has a
> few problems (app_voicemail w/ IMAP storage). I'm pretty sure these
> issues are more likely with Google's IMAP implementation, not c-client
> or asterisk; when I change the IMAP server setting to a dovecot IMAP
> server, it works perfectly.
> 
> UW IMAP version: imap-2007b
> Asterisk: 1.4.21.1
> Linux asterisk_test 2.6.23-hardened-r7 #2 SMP Thu Jul 10 10:35:52 EDT
> 2008 x86_64 Intel(R) Xeon(R) CPU X5355 @ 2.66GHz GenuineIntel
> GNU/Linux
> 
> 1) Out of the box, voicemail messages are saved correctly through
> IMAP. One of the problems arises when attempting to retrieve voicemail
> messages. Also, the MWI is never "active" -- asterisk reports no
> voicemail messages (old or new).
> The problem appears to be that when the mail_search_full function is
> called. The SEARCHPGM that contains the header criteria
> ("X-Asterisk-VM-Extension") is never matched. When that criteria is
> changed to something else, say the header "From" and the value being
> "Asterisk PBX" then I get some voicemails. Changing this in both the
> messagecount and open_mailbox functions will make it so my MWI updates
> correctly and also will play the voicemails correctly when I dial the
> voicemail app. I'm not sure why Google's IMAP does not return any
> matches when the search is performed. When I connect to imap.gmail.com
> with mtest and retrieve the message, I see the
> "X-Asterisk-VM-Extension" exists in the header (along with all the
> others). I turned on mail_debug for the mail stream and I see exactly
> what it issues to the IMAP server (along with its response):
> NOTICE[32142] app_voicemail.c: 00000007 SEARCH UNDELETED UNSEEN HEADER
> X-Asterisk-VM-Extension 20001
> NOTICE[32142] app_voicemail.c: * SEARCH
> NOTICE[32142] app_voicemail.c: 00000007 OK SEARCH completed (Success)
> 

Curiously, what are the flags for your messages when using GMail? I've read that 
GMail doesn't use the typical IMAP keywords, so perhaps there is some issue with 
words like UNDELETED or UNSEEN. This seems unlikely, though, since you said that 
if you change the header being searched for, you do retrieve messages. It may be 
worthwhile to explicitly mark new messages as UNDELETED and UNSEEN.

> 2) Next, I noticed after listening to voicemails (through my phone)
> the messages are not marked as "seen" so they still appear as new
> messages in the GMail web interface (and I'm sure through IMAP too). I
> believe when mail_fetchstructure is called in play_message, that
> normally sets the "seen" flag or the IMAP server sets the "seen" flag.
> This is not happening with Google Mail, so I used mail_setflag to
> "manually" set the flag after mail_fetchstructure is called. This
> fixed it so the messages are marked as read after listening to them.
> 
> I have a couple quick workarounds that solve these problems when using
> GMail with IMAP storage. I grabbed app_voicemail.c revision 129158
> from /branches/1.4/apps to work with.
> 
> 1) For this first problem with no results being returned from the
> search for voicemail messages, I just changed two lines:
> Line 2560 (function messagecount)
> Old: hdr = mail_newsearchheader ("X-Asterisk-VM-Extension", (char *)mailbox);
> New: hdr = mail_newsearchheader ("From", "Asterisk PBX");
> Line 4910 (function open_mailbox)
> Old: hdr = mail_newsearchheader ("X-Asterisk-VM-Extension", vmu->mailbox);
> New: hdr = mail_newsearchheader ("From", "Asterisk PBX");
> This is definitely not a good solution. With this, sharing a single
> IMAP mailbox for all users would be out of the question. Thats not my
> situation, so its getting me by for my testing. Ideas? If the problem
> is Google's IMAP, I doubt we're going to get them to change (at least
> not any time soon). It could be something else, but again, I don't
> have any problems when I switch to a dovecot IMAP server and I haven't
> heard anyone else having this problem.

You're right that your temporary solution is not a good one. I think you're 
right that it's just a GMail implementation problem. I tried some quick 
searching to see if there were known bugs about searching based on custom 
headers but didn't find anything. I'll keep searching though.
> 
> 2) This fix took took a couple more lines, but I think this could
> probably stay in without affecting other users; more testing needed to
> confirm.
> Starting after line 4590 (function play_message)
> Inserted this:
> char tmp[10];
> snprintf(tmp, sizeof(tmp), "%lu", vms->msgArray[vms->curmsg]);
> mail_setflag(vms->mailstream, tmp, "\\Seen");
> This then sets messages to read in GMail. Again, I'm not sure if this
> is caused by Google's IMAP or what. This may also be a bit neater
> (better?) in a function like its brother vm_imap_delete.
> 

I think explicitly marking the message as "seen" seems like a good idea to me. 
It's the type of thing that won't hurt for server implementations which 
automatically mark the message as seen and fixes GMail (and any other server 
implementations) which does not automatically mark the message as "seen."

> I'm set for my testing and I'll update if I run into anything else.
> What do you think about merging changes in so GMail users can be
> happy? Maybe even using some kind of 'gmail=yes' option in
> voicemail.conf to enable these "special" changes to app_voicemail.c.
> Thoughts? Ideas?

The idea of having to explicitly state the IMAP server in use (and furthermore 
have to code for each specific one's quirks) seems like a really bad idea to me. 
  It will make what is already ugly code much worse. If at all possible, it 
would be better to place in extra statements which should work for any IMAP 
server implementation, even though they may not be absolutely necessary on all. 
The search problem with GMail seems like a flat-out bug to me in GMail's 
implementation, and I have no idea how to get around that.

> 
> --Marc

Mark Michelson