[asterisk-users] strange SIP-SIP delay

[Julian Lyndon-Smith]

I've got the following setup:

PhoneA ->
  router ->
   vpn ->
    router->
     asterisk (SIP / ISDN)

PhoneB ->
  asterisk (SIP / ISDN)

If phone A is connected to phone B (sip-sip), there is a noticable delay 
(up to 2-3 seconds) between me speaking and the other end hearing.

If phone A calls out via the ISDN and back in  to the DDI of phone B (ie 
SIP->ISDN->ISDN->SIP) then there is no delay at all !

Any clues on where I might start looking for this ?

Julian