An update on this issue: It seems to be an echo cancellation problem. My setup is as following: FXO -> IAX2 server 1 - IAX2 server 2 <- FXO 1st call has a terrible echo. If I start server 2 and immediately issue zap destroy channel X (one of the 4 fxos), than the call goes through without a problem.