I hate to say this, but I think this is something you will have to live with, since most, if not all of us experience the same thing. What I consciously do is to wait the 1 to 2 seconds once I answer an incoming call to say hello or say hello twice before the calling party hears me and respond.
IMHO, this might just be a minor issue with using GV in a way it was not intended.