I did some further experimenting, and this only occurs if a call is forwarded through one of the SIP slots, but does not occur if GV is dialed directly from the phone port.
Example:
SP1 configured to a SIP provider, X_InboundCallRoute = {(1xxxxxxxxxx):sp2},{ph}
SP2 configured for Google Voice
Pick up the phone, dial **2xxxxxxxxxx - everything works fine, even if the remote party does not answer the phone before 30 seconds.
Send a call to 12223334444 through SP1 - if the remote party does not answer within 30 seconds, there is no audio.