The way I understand Upstream should be close to 100% of your upload, and restricted should be ~100Kbps per simultaneous call that you are expecting (although at 640Kbps available, reserving 400KBps for calls might be an overkill - 200Kbps seems more reasonable).
For sure, your download speed doesn't apply. From the manual:
QoS only applies to upstream traffic to the WAN side