...adding to drgeoff's explanation, and specific to Google Voice: you cannot set caller ID name. Your description of doing something with caller ID name while the numbers were with another service provider rarely works. Generally, CNAM may be reset to nothing when a number is ported, or the old CNAM may remain, but this is entirely up to the CNAM database (LIDB) providers. Your ported numbers may retain their old CNAM information for some undetermined time, and then suddenly their CNAM LIDB entries may be deleted. They will not respond to requests from end users; only authorized service providers can do that.
Aside from that, your plan to have multiple people call out or receive calls on the same Google Voice number is not going to work. AFAIK, you can only have two simultaneous calls (channels) at a time.
Whatever modest amount of money you may be saving by using Google Voice is easily offset by these limitations. Use a paid commercial SIP VoIP Internet Telephone Service Provider (ITSP) instead of Google Voice. You'll gain control of CNAM, be able to set up as many extensions as you need, configure them for either simultaneous ringing or a hunt group, set up an IVR, etc.