Actually, if you just don't worry about it, it has little to no impact on usage.
When you listen to a voicemail message over the phone, the system marks it as "read" (well, of course it is marking it as "listened-to" over the phone, or "read" on a screen). Once you listen to the message, it will no longer be flagged as a message waiting for you to check (the next time you call VM, it will say you have no messages). You can "archive" the message over the phone, or leave it as a read-item in your inbox. Since the archive is unlimited in size, you can either take no action, or, if it's some sort of personal/confidential/sensitive message, go into the UI and delete it.
Over the years, very few people have complained about this.
The best source of information about Google Voice is on the official, Google-owned Google Voice help forum - the link is in my signature below.