I wasn't aware of that, you're using the "Tax everyone" option instead of taxing by destination then?
I'll bring this up to our development team to add to the wishlist as it really isn't necessary to collect the buyer's location for tax in your case. It shouldn't hurt anything but it is always better to keep things simple and not clutter up the cart with unneeded options.