We haven't seen any such problems from our end, but this thought experiment might fill-in the big picture some:
Picture a V shape, with your personal computer at the angle; one arm of the V might be very short and fairly direct (from your computer and ISP to your main site's server), whilst the other arm might be quite long and convoluted (from your computer/ISP to our servers) and having more opportunity for temporary or persistent interference or bottlenecks.
Now, consider that, while the above may be the case for your computer using your ISP, it would be different for any other computer and ISP; for instance, the V-arm from our office to our servers' datacenter is quite short and direct indeed, whilst the other V-arm from us to any given Merchant's site would likely be longer and more complex.
In other words, what you experience is largely a function of the potential differences in connectivity between any single node with regard to any two other nodes on the Internet; those differences are particular to the perspective as seen from your node, are not necessarily indicative of a general case that everyone else would experience the same way, and can change over time as Internet traffic and infrastructure changes.
All that said, you might look into this thread here about using your own button images, or even text links, to minimize the elements your site must draw from our server:
http://www.e-junkie.com/bb/topic/2495#post6045
