Aside from disabling "Let buyer edit quantity" in each product's settings (so buyers could only ever order one of each item at a time), we really don't have a way to limit whole-cart quantity to some total maximum, although we could suggest a way to offer limited quantities for each item, if that could work for you.
About all we could suggest for your fee quotes on large-order shipping would be to configure free shipping for orders above a certain total amount or quantity, though you'd want to be clear to buyers that you will charge them separately for shipping and allow them to cancel the order for a full refund if they don't accept the shipping rate you quote.