That is correct; the only way to change quantity is for the buyer to manually edit the quantity in their cart.
If you want to set an initial default order size, say 2 units, you can add &quantity=2 to your Add to Cart button URL, so when the buyer clicks that button the first time, they'll start with 2 units of that item already in their cart.
You can also combine that with disabling "Let buyer edit quantity" in the product's settings, so continuing with the above example, clicking the Add to Cart button would add 2 units of that item to the cart, and the buyer would be unable to alter that quantity.