Presumably your T-shirts will ship for the same rates as your books? If so, then in the settings of each product, enable Shipping/Buyer's Address, and in every product's Shipping Options, set Weight = 0, Package Type = Regular Envelope, and Capacity = 3.
Now, to calculate shipping charges as you described, we're going to "game the formula" of (rate X weight) to calculate a flat rate for each package of up to 3 items, rather than each individual item.
Go to Seller Admin > Cart Shipping Settings and set up the following:
Shipping Origin: select United States and enter your ZIP code;
Shipping Destinations: if you will only ship to certain countries, select them in the list, or otherwise select nothing for Worldwide shipping;
Define Packaging:
- Regular Envelope: Cost = 0.00, Weight = 1.00;
Leave all other settings blank or 0.00;
Click Next to save settings and proceed to the Shipping Rules screen.
In the Existing Shipping Rules menu, select and Delete any Rules you may have saved there.
Set up your first Shipping Rule for Domestic shipping:
Shipping Rate Calculator: Price Per Unit Weight;
Shipping Amount: 5.00;
Country: United States;
Click Submit to save the Rule.
Next rule for shipping to Canada:
Shipping Rate Calculator: Price Per Unit Weight;
Shipping Amount: 12.95;
Country: Canada;
Click Submit to save the Rule.
Last rule for shipping to other International destinations:
Shipping Rate Calculator: Price Per Unit Weight;
Shipping Amount: 16.95;
Country: All Countries (this covers the rest of your Shipping Destination countries that don't have a Rule of their own);
Click Submit to save the Rule.
Click Back to Admin, and you're done!