The Shipping/Buyer's Address setting is what triggers collection of a physical address during checkout, so if an order contains no items with that setting enabled, then no address would be collected.
If you want to enable that setting to collect an address for some products without ever adding any actual shipping charges for those products, then yes, you would need to remove your Handling fee and figure out a different approach to add that shipping amount.
It sounds like you could set all your tangible products to ship in the same packing container type, with some absurdly high packing capacity like 9999, then in Cart Shipping Settings define the Cost of that container type as 3.95 (instead of specifying that amount as Handling).
For the digital items where you want to collect an address without any shipping fees, just enable Shipping/Buyer's Address but configure them with Weight of 0.00 and select a different container type that has a Weight and Cost of 0.00 in Cart Shipping Settings.