You would simply omit any values you aren't using, but retain the commas separating one value from the next on each line, like so:
SKU004a,16.00,,,Blue,M $16,
SKU004b,18.00,,,Blue,L $18,
SKU004c,20.00,,,Blue,XL $20,
SKU004d,22.00,,,Blue,2XL $22,
SKU004e,16.00,,,White,M $16,
SKU004f,18.00,,,White,L $18,
SKU004g,20.00,,,White,XL $20,
SKU004h,22.00,,,White,2XL $22,
Unless you need those SKUs to match an item-numbering system you're already using in-house, you could make those a bit more descriptive as well, like so:
004-BM,16.00,,,Blue,M $16,
004-BL,18.00,,,Blue,L $18,
004-BXL,20.00,,,Blue,XL $20,
004-B2XL,22.00,,,Blue,2XL $22,
004-WM,16.00,,,White,M $16,
004-WL,18.00,,,White,L $18,
004-WXL,20.00,,,White,XL $20,
004-W2XL,22.00,,,White,2XL $22,
