You would need to include the fabric choices in your Variants configuration, rather then putting them as Variants, and specify the stock for each possible combination of size and fabric. E.g., you would have Variants config lines something like this, where Option1Name is "Fabric" and Option2Name is "Size":
SampleSKU RS,58.00,0.00,6,Red,Small,
SampleSKU RL,52.00,0.00,6,Red,Large,
SampleSKU BS,51.00,0.00,6,Blue,Small,
SampleSKU BL,53.00,0.00,6,Blue,Large,