Seems a known problem for years. One forum has suggested using 100% wool felt in black as wool absorbs light but does not reelect it,

Well have you seen a shiny sheep.

Ordered a 1mm and 4mm square to test.