I'm sure this has been suggested before, I know there's a mod for it, it just seems odd this isn't in the game.
Sand is used in a lot of my favorite building materials and hunting it down in large quantities is difficult, particularly sand that isn't sandstone sand.
My suggestion is to then either add a new mechanical device or use the existing ones to first pulverize the stones of a respective material into gravel and then to grind it in a quern into sand.
Alternatively a new mechanical device to be hooked up to a power source like a tumbler to polish gems as well as break down stones into either gravel, sand or both with enough time.