Yeah, that's just how it works.
Specific colors of daub are made by combining specific types of sand and clay (usually with 1-2 options for each type) or by dying it. This means that some combinations just don't come together into anything meaningful.
I was in the same boat: nothing but red clay and granite. Fortunately, granite can be used to make a quern so I just made fire clay manually using calcined flint: the conversion rate is generous enough that just grinding the flint manually is the most laborious part of the process.
