Drifters are chunky guys and can just barely waddle, if you're overwhelmed only by then, just run. Their rocks used to be precise like sniper rifles, but not anymore, so at a distance a bit of moving around and they miss.
You had a gap somewhere, it happened to have spawn there(bad luck, but can happen with how spawns are handled, in this case you'll have to move your pen), there was a high enough block pile near for it to accidentally jump over or a modded block that acted as a gap. Critters will not path through fences, even if they should be able to jump over.
For a clear and humorous example, consider this bear I have trapped in a fence square that is a block below it and has a top just showing above and surrounded by delicious rabbits. For reference, brown bears can jump three blocks high.