The tower is there to allow drawing in bowtorns, in the case of a drifter/shiver getting in it probably would be a good hidey place.
The gap between the outside of the pit and fences hasn't caused any issues for me, shivers and drifters seem to enjoy dropping in, though as seen in the video bowtorns, aren't too thrilled with it (which is why there are trapdoors in the corners, to make them think they can cut the corner).
The gap between the fence and the inside of the farm is also fine, though stones can get through. It's not very common. Items can't fall through because their hitbox is too large for the gap, that's why your farm isn't losing items to the gap.
As for bowtorn projectiles, in previous testing on a different varient I had bowtorns shooting through the walls. I suspect the hitbox of their projectile is too small and occasionally can pass through the fence. Walls won't make a difference there since they're the same thickness. But bowtorn projectiles shouldn't be an issue as the bowtorns will hold off fire untill they have a clean line of fire. One thing to be careful of is their stab attack. If you get too close, they can stab through the walls. That's why in the video I stand on the inner edge of the path.