I consider rifts to be a mild form of noise pollution more than anything else. The game's ambient sound is very immersive and attractive to me when there are no rifts around. Perhaps if you improved their sound spam somewhat then they would be less annoying.
Honestly, I find rifts to be annoying just in general, same as drifters. It's more of an 'oh great, here we go again' than a credible threat in the game world.