Nobody likes having their stuff destroyed, and yet the most effective way to stop an onslaught is a few blocks of soil. I think this is fine -- good even, to keep as an option (possibly even the *default* option given peoples' friction with storms).
My idea is that, during temporal storms, the besieging enemies could be capable of damaging blocks they deem "in the way", for the duration of the storm, making them fracture or become glitchy in some new way, and allowing them passage through otherwise impassible barricades. More durable materials would prove more resilient.
At the end of a storm, any temporally shattered blocks would return to their normal, undamaged state.