Your temporal stability is going down for some other reason, are you eating? Being close to rifts also causes temporal stability loss. If an area is temporally unstable the stability will not go up at all unless you leave or restore it with other means, meaning that the area is not the issue unless you have a weird mod or weird configs.