I have similar issues running an i7-6 core with RTX-2070 8GB, 32GB ram (2x16), installed on fast SSD... the game is almost unplayable and increasingly annoying in intense moments. Running at 45fps and minimal settings makes no difference. I do not have these issues on other high spec games... maybe a few fames dropped here or there but nothing that interferes with gameplay. VS will freeze for a full 2-3 seconds at times before catching up. Frames dropping several times a minute. Seems to be a common issue. Have tried every suggestion I can find online as far as settings, nothing seems to make a difference. This is on vanilla with no mods, singleplayer.
The only fix has been to run a server on a second PC, so the hardware duties are split, which has its own undesirable annoyances for essentially single player game.