Came to this thread with the same issue - Working fully after editing the screenWidth and screenHeight in clientsettings.json with the correct resolution for my monitor, enabling the "layered on DXGI swapchain" setting for vintagestory above, and setting the video settings ingame to Fullscreen. Bit of a dance, but this fixed alt-tabbing and Window's HDR color settings going bad for me.