Debug: CPUmem dump lag spikes

[DandyDan]

Hi All,

I've been really enjoying this game recently and slowly adding mods as I discover things.

Problem I am having is that as I explore the map, my CPUmem raises from around 6000 until it creeps up to 10-12k. Once it gets too high, the game starts to stutter, then does a big freeze and returns back to baseline.

Can anyone explain what CPUmem really is, why it might be increasing, and why does it have to dump all at once instead of slowly as I play?

I've messed with all the settings and my key finding is that view distance really reduces the baseline, but not the issue of inflation and lag dumping. Gameplay is really stable if I am not travelling very far.

Thanks

 

Edit: After testing, i was not able to fix the issue. But i did find that the culinary artillery + expanded foods and the expanded matter + bricklayers sets are a major contributor. If i try to run with either set, it breaks performance.

I wonder if the .NET 7.0 version of the game works better with these mods, but most other mods don't seem to be updated right now. I know for sure that wildcraft is missing textures on the .NET 7.0 version.

Edit2: Actually the NET7 update works with wildcraft, the issue was that it just needs more space in the texture atlas. I'll continue to see if I can narrow down the problem to one mod.

Edit3: Found a correlation with the RAM optimization setting. Turning it to aggressive definitely increased the frequency of lag spikes on the NET7 version. Trying to figure out how to turn the setting off as there are only two options available: "aggressive" and "somewhat".

Edited July 15, 2023 by DandyDan

[DandyDan]

Ok, final update.

What i neglected to test in all my experiments was to let the game get well past initialization. If i create a new world and let it run for about 10minutes, the lag spikes are basically gone. It only spikes on occasion when i am sprinting to toward undiscovered chunks, and even then it doesnt always cause a spike. I've seen it dump 5-6GB without losing any frames.

Number and memory requirements of the mods loaded seems to have a big impact to baseline usage and how many spikes you will get, but i am able to load my full list with minimal issues after about 10min of settling time.

The NET7 update had a very noticeable improvement in performance as well. The memory management system seems to be way more reactive but stable at the same time. Also, baseline memory usage was like 50% improved over the standard version. 

Switching from single player to dedicated server would probably smooth out gameplay but i need to be able to pause the game at will.

[Streetwind]

"CPUmem" is basically what it says on the tin - memory allocated to the game processes on the CPU. In other words, main system memory. (As opposed to video memory, which is separate.)

The process you're seeing, where the memory usage creeps upwards and then suddenly resets to a lower number, is called garbage collection. It's at this stage that the CPU checks to see what parts of the stuff it has recently loaded into memory is still relevant, and dumps the rest in order to make room for more stuff going forward. Garbage collection can be done in different ways - rarely in big chunks, or continually in small bits, or anywhere on that spectrum. Which method is ideal depends both on your application as well as on the target system. Garbage collection takes CPU cycles and memory bandwidth, and doing a bigger chunk at once takes more of both - but doing it continually requires continually paying a smaller price. The "optimize RAM" setting likely changes the frequency of garbage collection, though I can't say for sure, nor do I know in detail what else it does (I'm not a dev).

If your system hangs during garbage collection, it's a sign that you're capped out on either CPU cycles, or memory bandwidth, or both at that moment.

From your posts, I deduce the following:

• Your computer probably has 16 GB of system memory installed. Windows tends to take around 4 for itself, which is why VS might go to 12 but no further. Though if "CPUmem" counts virtual memory instead of physical - meaning swapping to disk is included - then it's possible that you've only got 8 GB.
• Mods are a major source of memory consumption, and some are worse than others. You've named what are probably three of the top five memory hungry mods on the entire modDB. This will significantly increase the frequency and intensity of garbage collection by virtue of being faster to fill up all available memory and triggering a cleanup cycle through that.
• As your problems arise from terrain generation, and disappear after that, you are probably capped on CPU cycles at that time. Terrain generation is very CPU intensive. That changing the "optimize RAM" setting doesn't help further corrobates that - memory bandwidth is likely not what's limiting you.

 

...Or, you know, I might be wrong This is all guesswork, if educated guesswork. But maybe it helps.

[DandyDan]

Hi Streetwind,

Thanks a lot, it makes a lot of sense.

It seems that depending on the circumstances, having those mods installed increases the package size for garbage collection. Then, the CPU hits its limit while the collection process is ongoing and that stops the display or generation of new frames for a couple seconds (somehow).

Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz   3.60 GHz

16.0 GB RAM

RTX 2070 super

VS 1.18.6 NET7 on SSD

I did try the chunk load optimization setting at both ends of the spectrum with no improvement. However as I said the game runs without any real issue if I am not sprinting to generate new chunks.

[NukoSan]

Can the stutter issue be mitigated by chucking in another 16Gb or RAM? If that's the simplest fix, I am on board with it, can this be fixed by wither the VS creators or mod creators? Optimization is key!

[Jack Mackey]

3 hours ago, NukoSan said:

Can the stutter issue be mitigated by chucking in another 16Gb or RAM? If that's the simplest fix, I am on board with it, can this be fixed by wither the VS creators or mod creators? Optimization is key!

I added 64 GB trying to fix this (total of 96 GB) and it did nothing. The best fix is:

1) remove memory hungry mods like bricklayers and medieval fashion

2) run a local server and client combo instead of single player

Garbage collection seems to be blocking a thread/process for some reason (it shouldn't), so when garbage collection goes from 2-3 GB to 20+ GB, it takes longer and blocks longer, causing the big lag spikes.

[DandyDan]

Hi Guys,

Yes I also upgraded from 16GB to 64GB RAM and saw no big effect.

Removing Bricklayers had big impact.

One thing that I feel also really impacted both frames and lag spikes was changing the wildcraft settings to reduce the swamp plant spawn amounts down to 1-10% of the original values. Some plants like Tule and Cattails might be spawning 1500 tiles per iteration. I changed it to 15 and have a much smoother performance.

After all the changes i am seeing almost no lag spikes walking around, and maybe one lag spike per 30min when i open the map or the inventory. Especially opening the map / inventory the first time after loading the game, but not as much afterward.

Like Jack is saying, CPU seems to be the bottleneck and specifically the garbage collection process seems to be set up to run a massive load all at once instead of spreading that load out.

[l33tmaan]

That lines up with what I see when I have the task manager open while playing VS. I'll run around minding my own business until it caps out at round ~8-9 GB of memory before the game freezes for 10-15 seconds and it jumps back down to ~7 GB. 

No, I will not remove Bricklayers or the memory-hungry mods. It's too late for me to go back to vanilla.

[Branislav Kostic]

My game freezes like that with every Net 7.0 version, with or without mods, it fills up all of my 32GB RAM and starts to freeze periodically (2-3 seconds sometimes). It doesn't matter if I'm in already explored area or if I'm moving around, new game or older save it's all the same. Lowering some settings didn't give any effect other than taking it a bit longer to fill up my RAM, but once it does it just stays at 32/32 used for as long as I play. It takes about 15-20 seconds to free used memory after I exit to main menu.

My PC: i7 10700k, 32GB 3600MHz RAM, RTX 3080, Game is installed on m.2

I reverted back to 1.18.7 and game works like a charm