Hi, I am the friend mentioned by OP. The host is on a PC running Arch Linux. It has a Ryzen 7 3700X CPU (8 cores, 16 threads ), 32 GB of RAM running at 3200 Mhz and an RTX 3070 Ti. So it should be capable of running both the server and a client at the same time. I am running the host using the script from this guide: https://wiki.vintagestory.at/Guide:Dedicated_Server#Dedicated_server_on_Linux
The same PC also runs my client at the same time. I installed Vintage Story through this AUR package: https://aur.archlinux.org/packages/vintagestory. I don't think that will make much of a difference but I thought it would be good to mention all factors here.
When it comes to my networking, my internet connection is a 400 Mbit/s up/down fiber connection and my local network is handled by a UniFi modem and switch. The switch connects directly to the modem and my PC is hooked up to the switch by a CAT6 cable. I use mostly default configurations on my UniFi devices (other than the port forwarding to be able to host the server and binding a static IP to my PC). I don't run any network wide ad blockers or firewalls.
I used to have the most issues when looking at the map, especially when zooming out. After that nothing would work properly until relogging. This issue appears to be mostly fixed now with the optimizations done in 1.20.11. The issue now seems mostly related to chunk loading. Exploring the world (especially while riding an elk) eventually just seems to overload the game so much that doing something like breaking a block either does nothing at all or takes a good 10 to 20 seconds. I have already looked at both server and client logs to see if a cause can be found there but I don't see anything off in there. I have attached the logs from a test session I just did (the issue happens if it's just me connected as well). I hope this helps.
Thanks for looking it and for the tips so far!
server-debug.txt server-main.txt client-debug.txt client-main.txt