[Big-ish server, server software looks to be the cause, please help]

 

First of all I agree with you that the code could use some optimization. That being said I'm not sure how complex the server codebase is as it stands.]

 

1. We were already having performance issues with sectors at 5, and as we upped it to 7, and then later to 12, the problems stayed about the same. I only upped it because our player base was requesting it, and we observed no change in how bad the server was performing as we gradually upped it.

That's interesting that you didn't see any performance hits, did you notice anything else in particular? Hard limits, etc.? Just curious for myself.

 

2. ZFS is on a dedicated NAS, not local to the hypervisor itself. FreeNAS is running the ZFS zpool, so yeah.

That still doesn't sound necessarily great to me. The fact that it's on a NAS means that it's susceptible to more efficiency problems, though I have to take your word on that you're pretty confident that's not causing any issues.

 

3. I take it you didn't read the model of CPUs we're using. I guarantee it has HT ;)

Nice one, I've definitely seen harshly diminishing returns or even overloading happening with too many threads or unbalanced configs.

 

4. We had a much higher worker count, but saw zero benefit from it. When trying to profile it and look at the worker thread graphics that were generated, a lot of worker threads were doing nothing, so we lowered the count in an effort to increase efficiency. Either way, nothing changed.

5. At this point, I am 100% confident this is bad code, and I'm very optimistic about the next patch helping, as the beta patch notes already talk about server performance fixes.

I see the same thing, it's pretty frustrating not being able to parallelize it when it seems like such an easy candidate for any sort of sharding. Keep in mind that the extra workers because threaded will also end up hogging iowait time depending on the implementation, but I think you're definitely hitting the nail on the head with the fact that at the end of the day there seems to be maybe one or so threads per user.

 

6. Also, I turned profiling off to try and improve performance and reduce spam to the console (so I could actually read the console). Not sure why you're recommending it to be on in a situation that involves poor performnace.

7. I had already, multiple times, referenced the link you mention, as we were building the server.

I wasn't sure how much you had looked into  your proc and your running system. Those things can be far more complex than they let on to be *especially* if you're virtualizing which I'm assuming you're not. Also sorry about the link I didn't see it noticed, the messages were pretty dense and just looking at the numbers on your config I've seen similar issues with the same types of ratios that get exacerbated by fiddling with more settings.

[server.ini & worker threads]

In my experience the problem stems from a combination between the ratio between

workerThreads

and

generatorThreads/scriptBackgroundThreads

. They seem to recommend a 4:1 or 3:1 ratio and they don't seem to recommend anything above 3 or 4 generatorThreads even with a ton of cores.

[Avorion Dedicated Server Linux - Cash after Player(admin)-Game Quit/Logout]

Is that all you have of the logs? It seems like whatever happened it decided to exit higher up from where you started posting but I could be wrong.

[Big-ish server, server software looks to be the cause, please help]

So, curious situation, increased "aliveSectorsPerPlayer" to 12 (was 7), decreased "workerThreads" to 16 (was 32) & "generatorThreads" to 8 (was 16), leaving "scriptBackgroundThreads" at 16. And now the server seems to be running better.

 

In my experience ,12 is a bit overkill for aliveSectorsPerPlayer for most servers, you may want to cap closer to  8 or 10 as it seems to have an slowing effect on the worker threads the more you add both.

 

You probably want to start by increasing the ratio between worker threads and generator/script threads as per the docs  (https://avorion.gamepedia.com/Setting_up_a_server#Examples_for_different_machines:).

 

It seems like every person I've talked to has has similar issues when they don't stick to the 4:1 - 3:1 ratio the developers suggest. Also verify that your processor supports hyperthreading if you're going to exceed the number of threads per core or you'll run into more issues.

 

As a final note on the matter of ZFS, assuming you're using linux it's good to note that ZFS is not natively supported on linux, and has a rather unique read/write architecture. I'm not going to say that it's also compounding the issues but I will say that you may want to work with a more classic/predictable filesystem before you have these issues ironed out, or at the very least configure your ZFS cluster in an extremely vanilla manner. Just a thought!

 

Also here's a little example of how I might configure your server (having used much weaker boxes).

 

16x Cores

24GB Memory

saveInterval=600
sectorUpdateTimeLimit=300
emptySectorUpdateInterval=2
workerThreads=16
generatorThreads=4
scriptBackgroundThreads=3
aliveSectorsPerPlayer=8
weakUpdate=true
profiling=true
sendCrashReports=true
hangDetection=true
backups=true
backupsPath=
simulateHighLoadServer=false
sendSectorDelay=2
placeInShipOnDeathDelay=7