Agent-Piano Posted April 22, 2020 Share Posted April 22, 2020 Hello! I've been running a dedicated server for about one week that is averaging between 5-10 clients. In the last few days we started seeing issues where client "ping" goes to about 5s. This is then resolved by restarting the game server. I have been able to pin down when this starts by way of CPU performance charting for the system. Using those timestamps I've stepped back through server logs in attempt to identify any trace data. With profiling enabled, I'm seeing this entries which definitely "look" pretty bad. Thoughts? Also, the log file is too large for pastebin, alt. upload link below... https://gofile.io/?c=5oQHuB Server Hardware Windows Server 2019 VMware Hardware V11 6 vCPU (3 Sockets x 2 Cores) - 14000MHz Reservation, High Share Priority 16GB Memory SSD Based Storage VMware Host HP Proliant DL360 G7 - ESXi 6.0 15517548 2 x Intel Xeon E5620 @ 2.4GHz (16 Logical) - (Aprox 25% utilization) 120GB Memory (Aprox 40% utilization) 2020-04-22 13-18-07| saving sector (177:41) 2020-04-22 13-18-07| Player kevin moved to sector (119:219)from (119:220) server time taken for change: 0ms S2020-04-22 13-18-07e| ctoSrSeDecactttooarrbDDaaattsaaebb:aas seeL:: oLLaoodaaiddniignn ggS eSSceetccottroo rr( (1(1112720:0:2:1229021)3) 2020-04-22 13-18-07| )2020-04-22 13-18-07| 2020-04-22 13-18-08| sector (177:41) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\177_41" 2020-04-22 13-18-09| saving sector (177:35) 2020-04-22 13-21-13| saving sector (108:175) 2020-04-22 13-21-14| sector (108:175) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\108_175" 2020-04-22 13-21-14| Player Hewitt moved to sector (178:38)from (161:81) server time taken for change: 124ms SectorDatabase: Loading Sec2020-04-22 13-21-14to| r (S17e7c:t41o)r 2020-04-22 13-21-14| Database: Loading Sector (177:35) 2020-04-22 13-21-14| scheduled save for sector (178:38), 0x3859f7d0, entities: 579 2020-04-22 13-21-14| saving sector (178:38) 2020-04-22 13-24-58| scheduled save for sector (280:384), 0x3dea5410, entities: 1275 2020-04-22 13-24-58| saving sector (280:384) 2020-04-22 13-24-59| Player kevin moved to sector (120:191)from (120:190) server time taken for change: 0ms Sect2020-04-22 13-24-59o| rDSaetcatboarsDeat:a bLasoea:d Lionagdi nSge cSteocrt or( 1(2132:11:8189)1 2020-04-22 13-24-59| ) 2020-04-22 13-24-59| sector (280:384) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\280_384" 2020-04-22 13-24-59| scheduled save for sector (293:340), 0x7d699470, entities: 1579 2020-04-22 13-24-59| saving sector (293:340) 2020-04-22 13-24-59| Server frame took over 1 second. 2020-04-22 13-25-57| Sector[120:191]::buildUpdateChain: 0.0008684 2020-04-22 13-25-57| Sector[113:174] deferred update Hangar t_0 elements: 10: 0.0008665 2020-04-22 13-25-57| Sector[119:219]::buildUpdateChain: 0.0008634 2020-04-22 13-25-58| Player Handalaan moved to sector (219:79)from (223:71) server time taken for change: 0ms SectorDa2020-04-22 13-25-58t| abaSsee:c tLooraDdaintga bSaecsteo:r (Lo2a2d0i:n7g8 )S2020-04-22 13-25-58| ecto r (218:88) 2020-04-22 13-25-58| scheduled save for sector (219:79), 0x7d69ef10, entities: 1808 2020-04-22 13-25-59| saving sector (219:79) 2020-04-22 13-25-59| Server frame took over 1 second. 2020-04-22 13-25-59| [ void Server::update(float): 1178.5 ms; 100% ] 2020-04-22 13-29-24| scheduled save for sector (117:227), 0xfdf8caf0, entities: 732 2020-04-22 13-29-24| saving sector (117:227) 2020-04-22 13-29-25| Player kevin moved to sector (110:228)from (117:220) server time taken for change: 0ms Se2020-04-22 13-29-25| ctSeoctroDrDaattaababsea: sLoead:in gL Soecatodr i(n11g3: 2S26e)c 2020-04-22 13-29-25| tor (106:227) 2020-04-22 13-29-25| sector (117:227) saved to "D:\SteamDDSuite\Build\Active\save\BORED_Space_Bois\sectors\117_227" 2020-04-22 13-29-25| scheduled save for sector (119:219), 0xc2f8a530, entities: 447 2020-04-22 13-29-25| saving sector (119:219) I have also included my server.ini for review in case that helps you guys out. [Game] Seed=O3sWV4jQNd Difficulty=0 HardcoreEnabled=false InfiniteResources=false CollisionDamage=1 SafePlayerInput=false PlayerToPlayerDamage=true LogoutInvincibility=true LogoutInvincibilityDelay=30 DevMode=false ExplicitCallables=true BigWreckageDespawnTime=1800 SmallWreckageDespawnTime=900 MaximumFightersPerSectorAndPlayer=-1 MaximumBlocksPerCraft=-1 MaximumVolumePerShip=-1 MaximumVolumePerStation=-1 MaximumPlayerShips=-1 MaximumPlayerStations=-1 MaximumBlocksPerTurret=250 PlayerInventorySlots=1000 AllianceInventorySlots=1000 Version=1.0 sameStartSector=true startUpScript=data/scripts/server/server.lua startSectorScript=startsector [system] saveInterval=300 sectorUpdateTimeLimit=600 emptySectorUpdateInterval=0.5 workerThreads=32 generatorThreads=16 scriptBackgroundThreads=8 aliveSectorsPerPlayer=10 weakUpdate=true profiling=true sendCrashReports=true hangDetection=true backups=true backupsPath= simulateHighLoadServer=false sendSectorDelay=2 placeInShipOnDeathDelay=7 [Networking] port=27322 broadcastInterval=5 isPublic=true isListed=true isAuthenticated=true sendStatsToAdmins=true useSteam=true rconIp= rconPassword=PASSWORD rconPort=27323 [Administration] maxPlayers=69 name=SERVERNAME description=SERVERDESCRIPTION password=PASSWORD pausable=false accessListMode=Blacklist steamIdOverride=0 Thanks in advance for any support. Happy to get additional details around the issue to help work through it! Link to comment Share on other sites More sharing options...
Xeigho Posted April 23, 2020 Share Posted April 23, 2020 Im also seeing this issue. The ping spikes are a pretty noticeable 2-3 seconds Again with garbled sector database lines; 2020-04-23 18-52-01| Player Xeigho moved to sector (39:462)from (90:178) server time taken for change: 58ms SectorDatabas2020-04-23 18-52-01e| : SLeocatdoirnDga tSaebcatsoer: (L4o0a:d4i6n3g) 2020-04-23 18-52-01| Sector (38:461) 2020-04-23 18-52-01| scheduled save for sector (39:462), 0x7f88dc001d20, entities: 560 2020-04-23 18-52-01| saving sector (39:462) 2020-04-23 18-52-01| sector (39:462) saved to "/home/steam/avion/galaxies/xeigho/sectors/39_462" 2020-04-23 18-52-01| scheduled save for sector (90:178), 0x7f88dfda66f0, entities: 1935 2020-04-23 18-52-01| saving sector (90:178) 2020-04-23 18-52-02| sector (90:178) saved to "/home/steam/avion/galaxies/xeigho/sectors/90_178" 2020-04-23 18-52-04| scheduled save for sector (38:461), 0x7f885696c5a0, entities: 1191 2020-04-23 18-52-04| saving sector (38:461) 2020-04-23 18-52-05| sector (38:461) saved to "/home/steam/avion/galaxies/xeigho/sectors/38_461" 2020-04-23 18-52-05| scheduled save for sector (40:463), 0x7f87fe263840, entities: 1645 2020-04-23 18-52-05| saving sector (40:463) 2020-04-23 18-52-06| sector (40:463) saved to "/home/steam/avion/galaxies/xeigho/sectors/40_463" 2020-04-23 18-52-40| SectorDatabase: Loading Sector (43:447) 2020-04-23 18-52-43| scheduled save for sector (43:447), 0x7f885d67f750, entities: 1622 2020-04-23 18-52-43| saving sector (43:447) SectorData2020-04-23 18-52-44ba| se: LoaSdiengc StecotrorD a(t43a:b44a4)s 2020-04-23 18-52-44| e: Loading Sector (42:456) Link to comment Share on other sites More sharing options...
Hammelpilaw Posted April 24, 2020 Share Posted April 24, 2020 Your logfiles look fine, just the logfile itself is a lil bugged, a harmless issue. You should change your server.ini: workerThreads=32 generatorThreads=16 scriptBackgroundThreads=8 CHANGE TO: workerThreads=6 generatorThreads=3 scriptBackgroundThreads=3 Further information regarding multithreading on servers: https://avorion.gamepedia.com/Setting_up_a_server#Multithreading But I guess you are having performance issues cause hosting Avorion on a shared server usely does not work very well... Maybe try to add more cores and a lil more ram to the VM.... Link to comment Share on other sites More sharing options...
Agent-Piano Posted April 28, 2020 Author Share Posted April 28, 2020 Your logfiles look fine, just the logfile itself is a lil bugged, a harmless issue. You should change your server.ini: workerThreads=32 generatorThreads=16 scriptBackgroundThreads=8 CHANGE TO: workerThreads=6 generatorThreads=3 scriptBackgroundThreads=3 Further information regarding multithreading on servers: https://avorion.gamepedia.com/Setting_up_a_server#Multithreading But I guess you are having performance issues cause hosting Avorion on a shared server usely does not work very well... Maybe try to add more cores and a lil more ram to the VM.... Thanks for the response, over the weekend I adjusted the configuration from 3 x 2 to 1 x 6 which seems to have led to some stability. I have also made the adjustment you recommend to provide hardware concurrency. From a CPU/RAM standpoint, neither the VMware host or the VM are tensioned, I could look at expanding the vCPUs laterally but based on what I'm seeing in performance data it looks as though those resources would not be used. Will be interesting to see how the game server performs with these changes. Link to comment Share on other sites More sharing options...
Agent-Piano Posted May 1, 2020 Author Share Posted May 1, 2020 Your logfiles look fine, just the logfile itself is a lil bugged, a harmless issue. You should change your server.ini: workerThreads=32 generatorThreads=16 scriptBackgroundThreads=8 CHANGE TO: workerThreads=6 generatorThreads=3 scriptBackgroundThreads=3 Further information regarding multithreading on servers: https://avorion.gamepedia.com/Setting_up_a_server#Multithreading But I guess you are having performance issues cause hosting Avorion on a shared server usely does not work very well... Maybe try to add more cores and a lil more ram to the VM.... After continued performance testing, we're still seeing a large amount of issues. It seems to be that once the server goes over a population of about 5-6 players the server becomes very unstable. Recently we experienced an evening where the server mass-disconnected all players three times in a fifteen minute period. Reviewing performance charts, there are not utilization or latency issues related to CPU, Memory, Disk, or Network. While it is operating in a shared environment there isn't any information that supports a hardware related problem. Any other thoughts on how to troubleshoot or remediate the problem? Link to comment Share on other sites More sharing options...
Agent-Piano Posted May 3, 2020 Author Share Posted May 3, 2020 We are also now seeing issues where player owned ships and stations are sometimes duplicating or cloning after players are disconnected from the server. Players are getting disconnected somewhat frequently. Sometimes the clone items disappear and sometimes they remain permanent. In other instances, ships are jumping to the incorrect system and to random coordinates far out of the range of their jump range (full quadrants, etc.). Hardware performance for the system continues to be very strong with no evidence of limitations. Still unsure as to why this application is struggling as much as it is. Link to comment Share on other sites More sharing options...
Hammelpilaw Posted May 5, 2020 Share Posted May 5, 2020 We are also now seeing issues where player owned ships and stations are sometimes duplicating or cloning after players are disconnected from the server. Players are getting disconnected somewhat frequently. Sometimes the clone items disappear and sometimes they remain permanent. In other instances, ships are jumping to the incorrect system and to random coordinates far out of the range of their jump range (full quadrants, etc.). Hardware performance for the system continues to be very strong with no evidence of limitations. Still unsure as to why this application is struggling as much as it is. 6 vCPU (3 Sockets x 2 Cores) - 1400MHz Reservation, High Share Priority 16GB Memory 6x 1400MHz is just not enaugh, as I said above this is no good setting for hosting Avorion. Depending on number of players online 16bg may also not be enaugh. You should upgrade. Link to comment Share on other sites More sharing options...
Agent-Piano Posted May 6, 2020 Author Share Posted May 6, 2020 We are also now seeing issues where player owned ships and stations are sometimes duplicating or cloning after players are disconnected from the server. Players are getting disconnected somewhat frequently. Sometimes the clone items disappear and sometimes they remain permanent. In other instances, ships are jumping to the incorrect system and to random coordinates far out of the range of their jump range (full quadrants, etc.). Hardware performance for the system continues to be very strong with no evidence of limitations. Still unsure as to why this application is struggling as much as it is. 6 vCPU (3 Sockets x 2 Cores) - 1400MHz Reservation, High Share Priority 16GB Memory 6x 1400MHz is just not enaugh, as I said above this is no good setting for hosting Avorion. Depending on number of players online 16bg may also not be enaugh. You should upgrade. The system isn't running 6 x 1400MHz. It's running at 6 x 2.4GHz and there is a cpu reservation for 14000* MHz on the host. The original number of 1400 was a typo, my apologies for that. So, the system running it has 6 x 2.4GHz processors which totals out to 14400MHz. Of that, 14000MHz is reserved on the host for only that VM so it is not competing for resources. While the wikipedia articles and FAQs seem to refer a lot to core and socket allocation I have not found any details which explain single thread speeds and what the requirements are. Based on that information, can you elaborate on your comments that my CPU configuration may not be adequate? Link to comment Share on other sites More sharing options...
Hammelpilaw Posted May 7, 2020 Share Posted May 7, 2020 Based on that information, can you elaborate on your comments that my CPU configuration may not be adequate? I guess your whole hardware is too old, also it is only virtualized. Can you reproduce your issues on a I7 or I9 CPU, using a dedicated machine and not a VM? Dedicated machines usely work better for hosting an Avorion server, also we still talk about a game - your Xeon CPU may result in a bad performance. You could try a benchmark and compare to your desktop computer or so... I just watched your CPU... it is 10 years old, you really should get better hardware. Also I recommand a Desktop CPU, not a server CPU. I have no issues on my server with 20 people online, but you need a high-end server for a really good performance. Link to comment Share on other sites More sharing options...
Agent-Piano Posted May 7, 2020 Author Share Posted May 7, 2020 Based on that information, can you elaborate on your comments that my CPU configuration may not be adequate? I guess your whole hardware is too old, also it is only virtualized. Can you reproduce your issues on a I7 or I9 CPU, using a dedicated machine and not a VM? Dedicated machines usely work better for hosting an Avorion server, also we still talk about a game - your Xeon CPU may result in a bad performance. You could try a benchmark and compare to your desktop computer or so... I just watched your CPU... it is 10 years old, you really should get better hardware. Also I recommand a Desktop CPU, not a server CPU. I have no issues on my server with 20 people online, but you need a high-end server for a really good performance. First of all, thank you continuing to stay engaged in the conversation to provide feedback. My hope is to work through understanding what kind of CPU requirements the dedicated server actually needs as they are not particularly well documented on the wikipedia. My observation in troubleshooting this issue is that the dedicated server engine needs a faster single-thread speed and that this is not a multi-core/socket issue. I have tried several configurations where I have added and reduced the CPU configurations for the Virtual Machine and the server engine simply does not use the extra processing power. The same is true with memory as I have added several GBs of additional memory and the server engine simple does not use it (even with adjustments to sectors in memory, etc.). With that said, my suspicion is that the limitation is more around CPU frequency (the speed of single threaded operations). I am in the process of implementing a new VMware host which has much higher CPU clock speeds and will test my theory, but having a grasp of what other folks are running without issue would be helpful. There is no reason that a Xeon class processor would not be able to run this server engine, but I do suspect that it's going to be looking for a minimum clock speed to run well and is not going to take advantage of having a significantly wider thread/CPU layout. Some i7 and i9 processors have much higher clock speeds, but that is not true for all of them. Accordingly, simple moving to an i7 or i9 isn't inherently going to lead to better performance. There is going to be some sort of coefficient of what the actual performance of the processor is versus the performance of the server engine. With that said, can you share what CPU configurations you're running on the system that hosts your server? I'm specifically looking for the model of CPU, memory speeds, and ideally chip-set. Being able to study the FSB speeds, cache speeds, and frequency will help us put some scientific data behind what is fast and slow If you have a post where you've already explained this I'd be happy to read it, just need some kind of cross-link. Thanks again for your time. Link to comment Share on other sites More sharing options...
Hammelpilaw Posted May 7, 2020 Share Posted May 7, 2020 With that said, can you share what CPU configurations you're running on the system that hosts your server? I'm specifically looking for the model of CPU, memory speeds, and ideally chip-set. Being able to study the FSB speeds, cache speeds, and frequency will help us put some scientific data behind what is fast and slow If you have a post where you've already explained this I'd be happy to read it, just need some kind of cross-link. Thanks again for your time. I dont know memory clock or chipset. I rented a dedicated root server, guess its running at default settings, google the CPU specs. I9 9900k 64G ram SSD Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now