Dedicated Server Crash

[ChipPatton]

It seems the dedicated server is running out of memory with only 2 sectors active when saving.  This server is a Dell PowerEdge U2 and has 236gb of ram and 32 Xeon cores (4 cpu's).  This runs on x64 CentOS7. 

 

This occurs in a system where there is a lot of traffic and there are at least 2 full carrier's worth of fighters out and flying around performing various tasks.  The ram usage for the game sits at about 5-6gb.

 

"bad_alloc thrown for a size of 140703480704992"

 

That's 140 Terabytes for a 421Mb save.

 

Edit, this has now gone from crashing every 5-20 mins to crashing as soon as the first player joins the server.

 

Edit 2, I have been able to stop this crashing by uninstalling Carrier Command.  I'm referring this to the mod author.

 

Triggered saving of all server data.
All sectors saved successfully.
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0xdb62b2 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x7ff88d115cf9 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x7ff88d11690b 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x7ff88d116fdc 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0xd207c9 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x7f5ab8 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0xc588d5 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0xa025c1 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0xb42082 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x753962 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x754090 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x7ff88d10d2b0 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x7ff88dbbde25 2>&1
addr2line -i -e /home/divinereapers/steam/avorion/bin/AvorionServer 0x7ff88c87534d 2>&1
bad_alloc thrown for a size of 140703480704992
backtrace() returned 15 addresses
[bt] #1 ./bin/AvorionServer(operator new(unsigned long)+0x22) [0xdb62b2]
[bt] #2 /lib64/libstdc++.so.6(std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)+0x59) [0x7ff88d115cf9]
[bt] #3 /lib64/libstdc++.so.6(std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long)+0x1b) [0x7ff88d11690b]
[bt] #4 /lib64/libstdc++.so.6(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&)+0x5c) [0x7ff88d116fdc]
[bt] #5 ./bin/AvorionServer(Script::getExecutionContext(std::string const&)+0x9) [0xd207c9]
[bt] #6 ./bin/AvorionServer(void GameCallbackSender::safeCallback<std::tuple<ut::uuid, ut::uuid>&>(std::string const&, std::tuple<ut::uuid, ut::uuid>&)+0x1f8) [0x7f5ab8]
[bt] #7 ./bin/AvorionServer(DelayedCallbackSender::sendDelayedCallbacks()+0xe5) [0xc588d5]
[bt] #8 ./bin/AvorionServer(cp::ScriptCallback::update(float)+0x11) [0xa025c1]
[bt] #9 ./bin/AvorionServer(Scene::sendRemainingEntityCallbacks(float)+0x92) [0xb42082]
[bt] #10 ./bin/AvorionServer(ut::Worker<ut::queue<std::shared_ptr<ut::WorkPacket> > >::workSinglePacket()+0x2b2) [0x753962]
[bt] #11 ./bin/AvorionServer(ut::Worker<ut::queue<std::shared_ptr<ut::WorkPacket> > >::operator()()+0x80) [0x754090]
[bt] #12 /lib64/libstdc++.so.6(+0xb52b0) [0x7ff88d10d2b0]
[bt] #13 /lib64/libpthread.so.0(+0x7e25) [0x7ff88dbbde25]
[bt] #14 /lib64/libc.so.6(clone+0x6d) [0x7ff88c87534d]
??:?
??:0
??:0
??:0
??:?
??:?
??:?
??:?
??:?
??:?
??:?
??:0
??:0
??:0
Exception in Game WorkPool worker thread 9: std::bad_alloc
Script Memory: 143.36MB
Sectors Total: 77
Workpacket [140701627041536]: 
Players Online: 1
Workpacket [140704470791936]: 
Workpacket [140704445613824]: 
Sectors Loaded: 5
Workpacket [140704428828416]: 
Workpacket [140705460098816]: 
Workpacket [140704906983168]: 
Workpacket [140705476884224]: 
Workpacket [140702172305152]: 
Workpacket [140704923768576]: 
Workpacket [140703866812160]: 
Workpacket [140702189090560]: 
Workpacket [140704932161280]: 
Players In Memory: 1
Workpacket [140705485276928]: 
Workpacket [140702675605248]: 
Workpacket [140705468491520]: 
Workpacket [140703246046976]: 
Workpacket [140704420435712]: 
Workpacket [140704462399232]: 
Workpacket [140702709176064]: 
Workpacket [140704454006528]: 
Workpacket [140705451706112]: 
Workpacket [140704437221120]: Sector[3:168]::sendRemainingEntityCallbacks
Workpacket [140704898590464]: 
Workpacket [140704940553984]: 
Workpacket [140705426528000]: 
Workpacket [140703220868864]: 
Workpacket [140703858419456]: 
Workpacket [140703850026752]: 
Workpacket [140703841634048]: 
Workpacket [140703833241344]: 
Workpacket [140704890197760]: 
Workpacket [140705434920704]: 
Workpacket [140703816455936]: 
Factions In Memory: 5
Workpacket [140703262832384]: 
Workpacket [140703254439680]: 
Workpacket [140703237654272]: 
Workpacket [140703229261568]: 
Workpacket [140703824848640]: 
Workpacket [140703212476160]: 
Workpacket [140702725961472]: 
Workpacket [140704915375872]: 
Workpacket [140702180697856]: 
Workpacket [140705443313408]: 
Workpacket [140702717568768]: 
Workpacket [140702700783360]: 
Workpacket [140702692390656]: 
Workpacket [140702683997952]: 
/bin/sh: line 1:  9492 Killed