Compression of Sector-Files

[unbekannt1984]

Hi,

 

I thought that it might be good to use a compression-library to reduce the filesize of all those sectorfiles (and so on) on your harddisk. I used the linux-program tar with gzip-compression on my server and reduced the size of my whole galaxy to round about 1/4th to 1/5th.

 

Well, I'll show you what I've done:

torsten@debian:~$ cd .avorion/galaxies/
torsten@debian:~/.avorion/galaxies$ tar -cvzf backup.tar.gz ./avorion_galaxy/*
... (lots of output from verbose-mode - v-option) ...
torsten@debian:~/.avorion/galaxies$ ls -la
insgesamt 1164788
drwxr-xr-x 3 torsten torsten       4096 Mär  3 13:13 .
drwxr-xr-x 4 torsten torsten       4096 Mär  2 17:34 ..
drwxr-xr-x 5 torsten torsten       4096 Mai  8 22:51 avorion_galaxy
-rw-r--r-- 1 torsten torsten 1192725478 Mai  9 09:52 backup.tar.gz
torsten@debian:~/.avorion/galaxies$ du -h
5,5G	./avorion_galaxy/sectors
63M	./avorion_galaxy/factions
1,6M	./avorion_galaxy/players
5,5G	./avorion_galaxy
6,7G	.

 

Gzip-compression could also be included into C/C++ programs by using http://zlib.net/.

 

As you can see above, the files of that Galaxy are using 5.5 GB, compressing it into a tar-archive with gzip-compression reduced the size to round about 1.2 GB.

[Spacefaring_Guy]

I'm surprised you didn't get at least 90% compression given it's all highly repetitive text with a limited character set.

 

Cool, though.  Given how easy it is to compress/decompress text, for a lot of people this should actually speed the system up as they'll have more CPU throughput than disk... if you've ensured the data is decompressed to memory (and doesn't get cached back to disk!) and the files are larger than your disk cluster size.

 

Where this might really shine is if the server engine allocated memory and stored recently active sectors there in compressed format for quick decompression instead of going back to disk.  Who knows?  Maybe it already does that.

[unbekannt1984]

I'm surprised you didn't get at least 90% compression given it's all highly repetitive text with a limited character set.

Well no, it's not just text in there.

torsten@debian:~/.avorion/galaxies/avorion_galaxy/sectors$ ls -la
...
-rw-r--r-- 1 torsten torsten      356 Mai  5 21:17 -99_76v
-rw-r--r-- 1 torsten torsten      267 Mär 27 18:52 99_-76v
-rw-r--r-- 1 torsten torsten  3092690 Mär 27 12:13 99_-91
-rw-r--r-- 1 torsten torsten      271 Mär 27 12:13 99_-91v
torsten@debian:~/.avorion/galaxies/avorion_galaxy/sectors$ cat 99_-91v
<?xml version="1.0" encoding="utf-8"?>
<view numCrafts="3" numWrecks="0" numAsteroids="1393" numStations="0" influence="0" factionIndex="0">
<coordinates x="99" y="-91"/>
<titles/>
<wormholes/>
<shipNumbers>
	<ships n="3" faction="2000412"/>
</shipNumbers>
</view>

Try the same with 99_-91 and you just get an output of a binary file. zcat doesn't work either on that file.

 

Cool, though.  Given how easy it is to compress/decompress text, for a lot of people this should actually speed the system up as they'll have more CPU throughput than disk... if you've ensured the data is decompressed to memory (and doesn't get cached back to disk!) and the files are larger than your disk cluster size.

 

Where this might really shine is if the server engine allocated memory and stored recently active sectors there in compressed format for quick decompression instead of going back to disk.  Who knows?  Maybe it already does that.

 

Well, sending data from a server over a slow (upload-) connection is faster when you've got less data to send - meaning, if you have a limited upload-bandwidth and send a sector-file to a client, the compression and decompression might save some time in the end.

 

I think CS:S was able to do the same with bzip-compressed maps.

 

After making the original post here, I found this:

Updated link for anyone late to the party

Following the link you'll find that statement:

Planned Features

Compression of server data, both when sending (making loading screens shorter) and saving to disk

 

Well, the idea isn't new...

[Spacefaring_Guy]

Ha!  I was totally focused on the server-side and totally overlooked the server-client communication aspect.  I believe the average sector is about 1MB, so getting that down to ~100K would be a massive improvement over my sad 6 meg DSL link.

[unbekannt1984]

Ha!  I was totally focused on the server-side and totally overlooked the server-client communication aspect.  I believe the average sector is about 1MB, so getting that down to ~100K would be a massive improvement over my sad 6 meg DSL link.

 

Ok, let's make an example:

-rw-r--r-- 1 torsten torsten  3092690 Mär 27 12:13 99_-91

That Sector-File is 3 MB, I'd expect it to be 600 to 750 KB when compressed. Most likely your upload-bandwidth is less then 6 Mbit/S, let's say 1 Mbit/S = 128 KB/S.

Uploading the 3 MB sector-file would take 24 seconds, the compressed file would take just 5 to 6 seconds.

 

If decompression would take the client 30 seconds (for an even higher compression-rate), all of this would be a bad idea.

 

So, you need to keep compression/decompression in a certain range - it shouldn't slow things down in the end.