mtsatellite/SETUP.md

SETUP MTSatellite

You will need a Minetest server with Redis support compiled in. Consult the Minetest documentation to figure out how to get such build. Furthermore you need the binaries mtdbconverter, mtseeder, mtredisalize and mtwebmapper in your PATH. Consult COMPILE how to build these.

Setting up MTSatellite takes six steps:

1. Backup your world
2. Convert world database into interleaved format
3. Start mtredisalize
4. Pre-compute the map tiles with mtseeder
5. Start the web server mtwebmapper
6. Configure and restart the Minetest server

Experimental: Optionally you can enable on map tracking of logged in players.

Backup your world

Stop your running Minetest server and make a backup of your world before you will start crying.

Convert world database into interleaved format

MTSatellite operates best if the block data of the world is stored in a LevelDB database with a key scheme called interleaved. With this key scheme you can pick up sets of neighbored blocks a lot quicker than with a plain database. See Z-order curve at Wikipedia to grasp the core ideas. MTSatellite can run on plain LevelDB or SQLite3 world databases but with slightly reduced performance. This should work but to our knowledge it is not used in productive setups. Stay with the interleaved format!

To convert your original plain SQLite3 or LevelDB database (Redis is not supported atm) to the interleaved LevelDB format you have to use mtdbconverter:

mtdbconverter -source-backend=sqlite /path/to/your/world/map.sqlite /path/to/your/world/map.db

Depending on the size of your world and the speed of your computer system this conversion will take some time. Change -source-backend=sqlite to -source-backend=leveldb if your world is stored as a LevelDB. mtdbconverter can also be used to convert your world back to the plain key scheme. Use mtdbconverter --help to see all options.

You can skip the conversion if you want to use a plain database.

Start mtredisalize

mtredisalize is the component which serves the block data to Minetest and mtwebmapper as a Redis look-alike server. Start it with:

mtredisalize                               \
  -host=localhost                          \
  -interleaved=true                        \
  -change-url=http://localhost:8808/update \
  -change-duration=10s                     \
  /path/to/your/world/map.db

This binds the server to localhost port 6379 the default Redis port. You can shange it with the -port= option. The -interleaved=true option is mandatory if you use the interleaved format of the database. Forgetting it will end up in the crying mentioned above. Set this flag to false if you are using a plain database.
The -change-url= option is a forward reference to the mtwebmapper server which will be notified if the world has changed. If it is not configured the tile re-generation is not triggered. As long as the Minetest server is down there will be no changes and therefore it is safe to configure it even if the mtwebmapper service is not running.
The -change-duration= option specifies the amount of time how long the mtredisalize server should aggregate changes made to the world before reporting them to mtwebmapper. It defaults to 30 seconds but the value can be increased or decreased depending how often you want to update the map. Decreasing it will increase the computing pressure on your system so configure it wisely.

Pre-compute the map tiles with mtseeder

Even in a dynamical Mintest world played with many players most of the data is static over time. To generate a basic map to apply only changes to use mtseeder:

GOMAXPROCS=6 mtseeder              \
  -colors=/path/to/your/colors.txt \
  -output-dir=/path/to/your/map    \
  -workers=3

This contacts the mtredisalize server running at localhost port 6379 to fetch the block data from. You will need a colors.txt to map the block nodes to pixel colors of your map. The repository contains a prefabricated or you can create an adjusted one fitting your server with mtautocolors.
If you want to have certain nodes to be transparent you can add -transparent=true to the options. In this case if a color from colors.txt does have a forth color component the numerical value between 0 (fully transparent) and 255 (fully opaque) will be the base transparency of the pixel. Every depth meter of the same material will reduce the transparency by 2%. This can be adjusted with the -transparent-dim=percent flags.
See mtseeder --help for all options.

The -workers= option and the GOMAXPROCS= environment variable are completely optional but very useful to exploit multiple processor cores on your machine. Set GOMAXPROCS= to the result of nproc and -workers= to a number a little lesser. You have to experiment with this to find a good setting.

Even with good CPU usage generating the map and overview image tiles take a while.

Tip: A lot of the Minetest map tiles are white/empty but are saved as dupes in the file system. To deduplicate them you can use e.g. hardlink. You can also run it as a nightly cron job to dedupe the map on a regular basis.

Start the web server mtwebmapper

This web server serves the Leaflet compatibles to the browser and is contacted by mtredisalize if something in the world has changed. In this case the corresponding map tiles are re-generated in the background. To start mtwebmapper use:

GOMAXPROCS=3 mtwebmapper           \
  -colors=/path/to/your/colors.txt \
  -web-host=""                     \
  -map=/path/to/your/map           \
  -web=/path/to/your/static/web    \
  -redis-host=localhost            \
  -workers=2                       \
  -websockets=false

For the colors= options applys the same as said above. You can also add -transparent=true for transparency as mentioned above. The web-host= is the interface the server ist listening on. "" means all interfaces. The port defaults to 8808. For a productive setup you may consider running it behind a reverse proxy.
-map= has to be the same path as used by mtseeder.
-web= is the path to the static web data (Leaflet, HTML, CSS, etc.). You can take it from the repository

To fetch the block data from the mtredisalize you have to use the option redis-host=. If you omit this then there will be no background job to re-generate the map. This is useful if you want to serve a map that is only generated once whith mtseeder.

To see all the available options use mtwebmapper --help.

The GOMAXPROCS=/-workers= setting has to be adjusted to your system capacity. Do not give to much ressources to this if you planning to run the mapping webserver on the same machine as the Minetest server. On the other hand assigning more cores to it definitely helps to boost up the performance.

Setting the -websockets=true flag enables websocket support for the server. With this feature turned on and changing the line (in web/index.html) from

var useWebsocket = false; // Set to true if you want websocket support

to

var useWebsocket = true; // Set to true if you want websocket support

the web client gets an extra 'auto update' button. When switched on the server informs the client if something in the maps has changed. The displayed map will then update automatically without the need of manual pressing the 'update view' button. Of cause your browser needs Websocket support, too.

Configure and restart the Minetest server

Now everything is in place and the only thing left ist to re-configure the Minetest server itself. You have to open your /path/to/your/world.mt file in your text editor and replace the backend with a Redis configuration:

backend = redis
redis_hash = IGNORED
redis_address = localhost

You may have to set redis_port too if you run mtredisalize not on port 6379.

Now we are all done and you can fire your Minetest server up again. :-)

Enable on map tracking of logged in players

MTSatellite can display logged in players on the map.
This is an experimental feature and its only confirmed working on GNU/Linux systems. OS X and *BSD should work, too.

To use it install the track_players mod. Simple add a checkout to your mods folder and activate it in your world.mt file.

...
load_mod_track_players = true
...

This minetest mod writes players position to a named pipe aka FIFO. mtwebmapper is able to read from this file and serve these positions as GeoJSON to the browser. The FIFO has to be created before the start of the minetest server.

$ mkfifo /tmp/mt_players_fifo

The path to the FIFO can be changed in track_players/init.lua

...
local fifo_path = "/tmp/mt_players_fifo"
...

To use the feature in mtwebmapper add the argument -players=/tmp/mt_players_fifo to the list of command line arguments.

Caution: Please start mtwebmapper before the minetest server! Caused by the nature of FIFOs and the single threaded execution of minetest mods the minetest server will block if there is no consumer reading the player positions.

The player tracking is well integrated with the websocket support. If you enable websockets you will be able to see the players moving on the map.