When I am trying to import a Europe pbf file with osm2pgsql, my diskspace keeps running full during the index creation. After the import of nodes, ways and relations the tablespace of the database has a size of 233 GB and there is still 244GB disk space available. But during index creation, these 244GB are completely used up. When I imported a europe data set one year ago, the disk space was enough for the database + the prerendered tiles. Any suggestions as to why the index creation might use up so much disk space this time? Is this normal? I'm using the following import command:
And the following postgres config:
My Stack: Mod_tile, renderd, mapnik, osm2pgsql and a postgresql/postgis database. Maybe it's worth mentioning that the system has 4CPU and 16GB RAM, with another 16GB of SWAP space. This is the error message in the log file:
asked 25 Oct '16, 09:58 cosmo42 |
Turning on autovacuum did the trick for me. During index creation 204 GB were used (instead of completely using up the 244 GB of left disk space). The indexes now take up about 131 GB. answered 03 Nov '16, 09:14 cosmo42 Since most guides for tuning of OSM postgres configs state that autovacuum should be turned off, I think it is worth mentioning that autovacuum could be necessary if disk space is scarce. Also I found no drawbacks in speed with autovacuum turned on - actually the overall import time was a bit faster.
(03 Nov '16, 09:23)
cosmo42
|
The total disk space used by all indexes on an Europe import should be about 150 GB but I can't say how much space might be temporarily used in addition to that. If you don't need the capability to update your data with incremental updates, add the You could also create your PostGIS tablespace on a compressed ZFS volume which will not only give you about twice the storage space but in all likelihood even speed up database operations. See http://www.paulnorman.ca/blog/2014/11/zfs-settings-for-osm2pgsql/ for examples. answered 25 Oct '16, 11:24 Frederik Ramm ♦ Thanks for the hint. Unfortunately I need to run updates later. Could it have an effect that autvacuum ist turned off?
(27 Oct '16, 08:11)
cosmo42
No, autovacuum turned off should only be a problem for updates to an existing database, not for the initial import.
(27 Oct '16, 08:28)
Frederik Ramm ♦
|