When I am trying to import a Europe pbf file with osm2pgsql, my diskspace keeps running full during the index creation.

After the import of nodes, ways and relations the tablespace of the database has a size of 233 GB and there is still 244GB disk space available. But during index creation, these 244GB are completely used up. When I imported a europe data set one year ago, the disk space was enough for the database + the prerendered tiles.

Any suggestions as to why the index creation might use up so much disk space this time? Is this normal?

I'm using the following import command:

nohup osm2pgsql ~/osm/europe-latest.osm.pbf --create --slim --database gis --number-processes=4 --cache=22000 --multi-geometry --flat-nodes ~/osm/flat-nodes.bin

And the following postgres config:

shared_buffers 8 MB
work_mem 1 MB
maintenance_work_mem 4096 MB
fsync off
autovacuum off
checkpoint_segments 60

My Stack: Mod_tile, renderd, mapnik, osm2pgsql and a postgresql/postgis database.

Maybe it's worth mentioning that the system has 4CPU and 16GB RAM, with another 16GB of SWAP space.

This is the error message in the log file:

Creating geometry index on  planet_osm_polygon
pthread_join() returned exception: Throw location unknown (consider using BOOST_THROW_EXCEPTION)
Dynamic exception type: boost::exception_detail::clone_impl<boost::exception_detail::current_exception_std_exception_wrapper<std::runtime_error> >
std::exception::what: CREATE INDEX planet_osm_polygon_index ON planet_osm_polygon USING GIST (way)   failed: ERROR:  could not extend file "base/16385/8687919.15": No space left on device
HINT:  Check free disk space.
St13runtime_error
Error occurred, cleaning up

asked 25 Oct '16, 09:58

cosmo42's gravatar image

cosmo42
11112
accept rate: 0%


Turning on autovacuum did the trick for me. During index creation 204 GB were used (instead of completely using up the 244 GB of left disk space). The indexes now take up about 131 GB.

permanent link

answered 03 Nov '16, 09:14

cosmo42's gravatar image

cosmo42
11112
accept rate: 0%

Since most guides for tuning of OSM postgres configs state that autovacuum should be turned off, I think it is worth mentioning that autovacuum could be necessary if disk space is scarce. Also I found no drawbacks in speed with autovacuum turned on - actually the overall import time was a bit faster.

(03 Nov '16, 09:23) cosmo42

The total disk space used by all indexes on an Europe import should be about 150 GB but I can't say how much space might be temporarily used in addition to that. If you don't need the capability to update your data with incremental updates, add the --drop flag to avoid building unnecessary indexes.

You could also create your PostGIS tablespace on a compressed ZFS volume which will not only give you about twice the storage space but in all likelihood even speed up database operations. See http://www.paulnorman.ca/blog/2014/11/zfs-settings-for-osm2pgsql/ for examples.

permanent link

answered 25 Oct '16, 11:24

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
69.4k806291086
accept rate: 24%

Thanks for the hint. Unfortunately I need to run updates later. Could it have an effect that autvacuum ist turned off?

(27 Oct '16, 08:11) cosmo42

No, autovacuum turned off should only be a problem for updates to an existing database, not for the initial import.

(27 Oct '16, 08:28) Frederik Ramm ♦
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×200
×164
×11
×7

question asked: 25 Oct '16, 09:58

question was seen: 2,762 times

last updated: 03 Nov '16, 09:23

powered by OSQA