osm2pgsql –slim –drop import running out of disk space ( 1TB)

Hi all,

Unfortunately planet.osm.pbf osm2pgsql import crashed because of disk space lack (1TB available).

What I don't understand is that I used --slim --drop options and, after osm2pgsql crash, only 30% of disk space is used (259GB). So I guess the unecessary tables were already dropped when osm2pgsql went out of disk space. I don't think osm2pgsql need more than 700GB, even temporary, to build indexes.

I would like to relaunch the import, do you have any advice to use less disk space ? maybe

• use --flat-nodes osm2pgsql option ? if yes: do you think 1TB will be sufficent for node file + database ? or I should try to host node file on another 500GB disk ? Can you confirm that --drop osm2pgsql option will drop the node file at the end of the process ?
• turn on autovaccum as suggested here osm2pgsql import - disk space running out during index creation ?

In fact I got 916GB available, minus planet.osm.pbf file (50GB), so I got ~865GB. If it's confirmed that it is not sufficient for a full planet import, I'll add the information to this post ( Tile server hardware requirements ).

Below are more informations, thanks in advance for your help !

Augustin

osm2pgsql command

osm2pgsql --username $DBPG_USER --database $DBPG_DB \ --hstore \ --style $WORKING_DIR/openstreetmap-carto/openstreetmap-carto.style \ --tag-transform-script $WORKING_DIR/openstreetmap-carto/openstreetmap-carto.lua \ --slim --drop \ --cache 18800 \ --number-processes 4 \ --multi-geometry 20200102_planet.osm.pbf

system

OS Debian 9 Stretch Architecture x86_64 Processor model: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz 4 cores (1 thread per core) 32GB RAM/32GB swap disk space: 1To SSD

postgresql.conf extract (PostgreSQL version: 10)

shared_buffers = 8GB work_mem = 50MB maintenance_work_mem = 1GB min_wal_size = 1G max_wal_size = 2GB effective_cache_size = 20GB

osm2pgsql logs extract vs. disk space usage

On january 9th at 6pm, osm2pgsql was still processing relations

Processing: Node(5682827k 291.6k/s) Way(630406k 5.06k/s) Relation(3798620 120.78/s)

and 78% of disk space was used:

Filesystem Size Used Avail Use% Mounted on /dev/sda1 916G 670G 200G 78% /var/lib/postgresql/data

On january 10th at 8pm osm2pgsql crashed because of lack of disk space (logs below) whereas only 30% disk space is used

Filesystem Size Used Avail Use% Mounted on /dev/sda1 916G 259G 612G 30% /data

here are osm2pgsql logs with the crash:

Using PBF parser. Processing: Node(5682827k 291.6k/s) Way(630406k 5.06k/s) Relation(5590480 140.30/s) Standard exception processing relation id=8346358: TopologyException: side location conflict at 721638.68999999994 1016915.5699999999 Processing: Node(5682827k 291.6k/s) Way(630406k 5.06k/s) Relation(7379160 155.86/s) parse time: 191305s Node stats: total(5682827154), max(7094416299) in 19487s Way stats: total(630406201), max(759474335) in 124474s Relation stats: total(7379167), max(10509928) in 47344s Committing transaction for planet_osm_point Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Committing transaction for planet_osm_roads Setting up table: planet_osm_nodes Setting up table: planet_osm_ways Setting up table: planet_osm_rels Using lua based tag processing pipeline with script /docker_mounted_volumes/working_dir/openstreetmap-carto/openstreetmap-carto.lua Setting up table: planet_osm_nodes Setting up table: planet_osm_ways Setting up table: planet_osm_rels Using lua based tag processing pipeline with script /docker_mounted_volumes/working_dir/openstreetmap-carto/openstreetmap-carto.lua Setting up table: planet_osm_nodes Setting up table: planet_osm_ways Setting up table: planet_osm_rels Using lua based tag processing pipeline with script /docker_mounted_volumes/working_dir/openstreetmap-carto/openstreetmap-carto.lua Setting up table: planet_osm_nodes Setting up table: planet_osm_ways Setting up table: planet_osm_rels Using lua based tag processing pipeline with script /docker_mounted_volumes/working_dir/openstreetmap-carto/openstreetmap-carto.lua Going over pending ways... 434307049 ways are pending Using 4 helper-processes Finished processing 434307049 ways in 71654 s 434307049 Pending ways took 71654s at a rate of 6061.17/s Committing transaction for planet_osm_point Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Committing transaction for planet_osm_roads Committing transaction for planet_osm_point Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Committing transaction for planet_osm_roads Committing transaction for planet_osm_point Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Committing transaction for planet_osm_roads Committing transaction for planet_osm_point Committing transaction for planet_osm_line Committing transaction for planet_osm_polygon Committing transaction for planet_osm_roads Going over pending relations... 0 relations are pending Using 4 helper-processes Finished processing 0 relations in 0 s Committing transaction for planet_osm_point WARNING: there is no transaction in progress Committing transaction for planet_osm_line WARNING: there is no transaction in progress Committing transaction for planet_osm_polygon WARNING: there is no transaction in progress Committing transaction for planet_osm_roads WARNING: there is no transaction in progress Committing transaction for planet_osm_point WARNING: there is no transaction in progress Committing transaction for planet_osm_line WARNING: there is no transaction in progress Committing transaction for planet_osm_polygon WARNING: there is no transaction in progress Committing transaction for planet_osm_roads WARNING: there is no transaction in progress Committing transaction for planet_osm_point WARNING: there is no transaction in progress Committing transaction for planet_osm_line WARNING: there is no transaction in progress Committing transaction for planet_osm_polygon WARNING: there is no transaction in progress Committing transaction for planet_osm_roads WARNING: there is no transaction in progress Committing transaction for planet_osm_point WARNING: there is no transaction in progress Committing transaction for planet_osm_line WARNING: there is no transaction in progress Committing transaction for planet_osm_polygon WARNING: there is no transaction in progress Committing transaction for planet_osm_roads WARNING: there is no transaction in progress Sorting data and creating indexes for planet_osm_point Sorting data and creating indexes for planet_osm_line Sorting data and creating indexes for planet_osm_polygon Stopping table: planet_osm_nodes Sorting data and creating indexes for planet_osm_roads Stopping table: planet_osm_ways Stopping table: planet_osm_rels Stopped table: planet_osm_rels in 0s Stopped table: planet_osm_nodes in 1s Stopped table: planet_osm_ways in 1s Copying planet_osm_roads to cluster by geometry finished Creating geometry index on planet_osm_roads Creating indexes on planet_osm_roads finished All indexes on planet_osm_roads created in 2661s Completed planet_osm_roads node cache: stored: 2266194517(39.88%), storage efficiency: 91.97% (dense blocks: 275092, sparse nodes: 105299977), hit rate: 38.95% Osm2pgsql failed due to ERROR: CREATE TABLE planet_osm_point_tmp AS SELECT * FROM planet_osm_point ORDER BY ST_GeoHash(ST_Transform(ST_Envelope(way),4326),10) COLLATE "C" failed: ERROR: could not extend file "base/16385/4462032.9": wrote only 4096 of 8192 bytes at block 1292910 HINT: Check free disk space.