NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum

Hi,

I've imported France successfully and now I'm trying to import the full planet but can't get to the end of it. I've logged the output of osm2pgsql and seen a bad_alloc message:

osm2pgsql version 0.96.0 (64 bit id space)

Using built-in tag processing pipeline
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=18000MB, maxblocks=288000*65536, allocation method=11
Mid: loading persistent node cache from /home/renderer/data/flat_nodes/flat_nodes.bin
Mid: pgsql, cache=18000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: /data.osm.pbf
Using PBF parser.
Processing: Node(4725081k 9431.3k/s) Way(523620k 4.45k/s) Relation(646430 43.55/s)terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

And I also have some errors and fatals on the postgresql logs (EOF, terminating connection), but I supposed both are linked:

2018-09-20 21:35:26.442 GMT [134] LOG:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.442 GMT [138] LOG:  incomplete message from client
2018-09-20 21:35:26.442 GMT [138] CONTEXT:  COPY planet_osm_rels, line 646433
2018-09-20 21:35:26.442 GMT [138] STATEMENT:  COPY planet_osm_rels FROM STDIN;

2018-09-20 21:35:26.442 GMT [133] LOG:  incomplete message from client
2018-09-20 21:35:26.442 GMT [133] CONTEXT:  COPY planet_osm_line, line 116
2018-09-20 21:35:26.442 GMT [133] STATEMENT:  COPY planet_osm_line (osm_id,"access","admin_level","aerialway","aeroway","amenity","area","barrier","bridge","boundary","building","highway","landuse","layer","leisure","name","int_name","natural","oneway","place","population","railway","ref","service","tourism","tunnel","water","waterway","wetland","z_order","way_area","piste:type","piste:difficulty",tags,way) FROM STDIN
2018-09-20 21:35:26.442 GMT [135] LOG:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.444 GMT [137] LOG:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.447 GMT [133] ERROR:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.447 GMT [133] CONTEXT:  COPY planet_osm_line, line 116
2018-09-20 21:35:26.447 GMT [133] STATEMENT:  COPY planet_osm_line (osm_id,"access","admin_level","aerialway","aeroway","amenity","area","barrier","bridge","boundary","building","highway","landuse","layer","leisure","name","int_name","natural","oneway","place","population","railway","ref","service","tourism","tunnel","water","waterway","wetland","z_order","way_area","piste:type","piste:difficulty",tags,way) FROM STDIN
2018-09-20 21:35:26.449 GMT [138] ERROR:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.449 GMT [138] CONTEXT:  COPY planet_osm_rels, line 646433
2018-09-20 21:35:26.449 GMT [138] STATEMENT:  COPY planet_osm_rels FROM STDIN;

2018-09-20 21:35:26.451 GMT [136] ERROR:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.451 GMT [136] CONTEXT:  COPY planet_osm_nodes, line 1
2018-09-20 21:35:26.451 GMT [136] STATEMENT:  COPY planet_osm_nodes FROM STDIN;

2018-09-20 21:35:26.453 GMT [132] LOG:  incomplete message from client
2018-09-20 21:35:26.482 GMT [136] FATAL:  terminating connection because protocol synchronization was lost
2018-09-20 21:35:26.494 GMT [136] LOG:  could not send data to client: Broken pipe
2018-09-20 21:35:26.498 GMT [138] FATAL:  terminating connection because protocol synchronization was lost
2018-09-20 21:35:26.498 GMT [138] LOG:  could not send data to client: Broken pipe
2018-09-20 21:35:26.647 GMT [133] FATAL:  terminating connection because protocol synchronization was lost
2018-09-20 21:35:26.647 GMT [133] LOG:  could not send data to client: Broken pipe

I was importing with the following command:

osm2pgsql -d ${DBNAME} -H ${DBHOST} -P ${DBPORT} -U ${DBUSER} --create --slim -G --hstore --hstore-match-only -C 8000 --number-processes 8 -S /home/renderer/src/custom.style --flat-nodes /home/renderer/data/flat_nodes/flat_nodes.bin /data.osm.pbf

I have 32Gb of RAM, 8 cores, 1Gb of SWAP and 1Tb SSD (that's why I'm using flat-nodes). I've also tried with the -C option at 18000 (I've seen somewhere that beyond that value there was no difference). But I still have the issue. I really don't see how I could have a memory problem knowing that when I checked the usage during the import, I was around 20Gb of use max...

I've seen this github issue where it is advised to increase swap size but it seems strange to me, isn't the swap supposed to be used then de RAM is full? Anyway this would be a last resort solution. Also a guy suggest to remove the -C option and that will let osm2pgsql use what it needs, so I'm trying that right now but it seems strange too because I've seen in that doc that the default value is 800 if the parameter is not written.

I'm monitoring the usage of my server with this last solution just now, and the swap is almost full (960/1021MB), while the RAM is free (only 2Gb used on 32...).

Does anybody have any idea of where this problem may come from? And what I should do to successfully import the whole planet?

Thanks!

asked 21 Sep '18, 16:26

voharunado's gravatar image

voharunado
665510
accept rate: 0%


Surprisingly enough software changes now and then and you shouldn't be assuming that tips given for older version still apply now.

Current versions of osm2pgsql use a much more efficient caching algorithm and will not degrade as much as previous ones, which essentially required a cache large enough to keep all nodes in memory.

In any case 1GB swap is not going to get you anywhere, 64 would be more like it (don't forget you have pgsql processes any lots of other stuff running during the import too), maybe even 128.

permanent link

answered 22 Sep '18, 08:24

SimonPoole's gravatar image

SimonPoole ♦
44.7k13326701
accept rate: 18%

edited 22 Sep '18, 08:37

Thanks for that advice, it still seems strange to me because I thought the RAM was used first, so with 32Gb of it, I assume I wouldn't need much swap. Anyway, I tried this weekend to remove de -C option (as advised in the github issue I linked) and the process terminated without the bad_alloc message. I still have an error but not related to memory, so if it works, I'll stay with that.

(24 Sep '18, 10:26) voharunado

I successfully passed the processing step by removing the -C option from my osm2pgsql command. The process didn't go further but for a totally different reason not relevant to this topic. But I still wonder if this advice I've seen on github is good, because it seems to me that this will tell osm2pgsql to use only 800Mb of RAM so this will inevitably slow down the process no? It may have allowed me to continue the process beyond the processing step but I'm afraid I will loose a lot of time for the rest of the import...

(24 Sep '18, 15:19) voharunado

I can now confirm that removing the -C option worked.

However the full planet import took almost 3 days (244721s), but I just wanted it to work so it's fine for me.

(27 Sep '18, 16:08) voharunado

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×263
×15
×4
×2

question asked: 21 Sep '18, 16:26

question was seen: 3,758 times

last updated: 27 Sep '18, 19:20

NOTICE: help.openstreetmap.org is no longer in use from 1st March 2024. Please use the OpenStreetMap Community Forum