osm2pgsql td::bad_alloc when importing full planet

Hi,

I've imported France successfully and now I'm trying to import the full planet but can't get to the end of it. I've logged the output of osm2pgsql and seen a bad_alloc message:

osm2pgsql version 0.96.0 (64 bit id space)

Using built-in tag processing pipeline
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=18000MB, maxblocks=288000*65536, allocation method=11
Mid: loading persistent node cache from /home/renderer/data/flat_nodes/flat_nodes.bin
Mid: pgsql, cache=18000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: /data.osm.pbf
Using PBF parser.
Processing: Node(4725081k 9431.3k/s) Way(523620k 4.45k/s) Relation(646430 43.55/s)terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

And I also have some errors and fatals on the postgresql logs (EOF, terminating connection), but I supposed both are linked:

2018-09-20 21:35:26.442 GMT [134] LOG:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.442 GMT [138] LOG:  incomplete message from client
2018-09-20 21:35:26.442 GMT [138] CONTEXT:  COPY planet_osm_rels, line 646433
2018-09-20 21:35:26.442 GMT [138] STATEMENT:  COPY planet_osm_rels FROM STDIN;

2018-09-20 21:35:26.442 GMT [133] LOG:  incomplete message from client
2018-09-20 21:35:26.442 GMT [133] CONTEXT:  COPY planet_osm_line, line 116
2018-09-20 21:35:26.442 GMT [133] STATEMENT:  COPY planet_osm_line (osm_id,"access","admin_level","aerialway","aeroway","amenity","area","barrier","bridge","boundary","building","highway","landuse","layer","leisure","name","int_name","natural","oneway","place","population","railway","ref","service","tourism","tunnel","water","waterway","wetland","z_order","way_area","piste:type","piste:difficulty",tags,way) FROM STDIN
2018-09-20 21:35:26.442 GMT [135] LOG:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.444 GMT [137] LOG:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.447 GMT [133] ERROR:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.447 GMT [133] CONTEXT:  COPY planet_osm_line, line 116
2018-09-20 21:35:26.447 GMT [133] STATEMENT:  COPY planet_osm_line (osm_id,"access","admin_level","aerialway","aeroway","amenity","area","barrier","bridge","boundary","building","highway","landuse","layer","leisure","name","int_name","natural","oneway","place","population","railway","ref","service","tourism","tunnel","water","waterway","wetland","z_order","way_area","piste:type","piste:difficulty",tags,way) FROM STDIN
2018-09-20 21:35:26.449 GMT [138] ERROR:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.449 GMT [138] CONTEXT:  COPY planet_osm_rels, line 646433
2018-09-20 21:35:26.449 GMT [138] STATEMENT:  COPY planet_osm_rels FROM STDIN;

2018-09-20 21:35:26.451 GMT [136] ERROR:  unexpected EOF on client connection with an open transaction
2018-09-20 21:35:26.451 GMT [136] CONTEXT:  COPY planet_osm_nodes, line 1
2018-09-20 21:35:26.451 GMT [136] STATEMENT:  COPY planet_osm_nodes FROM STDIN;

2018-09-20 21:35:26.453 GMT [132] LOG:  incomplete message from client
2018-09-20 21:35:26.482 GMT [136] FATAL:  terminating connection because protocol synchronization was lost
2018-09-20 21:35:26.494 GMT [136] LOG:  could not send data to client: Broken pipe
2018-09-20 21:35:26.498 GMT [138] FATAL:  terminating connection because protocol synchronization was lost
2018-09-20 21:35:26.498 GMT [138] LOG:  could not send data to client: Broken pipe
2018-09-20 21:35:26.647 GMT [133] FATAL:  terminating connection because protocol synchronization was lost
2018-09-20 21:35:26.647 GMT [133] LOG:  could not send data to client: Broken pipe

I was importing with the following command:

osm2pgsql -d ${DBNAME} -H ${DBHOST} -P ${DBPORT} -U ${DBUSER} --create --slim -G --hstore --hstore-match-only -C 8000 --number-processes 8 -S /home/renderer/src/custom.style --flat-nodes /home/renderer/data/flat_nodes/flat_nodes.bin /data.osm.pbf

I have 32Gb of RAM, 8 cores, 1Gb of SWAP and 1Tb SSD (that's why I'm using flat-nodes). I've also tried with the -C option at 18000 (I've seen somewhere that beyond that value there was no difference). But I still have the issue. I really don't see how I could have a memory problem knowing that when I checked the usage during the import, I was around 20Gb of use max...

I've seen this github issue where it is advised to increase swap size but it seems strange to me, isn't the swap supposed to be used then de RAM is full? Anyway this would be a last resort solution. Also a guy suggest to remove the -C option and that will let osm2pgsql use what it needs, so I'm trying that right now but it seems strange too because I've seen in that doc that the default value is 800 if the parameter is not written.

I'm monitoring the usage of my server with this last solution just now, and the swap is almost full (960/1021MB), while the RAM is free (only 2Gb used on 32...).

Does anybody have any idea of where this problem may come from? And what I should do to successfully import the whole planet?

Thanks!