|
Hi, I imported a planet pbf with osm2pgsql using this command line: osm2pgsql -c -k -d <dbname> -s -G -K -U <user> -W -H <host> -S <custom.style> planet-200224.osm.pbf I'm now trying to update with minutely diffs and keep getting duplicate key errors no matter what settings I try. The command line for diffs is: osmosis --rri workingDirectory=/mnt/OSM/osmosis-minutely/ --sc --wxc - | osm2pgsql -a -k -d <dbname> -P 5432 -s -G -K -U <user> -W -H <host> -S custom.style - I'm using 64-bit osm2pgsql (v 1.2.1). I've tried adding swap space, specifying multiple -C values (2000 up to 16000), number of processes from 1 to 8, importing 60 seconds at a time up to 1 hour, always the same error. The error I'm getting is this:
Thanks in advance for any assistance.
showing 5 of 6
show 1 more comments
|
|
You need to remove duplicates from the change file before importing into osm2pgsql. In osmosis the relevant option is called Interesting, that seems to have worked but I don't recall needing that option in the past. Thanks so much for your help!
(17 Mar '20, 06:06)
skmoore
|
Is there any danger in removing the unique constraint on the id column from the nodes/ways/rels tables in Postgres to work around this?
What's in the change file that you get the error with?
I get it with any change file. Whether its minutely/hourly, etc. I have it scheduled with cron and I've let it run for several hours, get the same error for each changeset.
I have seen it successfully process a minutely diff a couple of times though, which is weird. That makes me think it's more of a RAM issue on the server, but there appears to plenty of free RAM...
It's a 4 core 16gb ec2 instance connecting to an RDS Postgres instance.
(perhaps I should have been more precise in the question).
What data is there in the change file that is interpreted as a duplicate key?
Seems to almost always be a node (think I saw an error on a way a couple times). I've checked the nodes table in the database for several of the duplicate features, some exist there and some do not.
For example, the last time I ran it I got this error, the node is question does not exist in the Postgres database:
Processing: Node(170k 0.2k/s) Way(0k 0.00k/s) Relation(0 0.00/s)DB writer thread failed due to ERROR: result COPY_END for planet_osm_nodes failed: ERROR: duplicate key value violates unique constraint "planet_osm_nodes_pkey" DETAIL: Key (id)=(33442470) already exists. CONTEXT: COPY planet_osm_nodes, line 40