I installed Nominatim using full planet data from 2019-03-25 a couple of months ago, but initiated updates (as described at http://nominatim.org/release-docs/latest/admin/Import-and-Update/#updates) this week. Several diffs were applied at about 15-30 minutes per diff and then osm2pgsql got stuck on a diff only 68MB large.
I interrupted osm2pgsql after about 6 hours and restarted it and it has been running at 100% CPU for almost 20 hours already.
During the first (interrupted) run its last output was
Processing: Node(238k 1.9k/s) Way(31k 0.16k/s) Relation(700 16.67/s)
while in the current run the count of processed relations is lower (570 vs 700)
Processing: Node(238k 1.9k/s) Way(31k 0.16k/s) Relation(570 11.18/s)
Note, however, that the first run was performed without any output redirection, while the output of the current run is redirected to a file which probably means that the most recent progress message is simply not flushed yet.
Having copied the update command as is, I didn't provide the --osm2pgsql-cache option, so it is using the default value of 2000MB. But the fact that osm2pgsql is running at 100% CPU doesn't suggest that the low cache setting can be the root cause of this problem.
Examining the stack trace at different times suggests that osm2pgsql is processing a complex relation and is busy with a deep recursion inside osmium::area::detail::BasicAssembler::find_candidates():
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
std::_Construct<std::pair<osmium::area::detail::location_to_ring_map, bool>, std::pair<osmium::area::detail::location_to_ring_map, bool> > (__p=<optimized out>) at /usr/include/c++/7/bits/stl_construct.h:75
75 { ::new(static_cast<void*>(__p)) _T1(std::forward<_Args>(__args)...); }
#0 std::_Construct<std::pair<osmium::area::detail::location_to_ring_map, bool>, std::pair<osmium::area::detail::location_to_ring_map, bool> > (__p=<optimized out>) at /usr/include/c++/7/bits/stl_construct.h:75
#1 std::__uninitialized_copy<false>::__uninit_copy<std::move_iterator<std::pair<osmium::area::detail::location_to_ring_map, bool>*>, std::pair<osmium::area::detail::location_to_ring_map, bool>*> (__result=<optimized out>, __last=...,
__first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:83
#2 std::uninitialized_copy<std::move_iterator<std::pair<osmium::area::detail::location_to_ring_map, bool>*>, std::pair<osmium::area::detail::location_to_ring_map, bool>*> (__result=<optimized out>, __last=..., __first=...)
at /usr/include/c++/7/bits/stl_uninitialized.h:134
#3 std::__uninitialized_copy_a<std::move_iterator<std::pair<osmium::area::detail::location_to_ring_map, bool>*>, std::pair<osmium::area::detail::location_to_ring_map, bool>*, std::pair<osmium::area::detail::location_to_ring_map, bool> > (
__result=<optimized out>, __last=..., __first=...) at /usr/include/c++/7/bits/stl_uninitialized.h:289
#4 std::__uninitialized_move_if_noexcept_a<std::pair<osmium::area::detail::location_to_ring_map, bool>*, std::pair<osmium::area::detail::location_to_ring_map, bool>*, std::allocator<std::pair<osmium::area::detail::location_to_ring_map, bool> > > (__alloc=..., __result=<optimized out>, __last=0x55b024d324e0, __first=0x55b024d31d20)
at /usr/include/c++/7/bits/stl_uninitialized.h:312
#5 std::vector<std::pair<osmium::area::detail::location_to_ring_map, bool>, std::allocator<std::pair<osmium::area::detail::location_to_ring_map, bool> > >::_M_realloc_insert<osmium::area::detail::location_to_ring_map const&, bool> (
this=this@entry=0x7ffd2f638998, __position=
{first = {location = {m_x = 2000, m_y = 0, static undefined_coordinate = 2147483647}, ring_it = <error reading variable: Cannot access memory at address 0xf91>, start = 164}, second = 208}, __args#0=..., __args#1=@0x7ffd2f638980: true)
at /usr/include/c++/7/bits/vector.tcc:424
#6 0x000055b01f0d28fe in std::vector<std::pair<osmium::area::detail::location_to_ring_map, bool>, std::allocator<std::pair<osmium::area::detail::location_to_ring_map, bool> > >::emplace_back<osmium::area::detail::location_to_ring_map const&, bool> (this=this@entry=0x7ffd2f638998, __args#0=..., __args#1=@0x7ffd2f638980: true)
at /usr/include/c++/7/bits/vector.tcc:105
#7 0x000055b01f0d5d66 in osmium::area::detail::BasicAssembler::find_candidates (this=this@entry=0x7ffd2f63c280,
candidates=std::vector of length 0, capacity 0, loc_done=std::unordered_set with 62 elements = {...},
xrings=std::vector of length 672, capacity 672 = {...}, cand=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/area/detail/basic_assembler.hpp:736
... more osmium::area::detail::BasicAssembler::find_candidates() frames ...
....
... more osmium::area::detail::BasicAssembler::find_candidates() frames ...
#68 0x000055b01f0d5e04 in osmium::area::detail::BasicAssembler::find_candidates (this=this@entry=0x7ffd2f63c280,
candidates=std::vector of length 0, capacity 0, loc_done=std::unordered_set with 62 elements = {...},
xrings=std::vector of length 672, capacity 672 = {...}, cand=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/area/detail/basic_assembler.hpp:750
#69 0x000055b01f0dae1d in osmium::area::detail::BasicAssembler::join_connected_rings (this=this@entry=0x7ffd2f63c280,
open_ring_its=std::__cxx11::list = {...})
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/area/detail/basic_assembler.hpp:802
#70 0x000055b01f0dbcb4 in osmium::area::detail::BasicAssembler::create_rings_complex_case (
this=this@entry=0x7ffd2f63c280)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/area/detail/basic_assembler.hpp:920
#71 0x000055b01f0dc842 in osmium::area::detail::BasicAssembler::create_rings (this=0x7ffd2f63c280)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/area/detail/basic_assembler.hpp:1087
#72 0x000055b01f0ce9b0 in osmium::area::GeomAssembler::operator() (out_buffer=..., ways_buffer=..., relation=...,
this=0x7ffd2f63c280) at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/area/geom_assembler.hpp:112
#73 geom::osmium_builder_t::get_wkb_multipolygon[abi:cxx11](osmium::Relation const&, osmium::memory::Buffer const&) (
this=0x55b02047cf98, rel=..., ways=...) at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/osmium-builder.cpp:159
#74 0x000055b01f0a9c3d in output_gazetteer_t::process_relation (this=0x55b02047ca30, rel=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/output-gazetteer.cpp:704
#75 0x000055b01f05d080 in osmdata_t::relation_modify (this=<optimized out>, rel=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/osmdata.cpp:129
#76 0x000055b01f06ee82 in parse_osmium_t::relation (this=this@entry=0x7ffd2f63cab0, rel=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/parse-osmium.cpp:183
#77 0x000055b01f06fe70 in osmium::detail::apply_item_impl<parse_osmium_t&, osmium::memory::Item> (handler=..., item=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/visitor.hpp:68
#78 osmium::apply_item<osmium::memory::Item, parse_osmium_t&> (item=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/visitor.hpp:206
#79 osmium::apply<osmium::io::InputIterator<osmium::io::Reader, osmium::memory::Item>, parse_osmium_t&> (end=..., it=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/visitor.hpp:220
#80 osmium::apply<osmium::io::Reader, parse_osmium_t&> (c=...)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/contrib/libosmium/osmium/visitor.hpp:229
#81 parse_osmium_t::stream_file (this=0x7ffd2f63cab0, filename=..., fmt="auto")
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/parse-osmium.cpp:128
#82 0x000055b01f04ff2d in main (argc=<optimized out>, argv=<optimized out>)
at /srv/nominatim/Nominatim-3.2.0/osm2pgsql/osm2pgsql.cpp:86
I wonder if anything is wrong with this run, or such anomalies are possible.
