What is causing this error when I try to update OSM data

Question

I'm still trying to get openstreetmap-tiles-update-expire to run correctly. I have managed to figure out that I needed the python3 versions of the psycopg2, shapely and lxml for the current trim_osc.py, but now I've run into this error. My initial Google search didn't provide anything helpful, and I'm just not sure where to start with this.

I'm running Ubuntu 18.04 server with all updates applied, and I completed the steps shown in switch2osm for 18.04. I am using the steps outlined here, as modified above to account of Python3.

I should also add that I had java-related errors running Osmosis. The basic error showed as "the trustAnchors parameter must be non-empty." Researched, and was able to apply the solution described here.

Any help understanding what is going on is appreciated.

-Tim

[2018-05-31 05:25:21] 32203 filtering diff
Traceback (most recent call last):
  File "/home/<user>/src/regional/trim_osc.py", line 103, in <module>
    tree = etree.parse(options.osc if not options.gzip else gzip.GzipFile(fileobj=options.osc))
  File "src/lxml/etree.pyx", line 3426, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1860, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1880, in lxml.etree._parseFilelikeDocument
  File "src/lxml/parser.pxi", line 1775, in lxml.etree._parseDocFromFilelike
  File "src/lxml/parser.pxi", line 1186, in lxml.etree._BaseParser._parseDocFromFilelike
  File "src/lxml/parser.pxi", line 600, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult
  File "src/lxml/etree.pyx", line 316, in lxml.etree._ExceptionContext._raise_if_stored
  File "src/lxml/parser.pxi", line 370, in lxml.etree._FileReaderContext.copyToBuffer
  File "/usr/lib/python3.6/gzip.py", line 276, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.6/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib/python3.6/gzip.py", line 463, in read
    if not self._read_gzip_header():
  File "/usr/lib/python3.6/gzip.py", line 406, in _read_gzip_header
    magic = self._fp.read(2)
  File "/usr/lib/python3.6/gzip.py", line 91, in read
    self.file.read(size-self._length+read)
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
[2018-05-31 05:25:22] 32203 [error] Trim_osc error

Accepted Answer

2

This is about 75% guesswork so I could be way off, but:

It looks like something is trying to read gzip data from a file which was not opened in binary mode. It also looks like the file was opened by the argparse module, controlled by this line:

parser.add_argument('osc', type=argparse.FileType('r'), help='input osc file, "-" for stdin')

Try changing 'r' to 'rb' and see whether that makes any difference. This should probably be done conditionally on the '-z' flag, but I have no idea how to do that with argparse...

permanent link

answered 31 May '18, 08:25

turepalsson
836●10●16●25
accept rate: 25%

That bought me about 15 lines :-)

[2018-05-31 15:26:19] 23082 start import from seq-nr 2990717, replag is 2 day(s) and 19 hour(s) [2018-05-31 15:26:19] 23082 downloading diff [2018-05-31 15:26:47] 23082 filtering diff Traceback (most recent call last): File "/home/tileserver/src/regional/trim_osc.py", line 118, in <module> nodesM.append(long(node.get('id'))) NameError: name 'long' is not defined [2018-05-31 15:26:48] 23082 [error] Trim_osc error [2018-05-31 15:26:48] 23082 resetting state

Pastebin of the output: https://pastebin.com/nyiGgwyV

openstreetmap-tiles-update-expire complains "rm: cannot remove '/var/lib/mod_tile/dirty_tiles.23082': No such file or directory"

When should dirty_tiles be created?

(31 May '18, 16:35) tim_rohrer

1

The long() thing looks like something is trying to run python2 code with a python3 interpreter.

(31 May '18, 17:15) turepalsson

It is running now. Not sure how long the import diff should take? The original data file was from the 28th, and I'm only covering the US state of California?

(31 May '18, 18:18) tim_rohrer

That's a bit "how long is a piece of string" :)

On hardware that imports the British Isles in 4 or so hours it normally takes another 3 to catch up from 30 hours behind. You might be able to work something out from that...

(31 May '18, 18:38) SomeoneElse ♦

That does help, I think. My import of California only took a few minutes; unfortunately, I failed to capture an exact time, but it wasn't long.

This has been running for close to an hour now. Four postgres processes running up high in top. So, I'll let it continue. It seems to be taking a lot longer than the original import.

(31 May '18, 18:47) tim_rohrer

Cool. It looks like the entire thing ran for about an hour and a half. I finally got "Done with import" in the log.

Weirdly, my other terminal session froze; second time that has happened. But, that is an different problem :-)

Thanks!

p.s. I opened an issue with https://github.com/Zverik/regional/issues regarding the binary read and the long. I'd like to hear some validation of these issues, and then I'd be willing to write up something for the wiki.

(31 May '18, 19:33) tim_rohrer

showing 5 of 6 show 1 more comments

What is causing this error when I try to update OSM data

Follow this question

Related questions