SOLVED

Following a comment below saying the script works for another user, I created a greatly simplified Overpass query to pull the missing data and it works fine. This tells me that the nodes are missing from the source query results so there's actually nothing wrong with my LUA script.

Thanks for the replies, if I can't work out the issue with the overpass query I'll open another question. Hopefully I won't have to :)


I've been working on a LUA import script for osm2pgsql and once I had it all working I checked the database in qgis and found that some ways are missing.

I can see them in the source XML file (from overpass), with all their node references but osm2pgsql does not import them.

I've created a small test script to try and visualise what is happening

inspect = require('inspect')

local tables = {}

tables.way_test = osm2pgsql.define_table({
    name = 'way_test',
    ids = { type = 'way', id_column = "id"},
    columns = {
        { column = 'name', type = 'text' },
        { column = 'geom', type = 'geometry' },
    }
})

function osm2pgsql.process_way(object)
    if object.id == 987525760 then
        print(inspect(object))
        print("line", object.as_linestring())
        print("poly", object.as_polygon())
        print("mline", object.as_multilinestring())
        print("mpoly", object.as_multipolygon())
    end

    tables.way_test:add_row({
        name = object.tags.name,
        geom = { create = 'line' },
    })
end

The above script is run with the following CLI arguments to version 1.7.0 which I compiled myself:

osm2pgsql --host=localhost --user=user --database=db --cache=40050 --flat-nodes=nodes.tmp -sWG --output=flex --style=test.lua data.pbf

This script does not add the missing lines to the test table but does tell me at all four of the as_ functions return NULL.

I've also imported the same data using the default style and the way does not appear in the lines table.

osm2pgsql --host=localhost --user=user --database=db --cache=40050 --flat-nodes=nodes.tmp -sWG --output=pgsql data.pbf

All this suggests dodgy data but the way exists and does have a geometry in OSM itself https://www.openstreetmap.org/way/987525760

While this is one specific way, there are a few I've spotted due to familiarity with the area. Some, like this one are missing while a few others seem to be a straight line from the first to last node without other nodes in between. Can anyone explain why this way is not being imported?

Versions:

osm2pgsql version 1.7.0 (1.7.0)
Build: RelWithDebInfo
Compiled using the following library versions:
Libosmium 2.17.3
Proj [API 6] 8.2.1
Lua 5.3.6

asked 29 Aug, 14:57

DarkSnow's gravatar image

DarkSnow
165
accept rate: 0%

edited 31 Aug, 11:48

https://github.com/gravitystorm/openstreetmap-carto/blob/master/openstreetmap-carto.lua is a fairly minimal implementation for the normal backend - maybe start from there and amend the functions you want to change as required?

(29 Aug, 20:04) SomeoneElse ♦
1

As other mentioned before, you should at least post the complete command line you have been using as well as the version number of osm2pgsql. Otherwise we are stumbling in the dark. When I try your script, osm2pgsql fails with an error message, so I would expect this not to put any data in the database. When I remove the offending lines (those with as_multilinestring() and as_multipolygon()) the script runs through add adds that way to the database.

(30 Aug, 08:20) Jochen Topf

Jochen, it's very interesting that the script is working for you. The print lines are for testing, but work fine for me on version 1.7.0, so can be safely removed.

If the way is being imported for you, how are you getting the source data?

I'm using the following query to get data from overpass:

[timeout:1800];
// Gather results from the UK
area["name"="United Kingdom"]->.boundaryarea;
(
  wr["piste:type"](area.boundaryarea);
  rel["site"="piste"](area.boundaryarea);
  >>;
  wr[~"^([A-Za-z]+:)?landuse$"~"^winter_sports$"](area.boundaryarea);
  (._; >;);
  way(r)["railway"](area.boundaryarea)->.siterailways;
  way["railway"="funicular"](area.boundaryarea)->.funiculars;
  way[~"^([A-Za-z]+:)?aerialway$"~"^.*$"](area.boundaryarea)->.aerialways;
  ((.aerialways; .siterailways; .funiculars;); >;);
  rel[site=piste](area.boundaryarea)->.sites;
  (way(r.sites)[!"piste:type"][!"aerialway"][!"railway"](area.boundaryarea);
  node(r.sites);)->.pois;
  (.pois; .pois>;);
);
// print results
out body geom;
(30 Aug, 12:12) DarkSnow

There is a whole lot of information missing in your question, the most important one: are you using 'pgsql' or 'flex' output?

You can't mix both. If you create a LUA file designed for flex, you need to use the flex output parameters on the command line to have it properly function. The same for pgsql. Don't mix them up.

Read the osm2pgsql manual carefully:

https://osm2pgsql.org/doc/manual.html

and realize chapters 6 (for flex) and 7 (pgsql) there should be treated as wholly separate information: either you need to read 6, or 7, depending on the choice of output. If you choose one, then forget about the information in the other chapter, as it won't be relevant.

permanent link

answered 30 Aug, 07:37

mboeringa's gravatar image

mboeringa
1.5k21527
accept rate: 9%

Thanks for that response. It's a lua script so I'm using flex, but I also tried pgsql with the default style and the way wasn't imported. I thought I was clear but I'll update the question to clarify and add the command line options.

(30 Aug, 10:50) DarkSnow

I'm not familiar with rendering, but looking at the tags, I'd say it's because there's no rendered tags on this way. Probably there are filters to keep only known tags.

permanent link

answered 29 Aug, 19:37

H_mlet's gravatar image

H_mlet
4.7k1573
accept rate: 13%

Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×258

question asked: 29 Aug, 14:57

question was seen: 170 times

last updated: 31 Aug, 11:48

powered by OSQA