i have a extremely large xml-file - which is derived from the field of geo informatics. i got it from a German subsite or the OpenStreetMap-Project: the Geograpical-Engineering-site that deilvers a weekly snapshot of OpenStreetMap of a certain area: i took the germany.osm.bz2 from here http://ftp5.gwdg.de/pub/misc/openstreetmap/download.geofabrik.de/

For doing some tests with xslt i want to run a request to find out certain entity - let us take for example the restaurants. we want to find out all the restaurants in the area.

now we can run that directly on the bz2 compressed file, that we downloaded - for example if we use the following code:

bzcat germany.osm.bz2 | xsltproc restaurants.xslt - > restaurants,csv

well i splitted the file with xml_split -which is a great perl-module from CPAN.

The problem: with the following xslt-processor i get only bad results - the parsed files werent not parsed enough i only get a minor set of informations when i run the code on a xml-file. see the xslt-processor - and below - a litte data-chunk out of the file i run and parse if you want to check it - just get the little dataset - note it is a splitted file

here you can get it: https://rapidshare.com/#!download|643p12|2523227518|germany-001.xml|100000

Note: see therefore the important lines: xmlns:xml_split="http://xmltwig.com/xml_split" and this one here:

 <xsl:for-each select="xml_split:root/node/tag[@k='amenity' and @v='restaurant']">

Note- you can run a little test - and see how long it takes to parse time xsltproc restaurants.xslt germany-001.xml > restaurants-001.csv

real    0m0.308s
user    0m0.283s
sys     0m0.022s

here we have the xslt-processor that contains the code for parsing...:

<xsl:stylesheet version = '1.0'

    <xsl:output method="text" encoding="UTF-8"/>
    <xsl:template match="/">

            <xsl:for-each select="xml_split:root/node/tag[@k='amenity' and @v='restaurant']">
            <xsl:value-of select="../@id"/>
            <xsl:value-of select="../@lat"/>
            <xsl:value-of select="../@lon"/>
            <xsl:for-each select="../tag[@k='name']">
                <xsl:value-of select="@v"/>
        <xsl:value-of select="./tag[@k = 'cuisine']/@v"/>
        <xsl:value-of select="./tag[@k = 'wheelchair']/@v"/>
        <xsl:value-of select="./tag[@k = 'website']/@v"/>
        <xsl:value-of select="./tag[@k = 'addr:country']/@v"/>
        <xsl:value-of select="./tag[@k = 'addr:city']/@v"/>
        <xsl:value-of select="./tag[@k = 'addr:street']/@v"/>
        <xsl:value-of select="./tag[@k = 'addr:housenumber']/@v"/>


and here below we have a data-chunk out of the xml-file that we have parsed: see it

<node id="52768810" lat="48.2044749" lon="11.3249434" version="7" changeset="9490517" user="wheelmap_visitor" uid="290680" timestamp="2011-10-07T20:24:46Z">
    <tag k="addr:city" v="Olching" />
    <tag k="addr:country" v="DE" />
    <tag k="addr:housenumber" v="72" />
    <tag k="addr:postcode" v="82140" />
    <tag k="addr:street" v="Hauptstraße" />
    <tag k="amenity" v="restaurant" />
    <tag k="cuisine" v="mexican" />
    <tag k="email" v="info@cantina-olching.de" />
    <tag k="name" v="La Cantina" />
    <tag k="opening_hours" v="Mo-Su 17:00-01:00" />
    <tag k="phone" v="+49 (8142) 444393" />
    <tag k="website" v="http://www.cantina-olching.com/" />
    <tag k="wheelchair" v="no" />

see the results - note there are missing some parts - unfortunatly..

51923772    49.0812534  8.5637183   Zur Talschänke

52040576    49.4635433  12.4287292  Emil-Kemmer-Haus

52141326    49.4144243  12.4143153  Gasthaus Plecher

52623232    48.9293634  8.2722549   Korfu

52664989    49.0435133  8.3919370   Restaurant Zentrum

52754898    49.3243828  12.3618662  Gasthaus Irlbacher

52762875    49.0099641  8.2528132   Langasthof Stober

52765672    50.0082768  9.2139632   Wirtshaus im Frohnrad

52768810    48.2044749  11.3249434  La Cantina

52768816    48.2051698  11.3257964  Indian Palace

52768826    48.2073264  11.3276147  Dorfstub'n

52768830    48.2075968  11.3281055  Le Candele

52774284    49.0319471  8.2888353   Zum Anker

well it is somewhat a problem that i get the results - ive tried alot but at the moment i am glueless why i get the little output - that is totally contrary to the tags i have in the xslt -processor - any idea and hint will be greatly appreciatdd

btw: after all i want to run approx 5000 files that are the result of the split - and subsequently i want to collect all the results in a mysql-database...

here you can get the original-file: http://ftp5.gwdg.de/pub/misc/openstreetmap/download.geofabrik.de ( germany.osm.bz2 01-Apr-2012 14:51 1.7G )

and here a splitted one: https://rapidshare.com/#!download|643p12|2523227518|germany-001.xml|100000

i have to refactor the coed -so the question - is - how can i get the mysql-results on a efficient way?

update:thx to the first answer in this thread i startet to refactor the code - but still lack of some better results. i have to retry it again..lots of changes were suggested - i did a quick walktrough on the xslt-parser: with the first trial of refactoring i got some funny results. But i will try again - i go trough all the xslt-processor-code and have a closer look if i find the errors and finally i try to refactor all the xslt-file. - any pointers and subbestions or code-snippets are greatly wellcome.

Well to me it looks like our ./tag[@k = '???']/@v xpath should be somewhat ../tag[@k='???'], because our context node is our original matching tag element, not the node element.

we should consider somewhat changing the context node to make this code clearer and avoid errors like this:

<xsl:for-each select="xml_split:root/node[tag[@k='amenity' and @v='restaurant']]">

Then we can use XPaths like select="tag/@id"andtag[@k='country']/@v`.

But above all we should consider refactoring this code to make better use of template instead of for-each.

what do you think - any pointer towards the refactoring is grealy appreciated

asked 24 Apr '12, 16:33

tagtheworld's gravatar image

accept rate: 0%

edited 24 Apr '12, 16:33

This is not an XSLT help desk. Your problem is not related to OSM except that you use OSM test data; it would probably be better to discuss these things in a forum dedicated to XML processing in general.

You write that eventually you want the data in a MySQL database. Is there any reason why you don't just bin the whole XSLT stuff and import the data into MySQL using Osmosis or something like https://github.com/skyebook/OSMGenerator?

permanent link

answered 24 Apr '12, 19:49

Frederik%20Ramm's gravatar image

Frederik Ramm ♦
accept rate: 24%

hello frederik thx for the answer - well it would be very interesting to import the whole stuff into a mysql-db. at which stage can we do a import into mysql!? At a very early stage of the data-processing,,,,

(24 Apr '12, 20:21) tagtheworld

hello after a quick look at the options of importing i muse bout the db-shemes. Is it possible to do certain requests on the data - if they are imported into mysql - i heard about the problem of consistency and such issues . i will dig deeper into the issues - come back and report all the findings. greeting

(24 Apr '12, 20:41) tagtheworld
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here



Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:


question asked: 24 Apr '12, 16:33

question was seen: 3,665 times

last updated: 24 Apr '12, 20:41

powered by OSQA