Printing a high quality map

Likely you don't want to start with a osm file. But even if … See the many other questions about printing: https://help.openstreetmap.org/tags/printing/?sort=mostvoted or https://help.openstreetmap.org/tags/print/?sort=mostvoted

I have downloaded a map section and would like to take it to a commercial printer to print in colour, large size eg. 40x50cm to be included in a guide book. How do I open the .osm file? And possibly, how do I convert into a conventional format?

You will want to use a renderer to convert the OSM data to a PNG. The most popular way is with osm2pgsql + Mapnik, but other options like Maperative also work. Your printer should tell you what resolution they want, but it's probably about 600 DPI.

I'd try using Tilemill to export to a PNG, with a scale factor set. You should be warned that the openstreetmap-carto, the default style used on the OSM front page, is not well suited for print for a variety of reasons.

You can try the Print plugin for JOSM: http://josm.openstreetmap.de/wiki/Plugins