How to convert .osm to .shp files?

Try using QGIS - Just import osm file using OpenStreetMap plugin (which comes with the default QGIS software).

Then left click the layers and save as shp files.

The only problem I've found is that some tags I think are truncated to 256 characters I think. But most tags aren't this long anyway.

I'm guessing (from the .NET error) that you're using the Google Code osm2shp and (from the fact that you're running Windows XP) that you haven't got much memory on your PC. The page that Gnonthgol referred to describes several alternatives - I'd read up about those and use one of them instead (at least one seems to support a temporary local database rather than in-memory storage which might be better for you).

You'll also probably need to think about what features you want in the resulting shapefiles, and try a small area first.

One final question - have you checked that the zipped shapefiles that are available for areas from e.g. here don't do what you want?

For the first there are several programs named osm2shp. I do not know what you have problems with. There is an overview of different programs on the wiki.

The error message System.OutOfMemoryException indicates that this is a .net application. You may find information on how to resolve it by searching the error. Or try another application based on a better framework.

Whenever you are working with big amounts of data like the planet file, you are going to use a lot of memmory. Use a computer with enough memmory and a 64-bit address space.