How to download(or extract) only the polygon within a administrative boundary?

Firstly you need to generate the .poly file. The easiest way is to download the City of Melbourne boundary in JOSM (probably a relation, in which case ensure you download all its members) and then save the downloaded data as a .poly file (option on File|Save... menu in JOSM).

Secondly use an extract, say of the whole of Australia or Victoria and use osmosis to obtain the clipped Melbourne data. osmconvert also does this and is a little less temperamental than osmosis.

You can try the service of OSM user Wambacher to find and download the administrartive boundary of any region / city via https://wambachers-osm.website/boundaries/

Try to find your place via search funtion or tree structure.

Then choose correct export format for your boundary.

Success?

Adding to what SK53 and stephen75 suggested.

You can use http://polygons.openstreetmap.fr to generate the .poly file. But you'll need the relation id of your polygon(for Melbourne, its 2404870). Save the generated .poly file and use it with osmconvert and geofabrik's Australia extract to get the data of only the Melbourne city.

Alternatively, you can directly get the Melbourne city data from Mapzen metro extracts. And MapIt Global provides the boundaries of areas covered by smaller city boundaries like Melbourne whose areas can be further used to get relation ids to get the poly files and then extract data from Melbourne using osmconvert.

Here's an Overpass query that returns all buildings in the City of Melbourne:

area
  [place=municipality]
  [name="City of Melbourne"];

way(area)[building=yes];
out;
>;
out skel qt;

You can then download the result as GeoJSON from the "export" menu.