Perhaps start by reviewing the documentation for the chosen inverter – that will often include guidance on how much space to leave around the inverter when mounting.
A metallic enclosure will to some extent act as a heat sink – but depending on size of enclosure it may still be necessary to arrange ventilation of the enclosure. Typically 2 grilles, and perhaps a fan. Some inverters include a fan on the integral heatsink that might help stir the air around any enclosure.
A metallic enclosure is generally preferred for EMC reasons as well as thermal management.
Some (not many) inverters are designed for installation without a protective enclosure, but most are designed to be mounted inside a suitable control panel or other enclosure.
Ventilation grilles, dust filters, fan modules etc are typically available as accessories for electrical enclosures.