They weren't used like that. Mortars are short-range weapons used to lob a heavy explosive shell over an obstacle. They were mostly used against fortified positions and to do that were themselves positioned under cover. Not an easy target for small arms.

This example from the Crimean War shows a mortar dug in below ground level.

3 or 4 others would be nearby and arranged to drop shells into a particular part of a fortification a few hundred metres away.

This is what a gun-battery looks like after it's been mortared. It's guns, which would far out-range a mortar in the open, can be seen to be well protected against incoming horizontal fire, but they are highly vulnerable to shells falling from above. The effect depends on the shell exploding inside the battery: it doesn't matter that the shell travels so slowly you can see them coming. Note that conventional guns cannot counter-attack entrenched mortars.

Getting mortars close enough to do this damage was arduous and dangerous work; the technique was dig a long series of trenches angled to prevent infiltration and then man-handle the mortars into position. Could take months.

In the classic age of siege warfare it was common for fortresses to surrender as soon as the mortars were ready to open fire. The fortress was considered to be indefensible and gentlemen did not kill each other once honour was satisfied. In the event that a fortress had to be stormed after dishonourably refusing to surrender, the rules said it was OK to slaughter the entire garrison. What fun!

Infantry have a rather different role, and their light weapons aren't of much use most of the time. Small-arms haven't decided a high-intensity operation for over a century; the other guy has to be softened up considerably before it's safe to get close enough.

Not now maybe, but we're looking at the mid C19! I seem to remember smallarms had some sort of a role at Gettysburg, for example…

Probably the most effective use of big mortars was shipborne on "bomb vessels" to attack harbour fortifications in the late 18th/early 19th C.

But hey, I don't wanna spiral off-topic too far – it's an outstanding model Richard's made.