I’ve amassed a few different options.
Some finger collets that allow far less stick-out, so less chatter potential. I’ve not used any of those for years.
ER32 seemed to be the largest size for general work, so that is what I bought (lathe spindle will accept 25mm). Since then I have purchased a through-collet chuck for use on the lathe. Useful because other sizes than the few standard shank-size collets that get used with the mill.
I bought a set of ER16 for finer work – particularly on the Raglan.
I needed a larger diameter holder for a particular job so bought an ER40 and just two collets. Been used twice.
A Clarkson came with the lathe and gets used occasionally – but not often. Useful when both machines were set up. Most of my cutters have plain shanks.
I have no Sheldon-type holders and cutters.
One day I might rationalise the whole lot and sell the others. The ER system would be retained.
I have both square and hexagonal Stephenson’s ER32 collet holders. Useful, when needed, but not used often.
I don’t make models. Most of what I do is for old machinery restoration/repair.