Hi Chris, I'm a bit late to this thread.
I use Sherline 6mm and 10mm end mill holders all the time.
Like you, I initially got some cheap cutters that had an undersized shank, a sloppy fit with lots of run-out.
Now I get all my end mills from the "CNCpoorboy" ebay shop. Select their HSS Cutters and put them in price order, low to high.
You will see a whole range of plain flatted shank end mills of all sizes and flute count. I get the 3 flute type and use them on silver steel, mild steel and brass with success.
They are generally British made, Osbourn or Clarkson and they fit in the holders with a nice satisfying push fit. I have yet to break one.
Under 6mm are less than a fiver with postage and always very quick delivery. They are also good for quality 4 facet stub drills and carbide cutters. I've got a few reamer from them as well.
I have no affiliation with the company but I've used them for about 3 years now and everything I've got has been top quality.
Cheers
Steve