I like the idea of what seems a type of "pillar tool" for the pin vice.
Ah,,, brass is a tricky material to drill. It is not as hard as steel, but it's not "soft" in a plastic sense, not to ordinary machining operations anyway. There are several points here…
1) A twist-drill with its normal geometry, including a large rake angle behind the cutting edge, will tend to dig in, causing the chatter you noticed, and likely to spoil the accuracy… as you found.
A cutting-edge for brass should have no rake, or more precisely a rake of 0º; often gained on a twist-drill by very carefully easing the edge back with a small oil-stone. (Same as the lathe tool for brass, where the cutting-edge goes away horizontally from its contact line along the centre of the bar.) It still needs the clearance, but the cutting action is more scraping than slicing.
'
2) If I understand your description correctly, you were cross-drilling a round bar. That can be a bit awkward, but you were correct to make the set-up as rigid as possible. (I assume you bolted the vice to the drill table.) Even with that, a twist-drill with its normal edges that will try to screw itself into the brass is even more likely to chatter and wander when you need it enter and leave the work-piece on cylindrical surfaces.
'
3) You may have used too many, too fine increments in opening up the hole. Each drill would have gained little support and guidance from the previous hole, especially as you were drilling a large hole in a round bar only twice the largest bore diameter.
'
4) Bearings are fussy. They are made to fine limits and the hole for them is best reamed, from a hole drilled to maybe 0.3mm smaller than the finished size. On a workpiece like that the reamer will need a bit of guidance so you don't end up with a mis-shapen hole.
I would use the pillar-drill either with a machine-reamer, or to guide a hand-reamer.
12mm reamers are readily available, but you could also make one of the type sometimes called a "Toolmaker's Reamer " for brass. It consists of a length of silver-steel of the intended diameter, with the end cut at a long angle, hardened and tempered. I have not tried them but you see them recommended occasionally on this forum.
'
5) You might in fact have been better boring the hole on the lathe, with the brass held in the 4-jaw-chuck; but this will entail very careful setting-up so the hole goes truly across the diameter. You'd also need ensure the boring-tool has space to exit into so it won't hit the chuck.
I use a simple saddle-stop when I carry out operations like that. It's a just a bar with a hole in the middle, clamped across the slide-ways by the clamp borrowed from the steady. (This is on a Myford 7, which has wide flats on the bed – you'd have to modify the idea to suit other bed-profiles.)
'
6) Starting with over-diameter bar will help but is a bit wasteful of rather expensive material, and gives a horrible interrupted cut with a risk of the tool catching and ruining the work. (A fine self-acting feed with light cuts will overcome much of that problem.)
hope this lot helps!