The Schlesinger limit for the internal taper of a mill is 0.0004" or just under half a thou.
Measuring the tip of a drill or bar held in a drill chuck won't give you an accurate impression, as you are very unlikely to have a drill chuck with accuracy comparable to,. let alone better than, the taper. You should measure the accuracy of the taper by poking a DTI inside.
With that level of error, the reamer will almost certainly just follow the existing hole anyway, it really should be ground.
Neil