A drill will cut oversize if
1 ) It is ground so that the lips are not of equal length.
This means that the intersection of the lips is not on the centreline.
Consequently, the drill will rotate about a centre point that is not on the centreline of the lathe.
2 ) If the drill is not held on the centreline of the lathe, because the drill chuck is inaccurate, the Tailstock barrel is worn, or the Tailstock is off the centreline of the lathe either vertically, or more likely horizontally, the drill will,effectively be used as a boring bar to cut an oversize hole.
Any of these faults is likely to encourage the drill to wander and produce an out of line hole, apart any flexing of the drill.
As each fault is eliminated, accuracy should improve.
Howard