I get why you would think that, in theory I suppose you are right. However in practice what normally happens is that the tool flexes and gets pushed inwards as the cut progresses and you end up with a tapered hole.
If you turn up a test bar with a normal tool on the outside diameter, lets say nominally 1" diameter and 4" long, then measure its actual diameter at each end, unless your lathe is very well set up you could easily find a thou or two difference in diameter across the length. Well this will also happen when you are boring a hole, as the conditions are just the same, but if you spin a cutting tool it does not matter that the saddle and spindle are not 100% in line, the tool will follow a circular path and the saddle will just be moving the job along past it. You only have to ensure the job is set up square on the cross slide.
A lot depends on the depth of hole in relation to its diameter, I read somewhere that anything over four diameters deep and you will encounter the problem of boring tool flexing.
Every job has to be taken on its own merits, there is no hard and fast rule to follow
For a relatively short depth hole, more often than not I will drill a hole somewhat under size, which could potentially have wandered off course due to the drill not being correctly sharpened or flexing off centre. I will then go down the hole with a boring tool to bring it back concentric again, and finally finish it with a reamer.
But for something like an engine cylinder bore, the between centres bar is the way to go, and even after boring it to a few tenths under size I will make an expanding hone/lap to finally finish it to size (I DO NOT mean those three legged glaze buster contraptions).
Howard mentioned he was making some tooling that is going to be used on the cross slide, in that case it would be a no brainer for me to machine it with a between centres boring bar on the cross slide where it will be used, it has to be bang on centre height and truly round if done that way.
Hope this helps, its a case of horses for courses.
Phil