Whatever size you use, it needs to be held rigidly. Assumes that the tool has been ground properly.
For most of our work, clearance angles are not too vital. A lot of the time, anything between 5 and 10 degrees seems to work for me. I avoid large angles, since that removes metal from behind the cutting edge, that could be used to conduct away heat, prolonging tool life
If it is flexing, surface finish and accuracy will suffer. The least that you will get is chatter.
As to not being on the centre height, unless it is, the tool will not cut properly (If at all when above, just rub)
Have done a lot of work with 1/8″ ,1/4″ and 5/16″ toolbits in holders that are as hefty as possible, and carbide tips, again in holders that are as rigid as possible.
Howard