Thank you everyone, I think that I have solved the problem by reading the answers here and on another forum.
Just clear up a couple of points which I forgot to mention’
1 I am using a jig to keep the drill bit in line
2 yes, I have been pecking at the work to clear the swarf from the bit
3 I am drilling at a fast speed (as fast as I can on the milling machine.
Now for the reason I think that I have been having which is so simple, but until someone mentioned it I missed.
And that was, I am trying to drill right through with a thread in the hole I am trying to cross drill and the thread is causing the bit to wander off course and breaking the bit.
I will try again with the remaining two #76 bits wish me luck
And thank you again for all the help gents