Yes, as Mechman says you've got into the work hardening problem, where the tip of the drill skates on the surface of the hole and no amount of rotation will progress the drill forward. Pressure is your friend, by which I mean drilling a smaller hole. But then the drill is (disproportionately) more fragile. If it cuts through the hardened area (which is a big if) it's likely to snatch and break, and that's if you can avoid putting side forces on the drill, which will also break it.
Having drilled a pilot hole it's not guaranteed you can drill it out to size, that material is going to grab the drill and snatch again, always supposing you can get under the hard skin.
Cutting oil or coolant would have helped, but are now irrelevant. You needed pressure on the cutting edge to keep it cutting. Once it skates you're lost.
I wonder if you would be better off going abrasive. Having got the hole area work hardened I'd rather have a go with a carbide bur in a Dremel kind of tool. Keep the area wet to keep the tool from burning out, but you should be able to make progress, if slowly. Once you've got a hole open it out – if need be with a stone point (they sell grinding burs for sharpening chainsaws – they have all sorts of other uses).
Carbide drills or carbide tipped drills both need lots of pressure, so you're fighting the inevitable trying to use them in a hand drill. And, as you say they are very brittle so sideways pressure will shatter them.
Good luck
Simon