A slot-drill rather than end-mill, surely, to cut those big slots?
(Use the end-mill for the block's exterior.)
It's a neat design but I think I'd prefer a rectangular block to support the tool-holders' full length. I'd also worry about unwanted rotation when slacking the device to change holders; and whether the taper would release as intended.
Still, try it and see how it all works out.
Regarding putting the height adjuster at the back (towards your side of the tool-post) rather than the front, I think that is the right approach. Mechanically there is no difference, but the less clutter near the tool itself the better.
You might in the end find a separate spanner more of a faff than a handle, so if 'twere I designing this, I'd do so to keep my options open – allowing simply swapping nut with handle or vice-versa. It may prove possible to adjust the handle's operating arc by washers to keep it in the optimum position for your use.