This does not help with easily removing the batteries, but might help slightly:
Could you engineer a conventional switch onto the battery cover ?
If projection from the battery cover is not an issue, you could mount a slide on/off mechanical switch in a small housing fixed to the battery cover and wire it into the battery circuit – assuming that you can get at the wires from the battery terminals.
If the battery terminals are soldered directly into the PCB, you will have to get more inventive…….
For example; A plastic pin or blade inserted through a hole or slot drilled in the battery cover in just the right place could be used to push one of the batteries away from its positive terminal in the battery holder against the spring on its negative end, breaking the electrical circuit. Attach the plastic pin to the unit with a small lanyard so as not to lose it. When you next want to switch the unit on, you just pull out the pin and the battery will contact the terminal again.
The thing about remembering to regularly check if the batteries are dead or remove them after use every time, is that being human, we will almost certainly forget !
If you still want to physically remove the batteries, the stud and wing-nut idea sounds best.