Better to stick with a tight fit on both parts for the dowel hole to give positive location.
Then you can grind the end off the dowel pins and drill and tap them so you can screw a high tensile cap screw in there for extraction, either with a simple puller made from a bolt and piece of tubing or a slide hammer than attaches via the threaded hole.
Or you can drill and tap threaded holes in one piece being dowelled together to use with jacking bolts to get parts apart.
Or drill the dowel hole right through the two parts but ream only part way through, so dowel sits in both parts but not all way through. Then you can knock the dowel out from the other side using a punch.
An old trick to get a reamer to ream oversize was to put a piece of paper lengthways along one flute of the reamer so it covers one cutting edge of the flute. Seen it done with machine reamers, not sure about hand reamers with the lead-in taper though.