Do you have a schematic you can share?
SCR power controllers normally use phase angle control. This needs a capacitor to provide a time delay. We would need to know the capacitor value as well. If the design purely relies on a voltage divider to set the gate voltage in relation to the input level you will not get very good control/ For a start the gate trigger point is not well defined, do you have a Diac or zener in the gate circuit? It will also be succeptable to spikes and noise.

What country are you in and what is the mains voltage? I don’t understand why you would have a bridge rectifier ” SCR then a diode bridge”. The bridge will do nothing as it is being fed with DC from the SCR. Do yo have the SCR across the DC terminals of the bridge and the AC terminals to the supply and load? This turns thr SCR into s full wave device but why bother? Assuming 240V mains half wave rectification will give you enough voltage for a 180V motor.

The potentiometer is not the problem, your control circuit is not matched to the load.
Also note this type of circuit is inherently dangerous and needs carefull construction to avoid shock hazards.

Robert.

Well as the *man said when asked for directions “if I was going there I wouldn’t start from here”
Your set-up will work but is not optimal. Fine if it’s the bits you have though.
We don’t know the cicuit the pot connects too so just changing it’s value is a gamble. That sid the 330K + 100k is not too far out. So should work.
How many connections are there to the potentiometer, 2 or 3 ? If 3 then Rob’s suggestion of a resistor at each end may help but I’d suggest 47k at the “fast” end.
Make sure the potentiometer has a PLASTIC shaft and  use a plastic knob. If the mounting bush is metal either ensure it is fitted to a properly earthed metal panel or connect a separate earth to a lug under the potentiometer munting bush.

Robert.