Probably, the speed control potentiometer sets the speed of the motor.
If you have a readout, it should indicate Mandrel speed, rather than motor speed..
Changing which pulley combination is in use will only change the speed and torque available at the Mandrel..
Lower speed, belt / pulley set up will give higher torque at the Mandrel
If the problem persists, you need to start looking to see where in the drive train the rotation ceases when the motor is running. That is the point at which you need to investigate..
Just be careful with rotating parts when the belt cover is open!
Howard