A residual current detector (RCD) is basically a current transformer. Normally the output of the secondary is proportional to the current in the single turn primary winding. However, if two wires are used then the output will be proportional to the algebraic sum of the currents in both wires.
For a single phase system both live and neutral wires are used. If there is no leakage the currents in the live and neutral should be the same magnitude but opposite sign. So the magnetic fields will cancel out, and the output of the current transformer will be zero. If there is leakage to earth then the currents will not be equal, the magnetic fields will not completely cancel, and there will be an output from the secondary.
The same principle applies to a 3-phase RCD. There are two cases, with and without a neutral wire. Consider a 3-phase system connected in star but with no neutral. Applying Kirchoff's current law to the star node the algebraic sum of the three currents is zero. Hence the algebraic sum of the three phase currents is also zero. Whether the system is balanced or not is irrelevant. In this case a 3-phase RCD simply has all three phases running through the current transformer. Intuitively the same is true for a system connected in delta, but I don't have the inclination to prove it mathematically.
If we now look at a 3-phase system in star, with a neutral line, the same applies. The algebraic sum of the the currents at the star point, including neutral, is zero. In this case four wires (three phases and neutral) are run through the current transformer.
So a 3-phase RCD may have three or four wires, and they will work irrespective of a balanced or unbalanced system.
Andrew