Voltage Control Mode Stability issues

The point of Dmitry’s suggestion is that the controller may be stalling due to hardware protection kicking in. When that happens, the FET driver will temporarily override the control outputs issued by the microcontroller, making it very difficult for the control logic to keep tracking the state of the motor. The controller may not be able to recover after the protection is deactivated, causing the motor to stall, which then results in the error code you specified.

One way to check if this is indeed the problem is to simply increase the hardware protection threshold. It is set using a resistor on your custom ESC board:

The threshold current should be at least two times the peak phase current of the device to avoid unintentional activation of the protection. See also:

This appears to be a case of integrator windup caused by suboptimal tuning. If you want the motor to always stop once a zero setpoint is received (regardless of the outer control loops, specifically the RPM control loop), set ctl.hard_stop = true.