You will need to have enough funds in your manager account to cover the withdrawal operation. We encourage you to have at least 1 XTZ in your primary account.
Additionally, you cannot withdraw the entire balance of your KT1 delegation account. You must leave behind at least 0.000001 xtz.