Cancelling visa and getting passport to work for a new employer

Dear Lawyer,

I am cancelling visa with my current employer; they hold my passport for more than a week and my new employer is waiting for my passport to start new visa process.

Now, my previous employer is telling me I have to pay all the penalties and bring the contract to Immigration (it was not said in the contract I have to stay with the employer for a specific amount of time).

On top, I only signed offer letter and not contract, then they issued my visa without a contract basically. Now, they say I have to pay for their mistake.

Can you please advise about my rights with this concern?

Thank you!