How to claim the security deposit from the landlord after leaving the property?

A landlord agent, after failing to live up to the terms of my monthly rentals agreement, including not making repairs, said I could vacate the property before the end of the lease if I want and he will return the deposit.

After I moved out, he turn around stiff me with my deposit claiming that I broke the lease for moving out before lease expiration. What is my legal recourse?