Sometimes the most detrimental thing to your Iman is setting unrealistic expectations for yourself. Set reasonable, achievable goals to avoid crushing your faith.
Once faith enters into the hearts, there is no way a person would easily give that up. So why would someone leave their faith? The answers are complex.