There is one Lawgiver, who is able to save and to destroy. Who are you to judge another? James 4:12
James is clearly writing here to the church meaning the believers that he is concerned with the evil that believers speak against one another. He writes that God is the only Lawgiver and the only Judge. He is the only one who is able to save or destroy. God is ultimately the one to whom each one of us will be required to give an account on the day of judgment. If God would have judged us by the law given to Moses, we would have been judged very guilty, but through the grace of God we are all saved through Jesus Christ. As we are saved by God's grace, we should not be quick to pronounce any judgement against.
In the similar manner, when we speak against someone, we are setting ourselves up as a judge. First, we should be judging our own hearts because we cannot call out others' sin without reconciling the sin within our own hearts. Therefore, we must purge evil off of our hearts in order to remove the impurities that hinder our fellowship with God and one another before speaking evil against one another.