Actually, I DON'T know how protestants reconcile the Old and New Testaments. I've heard protestants say that "the Old Testament God is different from the New Testament God," which is just wrong from our Orthodox understanding of it. Can you explain how protestants understand it?
This article explains it pretty well:http://www.parrett.net/~bhcoc/articles/old_testament_vs_nt.htm
The contrast between Exodus 21:23; Leviticus 24:20 and the passage in Matthew 5:38-39 is easily understood if we understand that two very different issues are being addressed. The Old Testament law was given to a nation of people. It represented not only their spiritual law, but also their civil law. For a government to operate it must be able to punish evildoers. Without punishment to act as a deterrent, there would be chaos. Even under the New Testament governments are charged with being a terror to evil works (Romans 13:3-4; 1 Peter 2:13-14). The statements of Jesus are not meant to do away with the government's right and duty to punish evildoers to maintain civil order.
But Jesus wanted it to be plain that his new kingdom was not a civil government. He states this plainly in John 18:36. The church was never meant to be the institution responsible for maintaining civil order. The Catholic church sinned greatly in becoming a political power and in trying to advance their doctrine with the sword.
As individual Christians we have no right to seek vengeance on those who have harmed us. Ours is supposed to be a reflection of the attitude of our Father who is good to all, without respect of persons. This is what Jesus is referring to in the sermon on the mount. But Jesus was not saying that governments should stop punishing evildoers. Those who use the sermon on the mount to oppose the death penalty are misapplying the passage, for it is not addressed to governments or to governmental officials.
As far as the command of God that the Israelites kill the nations in the promised land, there is much we do not know. We know that they were evil nations (Leviticus 18:1-25). we know that this is why God allowed and commanded the Israelites to destroy them. But what we do not know is what God had done to try to get them to repent before. I think it is an unwarranted assumption to conclude that God did nothing to bring them to repentance. Remember God does not change (Malachi 3:6). He is presently longsuffering to usward, not willing that any should perish but that all should come to repentance (2 Peter 3:9). If he is longsuffering now, he was always longsuffering. Just because we do not know how God dealt with those nations doesn't mean he never dealt with them.
The question that you ask as to why God used the nation of Israel to destroy the Canaanites is difficult to answer with any degree of certainty. God often used other nations to punish an offending nation. The prophet Habakkuk had difficulty understanding how God could use an evil nation to punish his own people who had been evil (Hab. 1:1-13).
God does not view violence and killing as always being an evil thing. In fact, if God can punish people without sin, then he can ask his people to do so, also without sin. Perhaps God knew the promised land would mean more if Israel fought for it. Perhaps God wanted the vision of violence to motivate the Israelites not to follow the same sins as those of the land of Canaan.
And remember, there was at least one example of the opportunity for forgiveness even during the conquest of Canaan. In Jericho there lived an evil woman, a harlot. She had heard what God had done bringing the Israelites out of Egypt. She grew to believe in the power of this God. She knew their city was doomed, and she aligned herself with the people of God (Joshua 2). Not only was she saved, she was recorded in faith's hall of fame as one who perished not, because of her faith (Hebrews 11:31). Not only so, but Rahab, the sinful harlot, by her faith became part of the lineage of Christ, and her faith helped to bring Jesus our Savior into the world (Matthew 1:5 -- Rachab).