I too was once troubled by what I saw as proof texts for the papacy showing up in the fathers. I don't want to write a treatise because it's already been done far better than I could do it... download Abbe Guettee's work above. But here are a few things I can throw at you, that you might be interested in knowing.
1) "Peter has spoken through Leo!" was the acclamation of the 4th Ecumenical Council, convened in Chalcedon. What were they saying? Well you probably know that St. Leo had written a tome (known as the Tome of Leo) which properly expounded what St. Cyril had said decades ago, concerning the one person of Christ who exists in two natures. What you may not know is, this was done after a very long deliberation (you can find copies of the acts of the council of chalcedon that discusses what went on during their meetings), comparing St. Leo's writing to the previously approved (at the 3rd Ecumenical Council in Ephesus) 12 Chapters of St. Cyril. Also important to understand is what precisely is meant by this acclamation, "Peter has spoken through Leo!" Aside from the fact that the pope was the first among equals and the successor of the apostle Peter (as is the Patriarch of Antioch), this statement is even more intriguing. If you read the life of St. Leo, you will find in it the story of his prayerful entreaty to the apostle Peter. St. Leo wished to defend the Orthodox faith of the unity of Christ and the duality of his natures without lapsing into either monophysitism or nestorianism, so he writes his Tome, and places it upon the tomb of the holy apostle, and prays and fasts for 30 days, asking Peter to correct any mistakes it might have in it (so wait a minute, why didn't he just speak ex cathedra and be infallible?? hahaha). At the end of the 30 days, Peter appears to Leo, telling him that he had in fact corrected the errors and so it could be sent along with the papal legates to Chalcedon. That, in short, is why the fathers of Chalcedon could indeed exclaim, "Peter has spoken through Leo!"
2) The Pope of Rome did in fact occupy a very important place in the governance of the Church. As first among equals he had the privilege of hearing disputes and recommending courses of action to be taken. What he did not have, however, was universal jurisdiction in the Church. We find Pope St. Gregory the Great fighting vehemently against anyone receiving the title "universal bishop." He even goes so far as to say that anyone who takes that title is the "forerunner of antichrist"!!! The papacy of Rome was revered for its Orthodoxy, especially during the 7 century battle of the Church to defend Orthodox christology. But were they somehow looked upon as infallible heads of the Church militant? God forbid. On the contrary, the whole idea of a council precludes the notion that somehow the pope could teach and act infallibly. There were a couple heretical popes that were anathematized at ecumenical councils, specifically Pope Honorius, who subscribed to the crypto-monophysite doctrine of monothelitism-- he was anathematized at the 6th ecumenical council.
3) Primacy of honor was not something inherent to Rome itself, simply because it was the See of Peter. St. Cyprian of Carthage and many others confirm that the "chair of Peter" is given to all Bishops, who all have the power of binding and loosing, and who all are charged with preserving the apostolic faith. Rome was a focal point of unity, being in reality the place of the martyrdoms of Peter and Paul, and a relatively safe place in the backwaters of the Empire from theological innovation. This changed, sadly. But this primacy was accorded to the bishop of Rome by his brother bishops. It was the ecumenical councils who affirmed the primacy of honor of the "apostolic See" and ironically it was also a later ecumenical council that afforded "equal privileges" to the apostolic See of Constaninople (see the canons of the 5th Ecumenical Council). Rome was first place by the universal consent of the Orthodox Catholic Church.
I encourage you to google "Joseph Suaiden" and look up his apologetics webpage-- his replies to James Likoudis are very well written and more than prove the truth of the Orthodox faith.
Hope this helps.