Learned Orthodox know we don't worship the Pope or think he IS the church. Both well-meaning conservative Vatican II Catholics and liberals fall for the latter, thanks in part to the modern media's focus on the person of the Pope. Conservatives fall back on it because without the traditional liturgy they have no other identity anymore; liberals don't believe in church infallibility so they think the Pope can order the changes they would like.
The argument against the papacy I respect is that of the old Anglican high churchmen, the Tractarians: patristic, very conservative, basically afraid that the Pope's authority, such as the development of doctrine (only an opinion, not our doctrine!), is Modernism's cousin. "He can change anything!" No, he can't.
I've said that I think a lot of the Orthodox convert boomlet 20 years ago would have been High Church Anglican 100 years ago: more accessible than Eastern Orthodoxy in the West at the time, no cultural barrier, etc.
The rebuttal to that principled position: the Episcopal Church, hardly more conservative than Rome. Rome teaches now what it did then, in full.