Christ has the authority, and He told His Apostles that He would "rubber stamp" their decisions! He essentially said "whatever sins you declare forgiven, I will with My Power forgive, for I trust you." So while this isn't Him necessarily saying that they are the agents who can forgive sin, they have become de facto forgivers of sin through their relationship to Christ. We all like to nit-pick on certain issues; sometimes, though, it's best not to.
While you may think it is "nitpicking", ask any cleric if he has personally forgiven any sins, and I guarantee the answer will be no.
From the OCA "Q&A" website: http://www.oca.org/QA.asp?ID=154&SID=3
"We do not confess "to" the priest; rather, we confess to God "in the presence of" the priest who, as the prayer before Confession clearly states, is God's "witness" and who, having witnessed our confession of sins offers pastoral advice on how we can better our lives and overcome the very things we can confess. Just as one would not attempt to diagnose, much less cure, one's own physical ailments, so too one should not attempt to diagnose, much less cure, one's own spiritual ailments. "
And here's the verse to which you refer: John 20:22-23
And when he had said this, he breathed on them, and said to them, "Receive the Holy Spirit. If you forgive the sins of any, they are forgiven; if you retain the sins of any, they are retained."
It's only through their reception of the Holy Spirit, by the grace of the Lord, that the apostles forgave sins. But still, they did not personally forgive them. Christ forgave sins through them. The same applies to our priests. They don't personally forgive sins. They are forgiven through the priest by the power of Christ, as evident in the prayer recited by the priest after you confess:May Our Lord and God, Jesus Christ
, through the grace and bounties of His love towards mankind, forgive you, my Child [Name] all your transgressions. And I, an unworthy Priest, through the power given me by Him
, forgive and absolve you from all yours sins."