Again, I would kindly disagree.
He is not wearing the mitre for "himself", but, for "Christ".
Correct me if I am wrong, but, I do believe that during the Liturgy the bishop, who is Christ's disciple wears the mitre. However, at key moments, he takes it off. Why?
For example, while reading the Holy Gospel, he takes it off, because we are listening to Christ's teaching. During the Great Entrance, he takes it off, etc.
These moments simply qualify that the mitre is not for the man who is wearing it, otherwise, he would be wearing it all the time.
I think if a man is called to be a bishop, and volunteers, than he ought to carry out all the duties of a bishop, whether he is comfortable with it or not.
Same holds true for clergy.
When someone bows to kiss the hand of a priest, it's not so much the "man" they are kissing, but, the office that he holds. With his hands he distributes the Eucharist. We are told that if we meet an angel and a priest along the road, we are to first greet and kiss the priest's hand before acknowledging the angel.
Therefore, the priest who removes his hand before it is kissed is doing his office a disservice. While I completely honor the humility this shows, truly....I still think it is wrong, because the person isn't kissing Joe, or Tom, or Mike who is wearing the fancy robes. They are kissing one of Christ's anointed priests, and the hand through which Christ Himself blesses them.