What does it mean to be gathered in Jesus' name? Indeed, anyone can claim to be gathered in the name of Jesus, but are they really? When people adopt heretical views about Jesus, then they are not gathered in His name but gathered in the name of their own personal Jesus which becomes an idol. Thus, they are not gathered in His name and thus are not a part of the Church. However, those who properly understand and experience Jesus are truly the Church. And we believe that the proper understanding of Jesus is only found in the Orthodox Church and her doctrines.
You are correct that the Church is not built upon any man but Jesus Christ, however, it is built upon the faith of particular men. The foundational faith that the Church is built upon is the confession made by St. Peter that Jesus is the Christ, the Son of the living God. All of the Apostles obviously shared this faith, or else Jesus would not have sent them to establish the Church. They built the Orthodox Church upon this confession of faith. So, as much as you Protestants would like to deny it, Apostolic succession is still the ultimate factor in deciding which Church we ought to belong to. The reason being that the Apostles built the Church upon the proper faith, and if we can determine that a Church was established by the Apostles, then we can also conclude that it was built upon the proper faith, and therefore the promise Jesus made that the gates of Hades would never prevail against the Church apply to that Church.
Many Protestants may now try to defend their lack of history by stating that they hold the faith of the Apostles even if they cannot historically trace themselves back to the Apostles, but I disagree with this. If they shared the faith of the Apostles then they would be a part of the Orthodox Church and would never have to exist in the first place. Protestantism was born from schism; from someone choosing to follow their own faith over the true faith of the Apostles. If they had the true faith of the Apostles then they would not exist as their own separate Church but would have remained a part of the Orthodox Church.