The primacy is one of honour only, and depends on the Orthodoxy of the holder of the see of Rome.
As far as I can see Leo of Rome had already gone beyond such ideas of fraternal primacy, and was the first bishop of Rome to attempt to impose a universal jurisdiction and the first to teach the filiioque.
When St Dioscorus became Pope of Alexandria, Leo sent him a letter instructing him that he must change the practices in Alexandria so that they exactly followed those of Rome. Such primacy was never accepted. Likewise, when he issued his Tome and expected that it would simply be accepted as the final word on the matter since issued by the bishop of Rome, it was in fact criticised because it did not seem to express a Cyrilline Christology.
There was a sense that Rome should have borne witness to the Apostolic faith, and when it did it could expect to hold a primacy of honour, but if it did not hold to the Apostolic faith then any sense of primacy was redundant.
During the persecution of the Church under Justin and Justinian, St Severus was effectively, and in terms of spiritual authority, the primate of the non-Chalcedonian communion and when in exile in Egypt was considered the de facto primate of Egypt as well as Syria, being looked on as the leader of the non-Chalcedonian communion.
It seems to me, as a personal opinion, that in some sense the primacy has a charismatic element and is not bound to a particular see but is a charism of the Holy Spirit which becomes manifest when the primacy is needed for the sake of the Church.
Father Peter