I saw a Catholic that explained it this way.. and I'm sure the Orthodox on here will disagree. I don't know if he's right or not, but he seemed to make sense:
"As far as I can tell, the distinction that the modern Orthodox make between "primacy" and "supremacy" did not exist as such in the early Church. Words and concepts change meaning over time. The Early Church Fathers would not have needed to make a distinction, as the meaning was already understood.
For example, the word "man" in the sense that it was used to speak of humanity in a general sense, but over time acquired a more restricted meaning. Now, we say "humanity" instead, and use "man" mostly as a synonym for "male".
My best guess is that for the Early Church Fathers, elaborating on primacy was unnecessary. It would be like a scientist elaborating on why mathematics is useful to prove a theory. It is assumed that mathematics can be used as a method of proof. The scientist could explain it, but it would be superfluous."
Similarly to how a lot were silent on the Trinity, until Arius and others started attacking the divinity of Christ. Why can we not place the Papal claims in the same understanding?