I believe the principle that supports what the Fathers have to say about it is that sex, to be whole, must have the whole of the each of the two persons involved.
By whole it means heart, imagination, attention, presence, feelings, desire, past, present and future.
The several forms of sexual sins basically involving not engaging one or many of the parts of the whole of us. We deny our partner our past, we deny them our future, we deny them the fullness of our desire, sharing it with other people, we deny the fullness of our imagination and feelings by commiting adultery in mind and in practice.
Also, we tend to see the partner as just part and not a whole. Fetishes, "consensual unchastity" - which is the norm today, seeing the other as just "a very intimate friend" with whom you share a pleasure not fundamentally different from sharing a intimate conversation, just more fun. Because this is wrong vision that debase human dignity, it is also sinful.
Plus, spiritually, we become one flesh with everybody with have sex with- at least that's how I understand the biblical teaching to be, and the ultimate reason why sex outside marriage is a sin - and that is a reason even stronger than the problem of dignity and why it is a sin of the flash. If we fornicators - and 99% of us modern people are, were or will be - could see ourselves spiritually, we would see a mass of flash from deformed bodies grotesquely united. Only repetance, baptism and confession can heal us into health individuality or to the proper union with a spouse.