Some food for thought:
As I mentioned before, the Virgin is never referred to liturgically as a wife, only as a mother, and the Mother of our God, no less, as the hymn It is truly meet, and any number of Orthodox theokia, say. Therefore, any approach by another man, including Joseph the Betrothed, would be seen as unseemly. This is not at all to denigrate or belittle Joseph, far from it.
Consider Joseph's situation: Joseph would have been familiar with what we call OT scripture. Exodus in particular is stuffed full of terms and imagery which we know are prefigurations of the Mother of God. Mary bears the Root of Jesse, the Bread of Heaven (John 6), the Word of God (John 1). The Ark contains the rod of Aaron, Manna and the Law. Mary is the human Ark of the New Covenant, a constant motif in both liturgical language, and in the iconography of all the feasts of the Virgin (the four-posted structure with a domed roof).
Now, Joseph was a good Jew, he would have been brought up with a strong sense of the sacred. He would have been raised knowing the stories in scripture of people touching the Ark of the Covenant and suffering instant death. He would have also known that only the high priest dared enter the Holy of Holies of the Temple to offer the yearly sacrifice to the presence of God who "dwelt there". Undoubtedly, at some stage, Joseph would have been inspired by the Holy Spirit to realise the true meaning behind these images and stories from scripture, as well as the temple rituals. Once the meaning of these became clear to him, how, then, could Joseph possibly consider marital relations with this woman, the living Tabernacle, the new Ark, the Holy of Holies, knowing that she has given birth to the Son of God? Not that sex is bad, evil or wrong between married couples, just as eating and cooking meat are not bad, evil, or wrong in themselves, but when put into service to God in the Temple, be it sacrificial animals, or the little daughter of Joachim and Anna dedicated to the Temple as a child, they became holy, and only the high priests could participate in the sacrifice. Christ Himself is the great and eternal High Priest, the "prince who eats bread before the Lord" (Ezekiel 44), a reading which is one of the standard OT readings at Vespers for feasts of the Mother of God.
Good man that he was, Joseph would most likely have regarded himself as utterly unworthy to be in the presence of such a treasure blessed by God, let alone consider sleeping with her. There is no need to invoke the Essenes, nor, as some have tried to say, a midrash or allegory, to explain the absence of consummation of the marriage of the Virgin and her betrothed. The truth is far richer and deeper than these pale imitations.