There are a couple of things that will give you some insight into where I am coming from.
It appears to be the case that the Protoevangelium of James (PoE) is pseudepigraphal.
For those of you that don't know how memory works it is something like this (in my current understanding). An event happens and we "file" away our understanding of the event and sensory experience associated with that event. When that memory comes to mind we are actually reconstructing the memory from those things, it is not like replaying a video recording of the event. In the process of mentally reconstructing what happened what we experience by way of our memory takes the form of what we think must have happened according to our interpretation of the event. It is at this point that we are somewhat suggestable and will remember certain details as being true of the original event when in fact they were not. Or, another way of putting it is that details will added to fill out the narrative based on what we believe must have happened and this can be for any number of reasons.
Where this comes into play is that given the circumstances of it's documentation by PoE (as I understand them at this point), and the way memory works this looks to me as being a case where the details of Mary's remaining a virgin for the rest of her life were added (in the corporate memory of the church) to her being a virgin at the time of Jesus' conception by the power of the Holy Spirit because it is thought that it must have been the case.
Again, I'm not saying that this is what happened, but to me it casts a shadow of doubt about this literally being the case. Along with that I have no hang ups about it or believe that it is weird.
Perhaps what is the most pertinent point to me is if a story like this, given the actually support that it has from a purely historical point of view, were put forward by a member of another religion about the basis of their religion Christians would target that and use it against them in a discussion of that faith. I believe very much that no religion, Christianity included, should be given a free pass from inquiry.
That is why I have no trouble seeing it as a story that is profoundly theologically true while it is the case that it may or may not have been literally true. In fact, I have no trouble with the idea of submitting to tradition while having actual doubts (if that makes sense).
Please note that I do not mean to give offence to anyone or to trash the Mother of my Lord. So, please forgive me if you think I have done either of these things!
In Orthodox prayers and hymns, the Virgin is never referred to liturgically as a
wife, only ever as a
Mother. As the woman who literally conceived and gave birth to God Incarnate, any approach by another man can only be regarded as unseemly, even an act of sacrilege, and this would include St Joseph, her betrothed. This is not at all to denigrate or belittle St Joseph, far from it.
Consider Joseph's situation: Joseph would have been familiar with what we call OT scripture. Exodus in particular is stuffed full of terms and imagery which we know are prefigurations of the Mother of God. Mary bears the Root of Jesse, the Bread of Heaven (John 6), the Word of God (John 1). The Ark contains the rod of Aaron, Manna and the Law. Mary is the human Ark of the New Covenant, a constant motif in both liturgical language, and in the iconography of all the feasts of the Virgin (the four-posted structure with a domed roof).
Now, Joseph was a good Jew, he would have been brought up with a strong sense of the sacred. He would have been raised knowing the stories in scripture of people touching the Ark of the Covenant and suffering instant death. He would have also known that only the high priest dared enter the Holy of Holies of the Temple to offer the yearly sacrifice to the presence of God who "dwelt there". Undoubtedly at some stage Joseph would have been inspired by the Holy Spirit to realise the true meaning behind these images and stories from scripture, as well as the temple rituals.
Once the meaning of these became clear to him, how, then, could Joseph possibly consider marital relations with this woman, the living Tabernacle, the new Ark, the Holy of Holies, knowing that she has given birth to the Son of God? Not that sex is bad, evil or wrong between married couples, just as eating and cooking meat are not bad, evil, or wrong in themselves, but when put into service to God in the Temple, be it sacrificial animals, or, in the case of Mary who was dedicated to the Temple as a child, they became holy, and only the high priests could participate in the sacrifice. Christ Himself is the great and eternal High Priest, the "prince who eats bread before the Lord" (Ezekiel 44). Good man that he was, Joseph would most likely have regarded himself as utterly unworthy to even be in the
presence of such a treasure blessed by God, let alone consider sleeping with her.