I've heard some noise from my ex-coreligionists (er...co-churchists...whatever) regarding how my conversion apparently is "a source of scandal" and a "wound to my soul" and the like...can't say it has caused me to lose even a second of sleep or tranquility, but hey, if that's how some people feel who am I to tell them anything beyond "I'm sorry you feel that way, but you are wrong"?
But I can understand where this feeling comes from, on the part of converts. Those who have never been Roman Catholic maybe can't understand the sort of hold being deep into RC ecclesiology can leave on a person's way of looking at churches and communions outside of the RCC. Even with the post-VII fuzzy language, when it is constantly drilled into your head that St. Peter = The Prince of the Apostles = The Roman Pope, forever and ever in perpetuity throughout the universe because that's how Jesus Christ Our Lord wants it to be, I can't be anything but sympathetic to those who find themselves in the unenviable situation of trying to break out of that way of thinking while still being unsure in where they are at present. I've been there myself.
All I can say is that you have to get past it. I still find myself thinking in "Roman" ways sometimes (not about this, but about other things), which surely presents a challenge to my priest in terms of pastoral care, but it really does boil down to what Scamandrius has said: Stop caring. It's way easier said than done, but it is absolutely necessary so that you are not susceptible to these kinds of doubts and worries that keep you from growing spiritually. In the end it is not about Rome's claims one way or another; it is about where God is telling you to be, and how you can respond to that. If it takes a long time to convert (and for me there were about three years between my last RC mass and my baptism into the Coptic Orthodox Church, though I couldn't say if that was a long time or not), then okay. Remembering that it is later than you think, you still have to get to a point within yourself where you trust those who are receiving you to be preparing for your entrance into the kingdom of God, which frankly with all due respect to Peter J, Papist, and any other Roman communicant who might read this, the Roman Pope or his entire church cannot affect by this or that Papal Bull or stance or whatever. Maybe they will say this or that thing that the RCs will argue amongst themselves as to whether or not they reflect true/authentic RC teaching. You have to let them have that discussion, knowing that whatever its outcome is has absolutely zero effect on your eternal salvation, even if they say that it does. Remember that it was heretics like the Donatists and the later some Protestants who either explicitly or implicitly believed that any one person should have such power as to render sacraments void or keep others from God. To give credence to any statement coming from anywhere that would say, in essence, that you jeopardize your soul by leaving communion with one bishop on principle, is thereby to give at least indirect assent to such views. In other words, you may, even after leaving the RCC, still indirectly believe in it to that extent that you frame your theology in direct rejection of it (rather than being separate from it for historical and theological reasons that predate your personal rejection of it), thereby still viewing Rome as the sun around which Christianity and all churches and communions are to orbit. I'd say that's a much bigger danger than some Catholic layperson or priest saying that you're naughty and God is going be mad at you if you leave Rome, as I have known more than a few non-RCC (for instance, all Protestants) who do some version of this without realizing it.
Synopsis version: They'll what they say. Once you know better, you'll know better. The world will keep on spinning. Don't worry. Pray.