I recall a story from the Desert Fathers, of one old monk who had taken a younger monk with him to visit someone, and they were offered food. They ate, even though it not only was before the time that they would have normally eaten, but the food itself broke the fast, regardless of the time it was served. As they were walking back to their cell, the younger monk stopped by a river to get a drink, and was promptly berated by the older monk telling him that you are not to refuse hospitality, but when you are by yourself, do not break the fast just because you are weak. I think the moral of the story is, if you are at someone's house, and they offer food or a drink that violate the fast, don't refuse because of your fast, because you might as well have told them "I am fasting. I fast from dairy and meat during lent." as such can lead to sinful thoughts either on your part or theirs.
Now, if - during the fast - you are invited to the house of someone who you have reason to believe will serve non-fasting food (say, they are a normal meat-eating non-Orthodox), if you accept the invitation, you have no right to then claim (when you are at their house), "I can't eat your non-fasting food." You have put yourself in the position to. If you care so much about the fast that you are willing to offend people by refusing their hospitality, and by necessity also informing them of the nature of your fast, it is better for you to simply decline the invitation.