I am reading the Desert Fathers, and there's one episode which answers this. I wish I could remember which Father it was, but, I tend to flip through the book and stop at odd pages here and there. I'll try to look it up tonight.
The story goes something like this:
Abba Something and his disciple went visiting to a neighboring monastery. The brothers there greeted them warmly and put out food for them to eat. They sat and ate, and in time got up to go home.
On the way home, the disciple stopped by a stream and bent to get a drink of water, because it was hot and dusty. Abba Something quickly called to him and told him to stop, that today is a fasting day. The disciple looked up at him and reminded him that they had just eaten a full meal....to which the Father replied something like ...we ate because it would have been rude to dismiss the other's charity, however, now that we are on our own, we are to fast.
Therefore, if you are in company of non-Orthodox, or even Orthodox who might not be fasting, it is not sinful to partake of their meal (within reason). However, the moment you part company with them remember it's still a fast.
I'll try to find which Father this was and get back to you tomorrow.