My personal opinion is that context is usually enough to make it clear that the people saying "God bless you" or "May God bless your ministry in your new parish" or whatever do not intend to confer a blessing upon the person they are talking to the way a priest would, but instead are offering a sort of prayer for God's blessing upon that person. A layperson doesn't usually wish a priest "God bless you" and push his fist into the priest's face expecting a bow and a kiss, after all. Similarly, when I pray "Lord, have mercy" for someone, I don't imagine I'm offering a sacramental absolution.
Maybe "back home" such things aren't done, but those are generally Orthodox cultures, and this is not. Here, in the larger culture, such "blessings" aren't really what we'd call blessings but rather prayers. So I don't see this as a big deal. If one is not in the habit of saying such things, one shouldn't start, but if it comes naturally, I wouldn't sweat it. But again, that's just my opinion.
Incidentally, it's not entirely accurate to say that a bishop is never blessed by a priest. Different traditions have different rubrics regarding such things, but sometimes it is not optional for a bishop to receive the blessing of a priest. If a priest hears a bishop's confession, or if he anoints a bishop during the Sacrament of Unction, for example, a blessing is a part of the rite.