The Lord said that we commit adultery in our heart when we look lustfully upon a woman. Noticing a woman's beauty does not necessarily mean looking lustfully, but "just noticing" in this way may be the beginning of lust and that adultery of the heart.
If we do not control our gaze, the demons will find ways to put before our eyes those things which can distract and ensnare us. The fact that we at least know not to look at other women while with our wives is an indication that such looking is probably not blameless. Of course, it may not be best to stare at the ground when a female boss or coworker is talking to us, but otherwise we should control our gaze to avoid all manner of temptations, lust being just one. Looking around here and there can also be a sign of inner restlessness and a desire to find some fulfillment or diversion outside ourselves through indulging the senses and the flesh. Instead, we could keep our gaze before us and keep our mind and heart in prayer, recalling our sinfulness and asking the Lord for His mercy.
St. Gregory the Diologist said of St. Benedict of Nursia:
I said that this venerable man did dwell with himself, because carrying himself circumspectly and carefully in the sight of his Creator, always considering his own actions, always examining himself, never did he turn the eyes of his soul from himself, to behold aught else whatsoever.
Chapter 3, http://www.fordham.edu/halsall/basis/g1-benedict1.asp
We can also ask ourselves about our perception of beauty, and whether this perception of beauty is one that is fleshly or whether it is spiritual. Do we perceive beauty merely in terms of the external appearance of the body, as does the world; or is our sensitivity to beauty a sensitivity to the beauty observed in a person who lives a life pleasing to God? In other words, is it true beauty that gets our attention and causes us to marvel - a beauty that comes from within and animates a person with a certain light or glow; or is it the counterfeit and deceptive beauty which is of the flesh and often leads us into sin?
Also, in noticing the beauty of the opposite sex, where is our mind afterwards? Do we continue to think about this person, allowing this person's appearance to occupy our imagination? If so, we are on a dangerous path. These are at least a few things to think about.