Of course, if the whole purpose of not stating from the outset that you don't like sports is to avoid feeling "unmanly" (not an unfair conclusion as you felt the need to state your heterosexuality multiple times in the first few sentences) you could always say "Sorry, I don't follow (sport X)." I know very few men who follow every sport (multiple sports, perhaps, but usually one of them is left to the wayside out of the American big 4 of baseball, football, basketball, and hockey), so not following the particular sport the other wants to talk about isn't too odd. Of course, this still doesn't work, as the sports enthusiast will still want to talk about last night's game, at which point basically apply the time honored male conversational gambit of allowing your eyes to glaze over and replying "yeah...uh-huh...you don't say... that ref was blind...." at appropriate pauses.
Oddly enough I actually enjoy basketball, but the entire time I lived in Chicago not one person ever wanted to talk about that. It was always "Cubs this", "Sox that", "da Bears", or even "can you believe that Blackhawks game"? Of course no one ever wanted to talk about the only Chicago team I had followed my whole life (and the only Chicago team to have actually won a championship during the decade). The only way I survived baseball playoff season of 2003 was by taking perverse pleasure in having just recently moved there from Florida and "my" team erasing hopes of a Cubs World Series for the next hundred years.