It's in 2:127.
St. John of Damascus has this to say:
They furthermore accuse us of being idolaters, because we venerate the cross, which they abominate. And we answer them: ‘How is it, then, that you rub yourselves against a stone in your Ka’ba and kiss and embrace it?’ Then some of them say that Abraham had relations with Agar upon it, but others say that he tied the camel to it, when he was going to sacrifice Isaac. And we answer them: ‘Since Scripture says that the mountain was wooded and had trees from which Abraham cut wood for the holocaust and laid it upon Isaac, and then he left the asses behind with the two young men, why talk nonsense? For in that place neither is it thick with trees nor is there passage for asses.’ And they are embarrassed, but they still assert that the stone is Abraham’s. Then we say: ‘Let it be Abraham’s, as you so foolishly say. Then, just because Abraham had relations with a woman on it or tied a camel to it, you are not ashamed to kiss it, yet you blame us for venerating the cross of Christ by which the power of the demons and the deceit of the Devil was destroyed.’ This stone that they talk about is a head of that Aphrodite whom they used to worship and whom they called Khabár. Even to the present day, traces of the carving are visible on it to careful observers.
Based on this, we would question the assertion that Abraham built it, and we would say it has ties to pre-Islamic Arab paganism.
The same guy I mentioned in another thread, al-Tabari came up with the idea that Abraham lived in Mecca and made the Kaaba. It's not in the Qur'an... that I know of.
"And [mention] when Abraham was raising the foundations of the House and [with him] Ishmael, [saying], "Our Lord, accept [this] from us. Indeed You are the Hearing, the Knowing."
Hmm... but that passage is bereft of context and requires lots of interpretation to get to a specific 'house' in Mecca, I think.
According to the Qur'an, Abraham said this:
Our Lord! Lo! I have settled some of my posterity in an uncultivable valley near unto Thy holy House
, our Lord! that they may establish proper worship; so incline some hearts of men that they may yearn toward them, and provide Thou them with fruits in order that they may be thankful. (Surah 14:37)
The Kaaba was considered the House of Allah long before Muhammad, Allah being the supreme deity of Arabian paganism.