Ok thank you. So it really didn't have much significance to the Christians or Jews of Arabia then?
No. The Cube was essentially a pagan shrine at that time, thus, nothing more than the pantheon of the Meccan paganism/mythology.
The author of the Qur'an even adopted from the Meccan pagans the legend concerning the protection of the pantheon against Abraha's Christian army.
"Hast thou not seen how your lord dealt with the owners of the elephants? Did he not bring their stratagem to naught, and send against them swarms of flying creatures which pelted them with stones of baked clay."
All Muslim scholars affirm that this event took place many years before the proclamation of Muhammad’s prophethood. Some said maybe during the time of his birth. What really happened was that Abraha, king of Yemen, constructed a church in San’a. An Arab man came and defiled the church and did some damage to it. Abraha decided to demolish the Ka’ba which was a sacred site for the heathen people of Quraysh and the site of their annual pilgrimage before Islam.
Abraha headed the invasion operation along with his generals who rode on elephants. It was said that there was a huge, strong elephant called Mahmud. God sent black or green birds to attack the invading army. Each bird carried one piece of gravel in his beak and two in his claws and hit the owners of the elephants. They claim that each piece of gravel penetrated the head of a man and exited from the backside (anus - lower opening of the rectum). On each piece of gravel was written the name of the victim. Abraha suffered a violent death (refer to the Zamakh-shari in the Kash-shaf, part 4, page 797; Baydawi, page 811;Jalalan, page 519; also the "Prophet’s Biography" by ibn Hisham, part 1, pages 38 and 39).
The Qur’an does not tell us why God sent these vultures to assist the heathens against the Christians. The Qur’an was content to record the episode of this battle between the vultures and the elephants. Thus, a chapter in the Qur’an was inspired under the title of Chapter of the Elephant.
http://answering-islam.org/BehindVeil/btv13.html