First of all, you have to realize that it is only called "Sunday" in English. In many languages, including the languages of the early Christians (e.g. Greek & Latin), it is called "The Lord's Day."
All of the early sources speak of Christians observing The Lord's Day on the first day of the week.
1. Scriptural references: Acts 20:7 and 1 Corinthians 16:2.
2. Didache 14 (Dated by Anglo scholars between 80 and 120)
3. St Ignatius to the Magnesians (c. 108)
4. Pliny the Younger's letter to Trajan (c. 112)
5. Epistle of Baranabas 25 (c. 120)
6. St. Justin Martyr's Apology 67 (c. 140)
And, of course, many others from the late second century onwards.
Some scholars speculate that the very earliest Christians (i.e. those in the first 10 to 15 years) went to the synagogue on Saturday to hear the readings and then held their distinctly Christian Eucharistic gatherings on Sunday. According to this line of thought, such activity continued until about 49 AD, when even the Jews in the diaspora decided to bar the Christians from attending the synagogue, since said Jews considered the Christians to be heretics. The ensuing street brawls caused the Emperor Claudius to kick the Jews (including Jewish Christians) out of Rome in 49 AD, as attested to in Suetonius' Life of Claudius and in Acts 18:2. After that, the two communities separated pretty quickly throughout the Empire. For example, some of the Jewish Christians who had been expelled from Rome ended up living with St. Paul in Corinth. At first, he would go to the synagogue to try to convert the Jews. As Acts 18 says, eventually the Jews got mad at him and "abused" him, so he cursed the Jews and declared that he would only focus on the Gentiles. No more synagogue going after that. Hence, in 1 Corinthians (which St. Paul wrote a few years after he cursed the Jews and left Corinth), St. Paul naturally assumes the Christians there would be meeting on the first day of the week (not at the synagogue on the Sabbath).
