It is true that the Early Church's depictions are of Christ as a beardless youth, but you forget that those depictions are of Christ as the "Good Shepherd" and, in fact, they are images that are based on a Pagan prototype, and so if a Roman Soldier (or another persecuting force) saw the depiction, they would not connect it to Christianity, they would see it as simply an image from their own Pagan religion.
However, if the Early Christians depicted the "Good Shepherd" with a beard, etc... Then it would have appeared somewhat odd to the Pagans, and would have raised a lot of suspicion if found.
The same can be said for Early Christian depictions of the Apostles, which often put the heads of animals on them. Are we to believe the Apostles had animal heads? No, these are images that have somewhat Pagan prototypes and to a Pagan would look somewhat normal, whereas to Christians, they knew who these images truly depicted.
This also goes for the "ΙΧΘΥΣ", the fish that is often seen in/near Christian places of worship. This word means "fish" in Greek. To a Pagan, it would have appeared to be a reference to Pisces and has roots within Greek Mythology. However, to Christians, it is pretty obvious the connections between fish/fishing/fishermen and Christianity.
So I would personally argue that we cannot take the "Good Shepherd" images as an accurate depiction of Jesus. However, the earliest depictions (other than Good Shepherd) often do depict him in a very similar manner to modern icons.
We can also look at the earliest images of the Apostles (that is, that depict them facially without animal heads) and it's amazing to find that their depiction across the Christian world is consistent, and in fact, is the exact same as their depiction in icons, even up to the present day.
The only Apostle that I'm aware of that has been depicted without a beard would be St. John, who would have been a teenager during Christ's ministry. It would make perfect sense for Christ to have had a beard. Of course, the longer hair could be debatable, but from what I've heard, it was normal for men from Galilee/Nazareth to have longer hair.
I also think there are some written records as to his appearance, though of course, these are probably disputed and are in no way comparable to the level of even the Church Fathers, but it's interesting nonetheless.