I think the Church Fathers quite clearly do teach there's power in the sign of the cross that is real and true whether or not we believe in it. That said, does my baptismal cross keep me from sin because I wear it? Of course not. Even wearing a relic of the True Cross would not do this. I could even destroy a relic of the True Cross, if I were so inclined. It has happened historically. As you said, the cross is not a talisman, and Christianity is not magic.
So, how to reconcile the absolute truth of power in the sign of the cross vis-a-vis our sinfulness in spite of having it everywhere? The cross, no matter how powerful it may be, does not deprive us of our own will. I can choose to "ignore" and even disdain (as many do) the power of the cross and live my own life without it. Often times, I do ignore or otherwise forget the cross that hangs around my neck.
However, many other times the power of the cross manifests itself to me. I see the crucifix on my wall, or on my prayer book laying on the table. I feel my baptismal cross tap against my chest. Sometimes, when I remember I'm wearing it, I pull it out and look at it for a second, usually crossing myself before putting it up. That's holy. That's sanctifying me, blessing me, in remembrance of the cross and it's saving power through our Lord. When this kind of attention is garnered by the cross from me on a particularly bad day, it helps center me. I spend at least a moment in prayer (even if that moment is just crossing myself), but that tends to linger in our minds, turning us to something more heavenly and in some way casting off earthly cares. That, I think, is the power of the cross for us. At least, it is for me. Granted, this doesn't stop me uttering a curse the very next moment when something goes wrong (which happens more often than I'd like to admit), but if not for my sinfulness, what spiritual heights would be achieved by meditation upon the Life-Giving Cross?
Just my two cents.