It is true that many holy fools walked around in rags or even stark naked, but this was an outgrowth of their holiness and their calling to holy foolishness. Not everyone is called to that (in fact it's quite rare), and if one seeks to emulate them in their ascesis, without the accompanying holiness and calling, it is quite possible one is deluded and in fact more arrogant than some uptight man wearing an Armani suit.
You cannot judge a person by what they wear. Someone might wear simple clothes out of sheer pride before men, or someone may wear expensive clothes out of sheer awe before God. It is not for anyone to say.
For myself, I was raised to wear my best when I go to worship God. Not because I think I can impress Him, but because that is what humans do. We give nice things to people we consider important. Yes, it's all nothing but filthy rags before God, but I think it's like a 3-year-old scribbling in crayon for his mother. It's nothing but a mess, but it's beautiful to the mother and child and that's all that really matters.
People dress rather formally at my parish, and it does not necessarily reflect the wealth. I would wear a suit if I could afford one.
So long as we do not cause others to stumble, what we wear is a personal decision. Whatever we offer God with a loving and pure heart is accepted by Him, I believe.