If you KNOW they are just "milking" charity, it may be a problem...
I generally fall under the "give without presuming what they'll do with the money" category, but every so often I'll run into this.
There's a lady I'm familiar with who "works" a certain NYC neighbourhood. Her routine is always the same: she enters cafes and other food establishments and solicits donations from customers. If you reject her without hesitation, she moves on without incident. If you give money without hesitation, she'll thank you and move on. But if you try to talk to her, she gets impatient, as if she'd like to move on, but feels pressured to stay, so she'll tell you a whole story about her grandchildren who are hungry and need something to eat, so she needs to get going, etc. If you offer to buy a large pizza for her, she'll say one of them is lactose intolerant and another's a vegan or something. If you offer to take her grocery shopping, or buy some stuff for her, etc., she'll reject all that too, but it doesn't seem to come from pride. If you tell her that you offered all that because you have no cash to spare, but can use a credit card, that's not good enough. When you find out through the chat that she's a RN at a local hospital and you ask why she's on the street begging if she's an employed health professional, you get several variations of F, MF, B, B, A, S, C, C, D, and a few other expletives hurled at you, along with curses upon your family (e.g., "I hope your MF mom gets hit by a bus because God hates cheap-A MF B's like you"). Then she'll move on to the next table as if nothing happened.
It's a tough situation to negotiate. I think, if you have little or no time, it is appropriate to give without hesitation, even if it is small. But if you have even a little bit of time, it's worth it to try and get to know the person. Sometimes you'll realise who's really in need versus who's using their free time to beg for supplementary income.