I can't speak Cantonese, but I guess that the situation of learners of Cantonese and Mandarin might be similar, so I am telling you about my experiences with Mandarin.
1. Pronunciation
The first time, I tried to speak to someone Chinese outside the classroom, I just said the simple sentence "Where are you from?" in Mandarin. The other person stared at me blankly. As it was a Chinese student in Germany, I asked him in German, if I did a grammatical mistake and explained word for word what I have said. The guy said, grammar and vocabulary were fine, he just couldn't understand me, because of my bad pronunciation. Well, it took me years, until I attempted again to speak Mandarin. ;-)
2. Looking non-Asian
Chinese people who don't have a lot of contact with Western foreigners, often assume, that Westerners can only speak English, but not Chinese and panicking they don't even listen to what the foreigners say. If a Westerner with excellent Chinese and a East Asian with broken Chinese go to a restaurant together, the waitress will still prefer to talk to the Asian looking person and not understand what the Western looking foreigner says. I once went to a restaurant with a very tall, blond guy, who was a vegetarian. He tried to explain it to the waitress and said in Mandarin: "I don't eat meat." There was nothing wrong with his pronunciation, but the waitress looked to me for help. Then I said: "He doesn't eat meat." And even though, I didn't speak better than him, the waitress understood what I said, simply because I look Chinese.