The difference lies in the way the word 'to' is used. It can either be a preposition, in which case you'd use the gerund form of a verb or as an infinitive marker (base form of a verb e.g. to eat, to play etc.) This is a tricky topic and it requires some research. A simple answer won't make you understand that concept. You have to develop a feeling for it.

In general, you learn that certain phrases that end with "to" have a gerund. Look at the whole phrase, not just one word. This site below gives some examples: http://www.englishteachermelanie.com/grammar-can-a-gerund-be-used-after-to/