It’s usually clear from the context, but you should notice that the preposition “to” is usually pronounced with no stress (about like the English version of 輕聲). Also most of the time, the vowel in “to” is reduced to a schwa (a neutral vowel similar to the vowel in 的 pinyin de5 — actually, “to” is often pronounced almost exactly like 的 in American English). This is typical of structural words such as prepositions. On the other hand, the number “two” is usually pronounced with a full vowel, sounding like 兔. Example (with IPA): I gave it to two people /aiˈgeivɪtːəˈtuːˈpiːpəl/ Or "go to two places", where “go to two” sounds like 狗的兔.

to 听起来像“特”的轻声，跟two完全不一样

听语境吧