"I like to talk to you" doesn't sound natural. You might say, "I would like to talk to you," or even better, "I would like to talk with you." But that is asking for someone to take a moment to talk with you and doesn't have to same context as the original statement.
I think it would be very rare for someone to say, "I like to talk to you." It would be more appropriate to affirm your enjoyment of talking with someone by using the phrase, "I like talking with you." When you say "to you" it is one sided, as if the person is not responding; use "with" instead.