"A little milk" is the correct one. We usually don't use "of" after "a little ".

To say a little of you need to insert a noun, for example, a little bottle of milk, a little cup of milk, a little sip of milk.

Thanks for all of your kind comments.

Sounds like a line from Oliver Twist, by Charles Dickens, a famous English author :)

“A little milk” is correct. We can also say “a little bit of milk”. We cannot say “a little of milk” in English — that is typical in some other European languages, especially Romance languages (i.e. languages from Latin). Edited for a typo.