You said that mathematical notation is countable, but generally, it is not. Generally, if you want to count things like ∑ and ∫, you would say "two mathematical symbols" rather than "two mathematical notations." In general, we will talk about writing complete statements "in mathematical notation." "In mathematical notation, Newton's second law is written as F=ma" If you want to measure mathematical notation, you would usually use things like "lines" or "pages". "The paper's introduction was immediately followed by five lines of mathematical notation."

Thank you very much!!!

James is correct. In this case, it's referring to the concept of mathematical notation as a single idea, so it's in singular form. If you were talking about the different kinds of mathematical notations (numerical, symbols, equations, etc.), then you could use the plural form. Example: "Now we will discuss the different ways to use mathematical notations." Mathematical Notation as a concept is singular. Mathematical notations as a list would be plural. English is weird, so I understand your confusion. Don't worry about it too much though - most English speakers wouldn't notice difference, and most don't even know which way is correct. I'm a native English speaker and I still had to think about it for a minute to figure out which way was correct lol. Your English is very good so I hope this helps! Keep up the good work!

It is being used here as an uncountable noun, which is correct in this case. Plenty of nouns can be used both countably and uncountably.