Im not native English speaker but in my opinion "not to be late" is correct, because "late" is an adjective so it should be with "be", after preposition "to" is normally a verb so "to be" is a correct one however in spoken English you can use both I think