He said "See you," to say "goodbye" is correct. "He said see you by saying goodbye makes no sense because goodbye does not mean "see you." But "See you" is a way to say goodbye.

They each carry different meanings. The first- 'by saying goodbye' means that he said goodbye through the means of, or by using the expression 'see you'. So you can see why it doesn't make much sense as 'see you' is a way to say goodbye, not vice versa. The second 'to say goodbye' just means that his intention was to say goodbye and he did it by saying see you... or He said goodbye by saying see you. This one makes actual sense as that's what one does by using the expression 'see you'.

both of them mean the same thing but second version sounds much better