A dinner is a formal occasion, for example at a wedding. A dinner party would be less formal, but is still an event.
However, even if you are inviting someone to a formal occasion, you would still invite them to dinner, not to "a dinner."
The fact that it is a formal occasion would be implied by the context.