We need more context. Can you provide link to the source, or give a few sentences that precede it? It probably means "It makes me angry that people pay so much attention to this number, because I don't think it's very important." Here are some ways it could be used. "People pay too much attention to their credit score. It's not a measure of whether you are a good person, it's just a damn number." "The Dow Jones Industrial Average broke 24,000 today. And that means absolutely nothing. It doesn't tell you whether the economy is good or bad, or whether the stock market will rise or fall. It's just a damn number."

Without a broader context, it is hard to be sure. Generally, in movies, this is said to be "human" and go against the odds. e.g. - Captain, the computer says we have only 1% chance of surviving. - It is just a damn number. We are better than this.