"Everyone will see I'm nervous"
Much of the dread is about being seen. You picture your red face, your shaking hand on the glass and the pause before you answer, and you assume these are as plain to everyone else as they are to you. If they are, then going means putting your nervousness on show.
Thomas Gilovich, Kenneth Savitsky and Victoria Husted Medvec studied this assumption, and they called it the illusion of transparency. Across several experiments, people overestimated how far others could make out what was going on inside them. People telling lies thought their lies were more detectable than they were. People tasting something disgusting while trying to keep a straight face thought their disgust was more obvious than it was. The researchers' explanation is that your own inner state is so vivid to you that you can't fully set it aside when you imagine someone else's view. You know your heart is pounding, so it seems it must show. Mostly it doesn't, because other people can't see or hear your heartbeat.
Those experiments weren't about social nerves in particular, so I'm extending them. But the mechanism applies with some force. To you, a blush is heat flooding your face. To someone across a pub table in evening light, it's a slight change of colour on a person they're only half looking at, if they notice at all. The blank moment that feels like ten seconds to you is a short pause to them, and conversations are full of pauses.
There's a second finding about what happens when people do notice. Savitsky, together with Nicholas Epley and Gilovich, ran studies in which people committed some small social blunder and predicted how harshly observers would judge them. The observers were consistently kinder than predicted. My reading is that the person who'd blundered was focused entirely on the blunder, and the observers weren't.
Think about this from the other side, because you have the evidence already. Recall the last gathering you went to. Try to remember who there seemed nervous, who blushed, who said something slightly off. Most people can't come up with a single instance. That's not because nobody was nervous. I'd guess a fair share of any room is, to some degree. You didn't notice because you were busy with your own evening. Everyone else was too, and nobody was studying you.
None of this means nerves never show. Sometimes they do, and someone sees that you're a bit on edge. What usually happens then is that they're a little warmer with you, and that's all.