I’ve never gotten it as an effective insult, tbh. It boils down to:

I wish I knew half of you better, and I’m a crab about some of you

No? If he doesn’t like them as well as they deserve, then the “problem” is with him, not them. It’s definitely not welcoming, but it’s not exactly insulting if I’m reading it right.