think of the positive version of the sentence and it becomes more obvious it should be is:

“Either of us is going”

Or, to further stress the point, turn it into a question:

“Is either of us going?”

The subject is not us, it’s “either of us”, which is singular