To be sure we understand, suppose there is just one other couple, A1 and A2, in addition to your wife and you, W and Y.
So A1 could shake hands with W and Y, A2 could also shake hands with W and Y.
Then, when you stop and ask, A1 would say 2, A2 would say 2, and W would say 2.
Another scenario, which matches the result you postulated, is A1 shakes hands with W an Y, A2 shakes hand with no one. Then, when you stop and ask, A1 would say 2, A2 would say 0, and W would say 1.
You don't count your handshakes as part of the answer. Is this correct?