Do you keep your happy hearts at four or six full? Does your brother keep his at four or six full?
I think it only removes four hearts at one time, so if you either feed your Tamagotchi enough snacks to fill its two invisible happiness hearts or play the games perfectly to fill its two invisible happy hearts and your brother does not, your Tamagotchi is more likely than his to be at five or six happiness hearts when you get bad mail, so you'll only be left with one or two hearts.