I only get annoyed if I'm at work - where most of my job involves working online. When the internet is down I literally have nothing to do but drink tea, eat cake and wait until we get reconnected.
If we lose connection at home I just wander off and read a book, watch TV, play on the DS (AC:WW catch up), go sit in the garden, walk the dogs... there's a ton of stuff to do when the internet is down (including the boring stuff like housework and grocery shopping and preparing family meals).