I remember Andy talking about this in class. The story that he told us was that Nobel's wife had an affair with mathematician Mittag-Leffler so Nobel wanted revenge and would never give this award for mathematics. However, I found some information saying that this is almost completely false since Mittag-Leffler was 'a life-long bachelor.' The article states that Mittag-Leffler was a leading Swedish mathematician who antagonized Nobel (For what? The article doesn't say). Because of this Nobel decided not to give a prize in mathematics simply because he didn't want Mittag-Leffler to be a winner. Which story is true? Either one of them? I have no idea but I understand why math people like to say that Nobel's wife had an affair.

http://www.agnesscott.edu/Lriddle/women/prizes.htm#nobel

scroll down to the section labled 'Nobel Prize'