While I do read non-technical books, I don't read fiction for the same reason. It does feel like a waste of time when I could simply watch TV/surf the net instead :)
I read a lot of SciFi, and have actually found that while I might not learn a ton from a book directly, I'm often inspired to do nonfiction reading on ideas or events mentioned in the book.
