I've had the same problem for a while now on my highly mistreated iPhone 3GS as well. It might have something to do with any of the following, each of which are enough to void the warranty:
1) It's jailbroken and unlocked
2) It's been submerged in both fresh and saltwater
3) Sand somehow got in there
4) I've pried the thing open and disassembled/reassembled it in order to fix the home button issue, which I thought might have been caused by the aforementioned sand and water damage (only helped briefly and intermittently)
3rd parties do sell replacement home buttons online, and I've considered going ahead and replacing it, now that I know the pitfalls of opening this sucker up (watch out for those delicate ribbon cable connectors). It's not been too bad though, and I've made it long enough to replace it with the new 4S, already on pre-order. Hopefully, the new one lasts longer than this one did, as I'm probably not going to treat it any more kindly.
[Edit] After reading through the comments, I found this potentially helpful advice from "Kevan":
This problem is extra annoying because for the first few months it happens it just unpredictably fails to recognize Home button presses sometimes.
But since merely buying an iPhone (and not taking the time to jailbreak it) already makes me a supplicant to the corporate interests that render their verdicts on what I may and may not do with it, I suppose that hey, if dirty wants his money, i'ma just give him his money...buying a new iPhone 4S will fix this (temporarily, one might surmise, since it's likely the very same home button assembly).
1) It's jailbroken and unlocked
2) It's been submerged in both fresh and saltwater
3) Sand somehow got in there
4) I've pried the thing open and disassembled/reassembled it in order to fix the home button issue, which I thought might have been caused by the aforementioned sand and water damage (only helped briefly and intermittently)
3rd parties do sell replacement home buttons online, and I've considered going ahead and replacing it, now that I know the pitfalls of opening this sucker up (watch out for those delicate ribbon cable connectors). It's not been too bad though, and I've made it long enough to replace it with the new 4S, already on pre-order. Hopefully, the new one lasts longer than this one did, as I'm probably not going to treat it any more kindly.
[Edit] After reading through the comments, I found this potentially helpful advice from "Kevan":
Depressing to hear this. I had this problem too, but there were a couple good tips at this page: http://www.tipb.com/2011/01/04/tipb-bug-home-button-working-... This is the formula that worked for my home button, which works like normal now:
1.) Open any application
2.) Press and hold the power button until the slide to shutdown swipe bar appears.
3.) Release Power button
4.) Press and hold Home button Lightly until screen returns to icon screen
5.) Enjoy your functioning home button!!!
This procedure recalibrates the home button after normal wear or heavy use. [/Edit]