According to a number of physicists I've talked to, at our current rate of energy release this is already a concern. I've heard that we'll need to worry about this in the next 300-400 years.