It might not technically be a desert, but saying it's "far from" is incorrect too. Semi-arid climate zones are as close to deserts as you can be without actually being a desert. For recreational purposes in particular, people tend to look mostly for open water or mountains, sometimes forests or canyons. Miles and miles of "scrubby prairie" lacking any of those might as well be a desert.

Once you get past Abilene or San Angelo you're getting into the eastern reach of the Chihuahuan Desert, so around 4 hours. Some maps put it west of the Pecos but that's not the official line, to be fair different maps seem to put it in different places.