The “97% consensus” number is based on surveys of scientists in general, not just climate scientists. Including scientists when they agree with a position, but not when they disagree (because they aren’t real climate experts) is a very strange debate tactic.
"Among the most-cited is a 2013 study of nearly 12,000 abstracts of peer-reviewed papers on climate science published since 1990, of which just over 4,000 papers expressed an opinion on the cause of recent global warming. Of these, 97% agree, explicitly or implicitly, that global warming is happening and is human-caused."