I had a similar epiphany about linguistics, much of which appears to be the study of example sentences that linguists come up with.
Since you could invariably tell when something was an example sentence, this was obviously a different language from what people actually used. (Of course there are linguists who go out and study real language-use, but some actively dismiss this as mostly irrelevant "performance")
Those example sentences linguists come up with are actually very reliable. Here's Sprouse & Almeida. 2012. Assessing the reliability of textbook data in syntax: Adger's Core Syntax. https://doi.org/10.1017/S0022226712000011
" [...] This suggests that even under the (likely unwarranted) assumption that the discrepant results are all false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the minimum replication rate of these 469 data points is 98%."
Interesting result, but seems to answer a different question: do naive native speakers find those examples acceptable?
The first problem is that asking people how they speak or what is acceptable does not accurately represent how they actually speak when not observed/asked:
The second problem is that, even if the first isn't an issue, the most this can demonstrate is that linguistic example sentences are a subset of actual language.
That's a fair point, but then the question is what should the question be? There are many subfields and disciplines of linguistics all of which are imo fascinating. In theoretical/generative linguistics the distinction between acceptability and performance is relevant, and acceptability seems to be a very robust measure.
> subset of 18 to 21 year old American women.
This is perfect!
I had a similar epiphany about linguistics, much of which appears to be the study of example sentences that linguists come up with.
Since you could invariably tell when something was an example sentence, this was obviously a different language from what people actually used. (Of course there are linguists who go out and study real language-use, but some actively dismiss this as mostly irrelevant "performance")