Okay, I feel like a crazy person arguing against a luminary of the field, but this is so misleading as to be disingenuous and make me trust the guy less. He's throwing causality and explainability into the same bucket and arguing against the need for the latter.
>We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications.
>Yes, that can be useful. Or at least reassuring.
>But sometimes people have accurate models of a phenomenon without any intuitive explanation or causation that provides an accurate picture of the situation.
It goes on to argue mostly against the need for intuitive explanations, not the establishing of causal relationships.
>Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing.
>But we don't actually have a "causal" explanation for it, though we do have an accurate mathematical model and decades of experimental evidence.
The physical models we have are causal ones. The intuitive abstractions like bernoulli's principle may not work, but analysis based on navier stokes sure does. You plug your shape (cause) into the equations and see what forces (effect) occur. That's causation.
>You know what other life-critical phenomena we don't have good causal explanations for?
> The mechanism of action of many drugs (if not most of them).
Using an industry that's nearly synonymous with a randomized controlled trial as a refutation for the need of a causal relationship is crazy talk. The mechanism may be missing, but the causal explanation is that via a series of RCTs it's established that the drug causes the effects.
I get that half of this is trying to go against a percieved need for intuitive explanations, but it weirdly lumps causation in there.
"How much does this input seem to confuse this output? What is the pattern across inputs for how this model is systematically confused?"
Causality -> counterfactuals
"How would the outcome be different if x was different? If I acted differently, would I get a more favorable outcome?"
You're right to say these are two different things. They are.
And they're different still from interpretability, i.e., "What are the explicit patterns that this model is seeking in the data?"
DL practitioners routinely mix up explainability and interpretability but I would never in a million years have seen LeCun be so intellectually dishonest as to lump causality in there with them.
The thing is, I would claim that causality, explainability and interpretability are all mixed together in human informal discussions of various phenomena. As others on the thread have pointed out, Pearl's causality isn't everyone's causality. A tension structure can disrupt our "common sense" idea of what's holding up what but tensions structure doesn't seem at all like a black-box, unexplainable item. The way the article mixes the range of these issues seems definitely wrong but I don't think that means the line between all things is normally crystal clear.
> It goes on to argue mostly against the need for intuitive explanations, not the establishing of causal relationships.
And then fails to acknowledge that we don't even have intuitive explanations for a lot of AI models! And for a layperson, it's just a complete black box; for cases where we do have intuitive explanations, I'm not sure experts are at all effective at translating them into something a layperson can understand.
I'm not a physicist or aeronautical engineer, but I can grok intuitive explanations for how airplanes work without much trouble.
The physical laws are correlational, not causal. F=ma doesn't tell you if acceleration causes force or vice versa. Causation is a humanly imposed concept . That's why we need to invent things like the Chronology protection conjecture.
Causal effects are (typically) defined via potential outcomes[0]. If I do X, what happens to Y? The laws of physics can be used for that, which is what I mean when I say they're causal.
If our physical laws couldn't predict what effect a given cause has, we'd say they're wrong. Like if I couldn't swap out one mass on a spring (in an known scenario with an ideal spring blah blah) for another mass (cause) and predict how much lower the new mass will hang (effect), we'd say we don't understand spring physics.
There are unresolved foundational questions like those discussed in your link, but those aren't practically relevant, just like missing foundations of set theory wouldn't prevent us from balancing checkbooks. There's some notion of arithmetic I'm using for my checkbook, just like there's some notion of causation when I apply a 10N force (cause) to my 1kg mass and get 1 m/s of acceleration (effect). Foundationally formalizing it is neat, but unnecessary.
[0] Pretty much all of the rest of the time, they're still compatible with this definition.
i see what you mean but it 's kind of circular: the causal model is assumed, then observations are made and a law is formed which can be used to make predictions. The law itself is symmetric in time though and can make predictions in reverse, so the causal model is not baked in it. Regardless, this causal model works for physics which has well defined hypotheses and well defined deterministic systems. In biology , establishing causation (and thus explanability) is tricky because , even though hypotheses are well defined, the systems are not very deterministic.
In ANNs OTOH, even though the systems can be very deterministic, there is very little to make in terms of hypotheses. An explanation of the sort "you have cancer because neurons 10, 18, and 19 fired" is not satisfactory enough to pass the human test. It may be that for some complicated problems, searching for patterns in the neurons in order to explain them may prove to be futile. Not that people should give up on that, but not everything has a neat closed form explanation. Lecun mentioned above that you may have recurrent relationships (which also occur in quantum systems), and these muddy the waters a lot, making it difficult to establish cause and effect. It is also a major pain in neuroscience, when real neurons are seen as an evolving dynamical system.
I think you're limiting yourself too strictly to the equations and missing their application. Suppose, not a causal model, but a causal question. My question is if I put a mass weighing X on a spring with spring constant k, what will the displacement be (after oscillations stop, under standard gravity g, blah blah)? That's a causal question, answered by hooke's law. You can construct analogous questions for all sorts of scenarios. I'd say a model can be considered causal if it can answer all (relevant) causal questions about what it describes.
Take something more dynamic, like a ball being thrown on a level surface (uniform gravity, ideal vacuum, etc). If I throw it with some force, it'll follow a perfect symmetric parabola and land with that exact same force. The force it lands with (y) is exactly identical to the force I threw it with (x). It's basically time symmetry, but simpler. The equation here is the symmetric y = x, so it won't help you define causation. But clearly the usual version of causation says that my cause is x and the effect is y.
Maybe where we differ is that you say it's "the causal model is assumed, then observations are made and a law is formed which can be used to make predictions," while I'd describe it as "interventions are made, then observations are made and a law is formed which can be used to make predictions (and the law itself doesn't depend on the intervention taken)." That's what makes the y = x scenario not symmetric. X is my intervention.
no physical law implies causation. most laws describe relationships between measurable quantities. there is no causation implied e.g. in e=mc2 or e=hv or maxwell's equations etc.
the causality is imposed post-hoc depending on your application, but it's not explained by the law itself. alternatively, causality is a prerequisite in order to have physical laws at all.
> An explanation of the sort "you have cancer because neurons 10, 18, and 19 fired" is not satisfactory enough to pass the human test.
Sure, but that doesn't mean we should be okay with not understanding why those neurons fired.
That is: if an ANN can reliably figure out if someone has cancer based on various inputs, then it should be possible to isolate the ANN's rationale for that determination. That probably ain't an easy task by any means, but unless ANNs are magic spells not subject to our physical laws, it is possible nonetheless.
hooke's law correlates spring length with force, but doesnt prescribe an order of events. You can increase the weight or move the spring to a higher gravitational field or something else we dont' know yet. However, as humans we can't reason of a way to increase the length of the spring without changing the force, therefore we establish a causation model that "if increase-weight then extends". However if by some empirical process we could confirm that "if extends then increase-weight", hookes law would still be the same (we can do by e.g. putting it in a moving frame of reference)
You can do it by adding more mass in a fixed inertial frame, or by leaving the mass along and changing the reference frame. You cannot decide which happens, which is an insight from relativity.
So, which was the cause? You cannot tell. You simply can do the math and see that it matches the experiment. Only later do humans try to call it cause and effect, because the math and the experiment match so often and so well.
But this all can change if we find an experiment breaking the models.
Before relativity, most people thought time was constant. They were wrong.
So the above poster is correct in what we usually call cause and effect to simply be stronger correlations that match (current) models.
That can all be changed in future understanding of Nature.
Causation: if I do X, what happens to Y? For any well understood physical situation describing X and Y, the laws of physics tell you what happens to Y when you do X. If I swap out a mass on a spring for a heavier mass, physics will predict how much lower it hangs. The cause is me swapping one mass for another. The effect is where it now hangs. That's the whole point. If it wasn't this way, we'd say our physical understanding is wrong.
Having worked on physics projects for decades, the laws of physics at the lowest level are most certainly not this simple or causal.
>For any well understood physical situation describing X and Y, the laws of physics tell you what happens to Y when you do X.
Not true. If I create two electrons from photon collision, and measure one, will it be spin up? No way to tell. Only aggregates about large enough systems give any reliable answer to such questions.
If look for radioactive decay from 10 U235 atoms for the next 5 minutes, will I see one? Again, there is no yes/no answer, only probabilistic ones. There is no underlying causality - only purely random events with no detectable cause. These are the dice that Einstein didn't like.
There's plenty of similar questions that don't have a simple answer - only answers about large aggregates.
Here's [1] a recent result showing causality is no where the neat and tidy thing you think it is. These results are all over modern physics.
For example, the most accurate physical theories, such as QED, get the right answer by summing forwards and backwards in time. In QED the "future" affects results today as strongly as the "past". Causality is a psychological interpretation, but is not in the math
or the underlying theories.
As far as I know, all quantum field theories (which underlie all physics at the moment) all have such ambiguity or uncertainty about causality.
The laws are math models that give answers. But the lowest laws are time reversible or time agnostic for the most part, and the foundational theories require travelling forwards and backwards to get the correct experimental values.
Similarly, the laws of physics at the lowest level are not causal, but probabilistic. Only when aggregated do some experiments seem causal.
From another direction, there's a massive body of literature on what is causal, and can you detect it. Read, for example, Judea Pearl's monograph "Causality" or some of his other stuff, or simply browse wikipedia starting with him.
For example, when you drop a ball, it falls to earth. Quantum mechanically, there is a probability it simply quantum tunnels to another galaxy. That it most often falls to earth becomes a law, but it is imprecise and not completely correct. It's an approximation.
So every law of physics is merely a strong correlation.
And there's currently plenty of experiments trying to disentangle these issues with causality and locality.
So sure, at the freshman physics level causality is a simple thing. But Nature does not follow almost any of those rules with certainty or unerring rigor. Those are approximations and simplifications.
The question is do you need a detailed causal explanation you can understand.
In the case of lift, you can simulate aircraft model and calculate the change of vertical momentum in particles going above the wing and below the wing. Usually 80% of the lift comes from above the wing. If you are modelling a brick, it's the other way around. But do aircraft designers need to know that or are they just satisfied with less?
No I don't think that is the question. The questions (plural) are whether you need detailed explanations or intuitively understandable explanations or explanations at all, as well as whether you have to build a detailed causal model or an intuitive causal model or establish causation at all.
Aircraft designers are happy to test something in a wind tunnel and establish that the change in shape caused a change in performance. But they knew to try that particular new shape that because of all the physical (causal) models at various levels of intuitive understandability.
The thing is the article is starting out with the sort of network of semi-causal explanations that aircraft designers have and pointing out that this isn't simple causality, isn't just a line of causal events.
But then the article indeed jumps from that observation to the claim that causation isn't something we should worry about - which feels just wrong to me (and I think a lot of people).
I mean, all sort of intuitive explanations feel like causal explanation but, of course, are much more sketchy. But this feeling like causality matters imo, the whole network of explanations that satisfy a human together seems to make things robust, not fragile, whereas AI predictions and models are renown for being fragile. But still, I think a big part of the situation is we haven't characterized fully what humans do here.
>The thing is the article is starting out with the sort of network of semi-causal explanations that aircraft designers have and pointing out that this isn't simple causality, isn't just a line of causal events.
I'd argue that it absolutely is simple causality. Sure we have to flail around in the dark a little to get to a better plane, but they physically test things during development. That's simple causality right there. I guess I'm arguing that engaging in active (not watching someone else do it) trial and error is a case of causal reasoning.
Then would it be better to say that these models allow you to make causal changes?
That is, sure, they don't tell you why the specific lift properties are there. However, they do enable a practitioner to make a change and know what impact it will have on the lift.
>We often hear that AI systems must provide explanations and establish causal relationships, particularly for life-critical applications.
>Yes, that can be useful. Or at least reassuring.
>But sometimes people have accurate models of a phenomenon without any intuitive explanation or causation that provides an accurate picture of the situation.
It goes on to argue mostly against the need for intuitive explanations, not the establishing of causal relationships.
>Now, if there ever was a life-critical physical phenomenon, it is lift production by an airliner wing.
>But we don't actually have a "causal" explanation for it, though we do have an accurate mathematical model and decades of experimental evidence.
The physical models we have are causal ones. The intuitive abstractions like bernoulli's principle may not work, but analysis based on navier stokes sure does. You plug your shape (cause) into the equations and see what forces (effect) occur. That's causation.
>You know what other life-critical phenomena we don't have good causal explanations for?
> The mechanism of action of many drugs (if not most of them).
Using an industry that's nearly synonymous with a randomized controlled trial as a refutation for the need of a causal relationship is crazy talk. The mechanism may be missing, but the causal explanation is that via a series of RCTs it's established that the drug causes the effects.
I get that half of this is trying to go against a percieved need for intuitive explanations, but it weirdly lumps causation in there.