Using Bayes to dismiss fringe phenomena

Observations of fringe phenomena like UFO's, ghosts or tooth fairies are often rare and vague, and one possible reason is that these phenomena are simply purely imaginative. Phenomena with low observation rates have statistical properties that might help us answer the following question: what is the a priori probability that a specific observation $$X$$ of fringe phenomena $$Y$$ implies that observation $$X$$ actually originates from $$Y$$? It might also help answer the more general question "what is the a priori probability that a series of observations $$X$$ of fringe phenomena $$Y$$ implies that observations $$X$$ actually originates from $$Y$$?". However, in this article I only pursue the first question.

The 'a priori' clause in the question means that we derive the probability without having to investigate the actual empirical data that constitutes $$X$$. This is very convenient, as investigating all the various instantiations (particular observations) of $$X$$ would take a herculean effort. The price we pay is that we only get to talk about a (heuristically generated) probability of '$$X$$ implying $$Y$$'. Anything else would require actual empirical investigation of each instantiation of $$X$$.

In this article I'll use UFO observations as a case phenomena, seeing that good statistical data exists on UFO observations. This allows us to establish a prior probability of reasonable quality. The treatment of UFO's presented in this article should apply to your pet fringe phenomena as well, by analogy. In a quest for stringency I'll substitute the abbreviation 'UFO' with 'UAP' ('Unknown Aerial Phenomena'). A UAP could be 'a flying saucer', 'The Millenium Falcon', 'a witch on a broomstick' etc. (assuming these actually exists). Here the word 'unknown' means that the phenomena in question isn't recognized as existing by the majority of scientists. Given this definition of 'unknown', a high quality observation of a UAP could hypothetically occur, without having the UAP immediately turn into an Identified Aerial Phenomena, if for no other reason than the latency of novell observations turning into established facts of science.

Disclaimer: I do not claim that (nor do I know whether) aliens have visited earth, and that question is besides the point of this article. The question at hand is: can we answer such a question prior to investigating potential evidence herefor?

In this first approach to dismissing fringe phenomena using Bayes, I take a hypothetical observation $$X$$ that in itself has a high probability of originating from fringe phenomena $$Y$$ (that is, $$P(X|Y)$$ is high) and show how $$P(X)$$ diminishes when the history of earlier observations are taken into account. You could of course also use Bayes to dismiss observations that in themselves merely have a low $$P(X|Y)$$, but that would be a somewhat less impressive feat than dismissing any (hypothetical) observation of high probability. Observations with low $$P(X|Y)$$ would basically dismiss themselves, without the need for fancy Bayesian analysis.
Do high probability observations exist at all? This question is beyond the scope of this article, but if you want to look into this question, there exists a bunch of case collections describing various observations .

To see how the Bayesian formula reduces $$P(X)$$, we first need to adapt it to our fringe phenomena of choice: UAP's and the observations of UAP's.

This is the vanilla version of Bayes formula:

$$P(A|B) = \dfrac { P(B|A) P(A) } { P(B) }$$

where $$A$$ and $$B$$ are types of events.

First we'll define the ingredients we need to transform the vanilla version into one that can answer our question:

$$UAP$$: the event that an observation $$X$$ originates from any Unknown Aerial Phenomena.

$$UO$$: the event that a human having an observation $$X$$ categorises $$X$$ as originating from an unknown aerial phenomena.

$$P(UAP)$$: the probability of an observation $$X$$ originating from an unknown aerial phenomena.

$$P(UO)$$: the probability that a human having an observation $$X$$ categorises $$X$$ as originating from a specific unknown aerial phenomena.

$$P(UAP|UO)$$: the probability of an observation $$X$$ originating from an unknown aerial phenomena, given that a human having observation $$X$$ categorises $$X$$ as originating from an unknown aerial phenomena.

$$P(UO|UAP)$$: the probability that a human having an observation $$X$$ categorises $$X$$ as originating from an unknown aerial phenomena, given that observation $$X$$ originates from an unknown aerial phenomena.

$$P(UO| \neg UAP)$$: the probability that a human having an observation $$X$$ categorises $$X$$ as originating from an unknown aerial phenomena, given that observation $$X$$ doesn't originate from an unknown aerial phenomena.

Now we can write the UAP-adapted Bayesian formula like this:

$$P(UAP|UO) = \dfrac { P(UO|UAP) P(UAP) } { P(UO) }$$

Now say we have a specific observation $$UO_1$$ from which we want to assess the posterior probability $$P(UAP|UO_1)$$. In human words, this corresponds to asking the question "what is the probability that a observation $$UO_1$$, categorised by the observer as originating from a $$UAP$$, implies that observation $$UO_1$$ actually originates from any particular $$UAP$$? In this case we'll apply the formula to this particular observation, like this:

$$P(UAP|UO_1) = \dfrac { P(UO_1|UAP) P(UAP) } { P(UO_1) }$$

All we have to do now is to estimate or find the ingredients on the right hand side, and we can calculate the value of the left hand side.

$$P(UO_1|UAP)$$ : If it was actually the case that a $$UAP$$ happened, what is the probability of it giving rise to the observation data of $$UO_1$$? Assuming we are dealing with a high probability observation we could tentatively set $$P(UO_1|UAP)$$ to be $$0.8$$.

$$P(UAP)$$: What is the prior probability that a broadly unknown aerial phenomena UAP exists? Making such an estimate without relying on empirical data will inevitably result in the estimate being very speculative. However, this is where Bayes will really shine for us: If we have an earlier calculation of a posterior $$P(UAP|UO)$$, we can reuse this to feed into the current calculation as the $$P(UAP)$$ value. This kind of chaining of calculations is a hallmark feature of Bayesian reasoning, and it is the raison d'ĂȘtre for the Bayesian methology when trying to dismiss a new observation without looking at the data of that observation.
Thus the question becomes: do we have such a calculation for UAP's? Fortunately yes, as there exists large bodies of observation data for UAPs. One governmental study of 3200 observations concluded that 22% of the observations could not be identified. These were separate from another 9% of cases which could not be identified due to insufficient observation data. In other words, the claimed to have sufficient observation data to eliminate all known phenomena as being possible causes for 22% of the observations in the study. Does this imply that "$$P(UAP|UO) = 0.22$$"? Unfortunately not quite: Even trained researchers might not be omniscient on all phenomena known to man kind: if some UO was caused by a smudge on the lense of a camera, and the researcher in question simply failed to identify lense smudges as being a possible cause, the researcher would think all possible explanations to have been ruled out, without this actually being the case.
At this point I can choose to go with a generic 0.5 estimate of $$P(UAP)$$, basically short circuiting the chain of Bayesian reasoning, or I can go along with $$P(UAP) = 0.22$$, keeping in mind that it is an upper bound. I choose the latter.
There are by the way many other reports on UFO observations. If you find a better data source from which to derive a prior probability please let me know in the comments.

$$P(UO_1)$$: What is the probability that a human having an observation $$UO_1$$ categorises $$UO_1$$ as originating from an unknown aerial phenomena? We don't need to (and for the sake of internal consistency, shouldn't) estimate this probability on its own. This is because it can be expressed in terms of already known factors:

$$P(UO_1) = P(UO_1|UAP) \cdot P(UAP) + P(UO_1| \neg UAP) \cdot P( \neg UAP)$$

Previously we decided $$P(UO_1|UAP)$$ to be $$0.8$$ and $$P(UAP)$$ to be $$0.22$$.

But how to decide $$P(UO_1| \neg UAP)$$? Well, this is just as undecided as $$P(UO_1|UAP)$$ which means that we must once again just choose something for the sake of continuing the argument. I'll choose two values, a high and a low, and see how both of these affects the final outcome. $$P_A(UO_1| \neg UAP) = 0.2$$ and $$P_B(UO_1| \neg UAP) = 0.8$$.
Finally, $$P( \neg UAP)$$ must be $$1 - 0.22 = 0.78$$.

This gives:

$$P_A(UO_1) = 0.8 \cdot 0.22 + 0.2 \cdot 0.78 = 0.332$$

$$P_B(UO_1) = 0.8 \cdot 0.22 + 0.8 \cdot 0.78 = 0.80$$

and thus:

$$P_A(UAP|UO_1) = \dfrac { 0.8 \cdot 0.22 } { 0.332 } = 0.53$$

$$P_B(UAP|UO_1) = \dfrac { 0.8 \cdot 0.22 } { 0.80 } = 0.22$$

Conclusion

With $$P(UO_1| \neg UAP)$$ being low, we achieved a moderate reduction of $$P(UAP|UO_1)$$. However, as it only got pulled down to about fifty percent chance of originating from a $$UAP$$, this would imply that one should withhold judgement regarding $$UO_1$$ until a scientific investigation of the observation has been carried out, or until existing scientific investigations of $$UO_1$$ has been reviewed.
With $$P(UO_1| \neg UAP)$$ being high, the reduction of $$P(UAP|UO_1)$$ was somewhat more impressive. So, when the observation could just as well have been from a known aerial phenomena, we can be somewhat sure that the observation in question indeed originated from a known aerial phenomena.

Future work

As noted in the introduction, a Bayesian approach could also be used to answer questions regarding a series of observations. Series of observations roughly divide into two kinds:

1. A series of $$n$$ observations of the same physical occurrence of some phenomena.
2. A series of $$m$$ observations of $$n$$ different occurrences of some phenomena.

The questions we would want to answer for these two cases would be:

1. What is the a priori probability that a series of observations $$UO_1...UO_n$$, claimed to be of a single occurrence of an $$UAP$$, implies that observations $$UO_1...UO_n$$ actually originates from a particular $$UAP$$?
2. What is the a priori probability that a series of observations $$UO_1...UO_m$$, claimed to be of a series of $$n$$ occurrences of some $$UAP$$, implies that at least one of $$UO_1...UO_m$$ actually originates from an $$UAP$$?

The latter question seems to me to be the most interesting, as it strives to include as much of the available data material on UAP observations as possible.

The former question seems to fit very neatly a deduction presented by J. W. Deardorff on the question on how to sum probabilities regarding observations of a particular event.

End notes

• What about the False Positive Paradox? Doesn't it basically nullify the human ability to identify rare phenomena?
A prerequisite for FPP to occur is that one has two classes of phenotypically similar observations. As an example, this is the case when trying to distinguish between 'good' and 'bad' tumours in medical imaging. As a counter-example, it is not fulfilled between the observations of cars and the observations of head lice. Even though I've seen many cars in my life, and I've never seen a head lice, their phenotype is simply too dissimilar for a sane human being to misidentify a head lice for a car.
Bringing the topic back to UAP's, we can conclude that to use FPP to dismiss UAP's, we first need to establish that all the claimed observations of UAP's are sufficiently similar to observations of well known earthly phenomena. Again, this will require research, and again this means we cannot conclude much about the topic from a position of prior reasoning.

• How do we chose an estimate of the prior probability? Unrealistic or unfounded estimates of prior probabilities leads to unrealistic or unfounded posteriors, so we need to choose wisely. The prior should ideally be based on a large body of earlier observation data. If no prior observation data are available, the prior should be set to 0.5 and then the posterior probability should not be trusted until the prior has been cycled through a large number of Bayesian updates, each occurring when a new piece of observation data is acquired.

• I recently read an article about 'Elk abductions' - yes, it's about Elks seemingly being lifted of the ground into weird flying objects. My initial reaction was to laugh out about how ridiculous that seems to me. It's almost silly!
However, on second thought, I don't think my reaction was indeed very scientific or rational. Elk abductions could indeed be occurring, no matter how silly I find the idea.
What's the name for this kind of emotional reaction, and how do I avoid having it inhibit my rationality?