The Naive Bayes Forecasting Algorithm

The Naive Bayes algorithm doesn't require a lot of computing resources, which is a big advantage of this tool when we're forced to analyze a huge amount of data in a relatively short time, and it's very fast with high-quality forecasts, and the low requirements come from the fact that this algorithm is mostly based on average.

It's based on Bayes's assertion of the likelihood of independent classes.

W-MOSZCZYNSKI-2021-6-17

Why were Bayes' assumptions called naive?

The English mathematician Thomas Bayes invented a method of determining the probability of a subjective hypothesis based on the existing probability and new data, and this technique assumes the absolute independence of predictors. In other words, the descriptive variables must be completely independent of each other.

Predictive variables are called independent variables, but no one as strictly assumes their total, mutual independence, and this assumption has been generally considered highly naive, which is why the algorithm itself has earned this somewhat pejorative nickname.

Americans say that if something walks like a duck, quacks like a duck and has a duck’s bill, it must be a duck. According to Bayes, each individual characteristic of a duck is enough to recognize the bird as a duck. He assumed that all these characteristics independently contribute to the probability that the bird is or is not a duck.

The Naive Bayes model, despite its unusual nickname of Naive, is able to outperform the accuracy of analysis of some of the more sophisticated predictive models.

It is necessary to get a continuous flow of information on a regular basis.

The Thomas Bayes probability formula can be written as:

(P(cmid x)=frac{P(xmid c)P(c)}{P(x)})   (1)

(P(cmid x)=P(x_1mid c)times P(x_2mid c)timesldotstimes P(x_nmid c)times P(c))   (2)

where: (P(cmid x)) — probability of the class on the basis of a given predictor (x); (P(c)) — probability of occurrence of the class; (P(xmid c)) — probability of occurrence of the predictor for the given class; (P(x)) — probability of occurrence of the predictor.

To understand this pattern, let's use a simple example.

Naive Bayes in forecasting wheat prices

Imagine a stock market analyst who, based on his own experience and observations, has come to the conclusion that wheat prices on the stock market go up when the weather is nice and not up when the sky is cloudy.

It should be noted that the observations are discrete. Weather has been described as three states, while price increases are described as two states (yes/no). Weather is a predictor and price increases are the result. In this situation, we cannot determine the dependencies of predictor and result on the basis of ordinary correlation.

Table 1. Relationship between increases in wheat prices and the weather

Weather	Price increase
No cloud cover	yes
Heavy cloud cover	no
Moderate cloud cover	no
Heavy cloud cover	no
Heavy cloud cover	yes
No cloud cover	yes
No cloud cover	yes
No cloud cover	no
Moderate cloud cover	yes
Heavy cloud cover	no
Heavy cloud cover	yes
Heavy cloud cover	yes
Heavy cloud cover	no
Moderate cloud cover	no
No cloud cover	yes
Moderate cloud cover	yes
Moderate cloud cover	yes
Moderate cloud cover	no
No cloud cover	yes
No cloud cover	yes
Heavy cloud cover	no

the ch-square dependence indicator or the information capacity indicator, both of which are used to measure the interdependence of categorical phenomena.

In order to use the Bayes algorithm, a bit of data must be reformatted to the form shown in the table below.

The analyst noticed that when the day was cloudy, wheat prices rose three times.

Table 2. Probability of a price change depending on the weather

Weather	Yes	No	Total	Probability	Fraction
No cloud cover	6	1	7	0.33	7/21
Moderate cloud cover	3	3	6	0.29	6/21
Heavy cloud cover	3	5	8	0.38	8/21
Total	12	9	21
Probability	0.57	0.43
Fraction	12/21	9/21

The analyst made 21 observations of price increases, eight of which were made on cloudy days, so the probability that the price of wheat will rise on a cloudy day is 3/8 = 37.5%.

Now, let's compare the number of increases in wheat prices on a sunny day, and seven observations on a sunny day showed six increases in prices, so the likelihood of the price of wheat rising on a sunny day was high, and it was 6/7 = 85.7%.

The appropriate observation setting in Table 2 makes it possible to determine the likelihood of individual weather conditions and the likelihood of an increase in wheat prices.

Is it sunny, or is the price of wheat going up?

If it turns out to be true, our stock market analyst could make a pretty good profit on it.

We can test whether this hypothesis is true by using the above-mentioned probability method.

(P(text{yes}midtext{sunny})=P(text{sunny}midtext{yes})times P(text{yes})div P(text{sunny}))

Thus (P(text{sunny}midtext{yes})=6/12=0.50).

In the above pattern, out of the 12 observed increases in wheat prices, six times the increases occurred on a sunny day.

(P(text{sunny})=7/21=0.33).

The likelihood of a sunny day is 33% and the likelihood of a wheat price rise is 57%.

(P(text{yes})=12/21=0.57). Now, if we substitute our calculations

For the model (1) we get the following value:

(P(text{yes}midtext{sunny})=(0.50times0.57)/0.33=0.86).

So it turns out that our analyst is going to find something that would allow him to make more money on the stock exchange than his peers.

On the other hand, we discovered another advantage of the Naive Bayes model very low time-series length requirements. No other classification model could work properly with such a short time-series of variables.

Advantages and disadvantages of Thomas Bayes’ theorem

Some people may be annoyed that we call this simple pattern of probability a model, others may be uncomfortable with the construction of the pattern itself (1), where the de facto average is calculated from the averages, which we know seems to be an incorrect practice.

The Naive Bayes model is widely used for classification and regression for very large and complex data sets. It is treated on a par with very complex machine learning models, and it often proves to be better than them.

The Bayes model works better for discrete variables, not only dichotomous but also multi-class variables. For continuous data, the assumption of a normal distribution of variables is taken, which, combined with a small data set (which is possible in this algorithm), is a difficult condition to meet.

This model also has its drawbacks. The first of these is the so-called zero frequency if the

So if we were to add a fourth state of the weather to our categories, for example, foggy weather, the model, not knowing this state of the weather from the training set, would assign a zero probability to it, and it would reverse the whole algorithm, because we know it can't be divided by zero.

Another disadvantage is the independence of the predictive variables, which we've mentioned before, which is that predictive variables have to be independent for most models, but nowhere else is this approach taken so completely.

We're going to turn our example around and try to estimate what the weather is like based on increases and no increases in wheat prices. So the predictive variable would be either an increase or no increase in this case. This variable can be thought of as two predictive variables: price increase: 1 – yes, 0 no and price increase : 1 – yes, 0 no.

It's easy to see that the two variables are very strongly interdependent, and this makes it impossible to apply the Thomas Bayes model effectively.

Applications of Naive Bayes

The Naive Bayes model is used where a quick reaction is required, and it works perfectly in classification systems even where there is a very small collection of training data or where there are very few classes.

These characteristics make this algorithm ideal for filtering spam and generally classifying categorical characteristics in processes with a large number of classes.

Collaboration: Ewa Moszczyńska.

Wojciech Moszczyński — graduate of the Department of Econometrics and Statistics of Nicolaus Copernicus University in Toruń; specialist in econometrics, finance, data science, and management accounting. He specializes in the optimization of production and logistics processes. He conducts research in the area of the development and application of artificial intelligence. For years he has been engaged in the popularization of machine learning and data science in business environments.