Artificial Intelligence
Recommendation systems are used to optimize economic activity in the areas of sales and costs. In online sales, there are two entities subjected to recommendation systems: customers and products.
When discussing the creation of recommendations, we can also talk about creating tools that assist in strategic decision-making. This involves analyzing sales channels, directions of development, and eliminating areas that burden the business. However, these types of strategic recommendations do not fit the topic I would like to discuss today.
Thus, the main focus of the basic recommendation system is the customer and the product.
What is the goal of recommendation systems for customers on e-commerce platforms?
Essentially, there are two goals that should be discussed separately.
First – it is about encouraging the customer to purchase more. Recommendation systems can significantly increase the volume of purchases made by individual customers in a simple and inexpensive manner.
Second – it aims to increase customer retention rates. The goal here is to prevent customers from leaving the stores. It is obvious that it is much easier to lose a customer than to gain a new one. This belief leads to a concern to do everything possible to retain customers.
Both of these goals: increasing purchasing power and retaining customers are achieved through different, very distinct tools. I mentioned that these goals should be considered separately. If we focus on customer retention, we should not aim to increase their purchasing effectiveness.
On one hand, this is true because such an approach aligns with the iron law of optimization, which states that one can only optimize one thing at a time, not many things simultaneously. On the other hand, it contradicts intuition and experience. If we have caused, through recommendation systems, that a customer started purchasing more than before, it is likely related to the fact that the customer was very satisfied with their collaboration with the store. A satisfied customer will stay in the store longer. Of course, there can always be a counterargument – customers can be drawn in by promotions and things they do not need or want. Customers, whether consciously or unconsciously manipulated, usually lose trust and leave. This is a good argument indicating that the recommendation system must operate on fair principles. Only then will it positively influence customer retention alongside revenue optimization.
Recommendation Systems Must Predict the Future
The recommendation system is based on historical events. Its operation relies on a list of transactions made by customers in the past. However, the true goal of a recommendation system is to predict what a customer will do. Predicting the future is the foundation of optimization. This optimization pertains to both the current purchasing process and the future supply of products in the store.
By serving an individual e-commerce customer, the recommendation system can predict which products to suggest to increase the chances of additional sales. At the same time, the recommendation system can help forecast the demand for specific products in the future.
Effective forecasting of the future is therefore the foundation of an efficient recommendation system.
What is the goal of creating recommendation systems for products?
Here we must point out a certain inconsistency. A recommendation system, as the name implies, serves to recommend or suggest something. Can we suggest anything to products? However, it is possible to optimize the structure of these products; we can observe trends and processes using statistical tools. This way, it is possible to optimize the product structure in the store, sending appropriate assortments to places where they are very much needed and removing them from places where they are not popular. With a recommendation system for products, it is possible to optimize the sales process and achieve high profits or reduce costs.
The primary entity to which the recommendation system is directed is the customer, as the customer makes decisions and is a somewhat unstable element that must be observed from a predictive perspective.
The recommendation system analyzes the collision of customer behaviors with a kind of „behavior” of products. Each customer has a unique behavior pattern that rarely changes. Products have certain periods of popularity, and they also have some connections with each other. Customers themselves validate products and the relationships between them. The role of data science is to catch these links between customers and products.
The most important tool in recommendation systems is clustering entities into clusters. This applies to both customers and products. In clusters, similar objects are grouped based on certain features. It is crucial to indicate those features that are important for the sales process. The above description may seem somewhat convoluted and unclear, so below I will try to clarify, step by step, how to create recommendation systems.
Internal Data Sources for E-commerce Recommendation Systems
The primary data source in e-commerce sales systems is the sales register, which includes customer logins. The purchase history of individual customers is tracked and modeled by data analysts.
Similar registers exist in retail sales, where customers use loyalty cards. According to research I conducted a year ago, about 70% of customers shopping in the largest Polish retail chains entered loyalty card codes during transactions. Customers are effectively encouraged to register in the store’s system. Retail chains like Biedronka use a number of promotions. Some of these are available exclusively to those who have registered their customer card. But let’s return to recommendation systems for e-commerce platforms.
Another source of data for e-commerce recommendation systems is static customer data. This data is collected when a customer account is created and is not present in the sales register.
Anonymous Customers
Some customers making purchases do not register with online stores. Sometimes customers do not disclose themselves in the system for various reasons.
Thus, there are transactions without customer ID information. However, these transactions contain a range of information that allows systems to easily match customer data with the customer ID number. This includes the credit card number, phone number, and address to which the customer wishes to receive the purchased product. We can also mention the unique address of the device from which purchases are made. A customer identified by metadata, who did not wish to reveal themselves, can be assigned to customer transactions using simple applications operating within the e-commerce system.
It should be noted that customers who see that they can be identified by intelligent recommendation systems, even though they did not disclose their data, may lose trust in the store, which could lead them to leave.
Multiple Accounts Customers
Multiple account situations occur when one customer account serves multiple people. For instance, a production company has one ID code in the online store, and purchases are made for all departments of the company. Hydraulic pipes, valves, mattresses, power tools, and paints are purchased. Worse still, purchases are often made by different people with varying behavior patterns. Private purchases can also be made from a corporate account.
In general, multiple accounts disrupt the efficiency of e-commerce systems. They should be eliminated, just like outlier values in modeling. Unfortunately, multiple accounts are often high-turnover accounts. One way to deal with such accounts is to encourage the company to create multiple accounts in exchange for discounts and promotions for specialization.
External Data Sources
Alongside the information stored in the online store’s databases, external data is also important. Unfortunately, customers do not make decisions solely based on what they see in the store. A large part of their decisions is based on external information. If someone previously saw dishwashing liquid for 5 PLN in another online store, they will not be tempted to buy dishwashing liquid for 8 PLN. A recommendation advertisement that appears at the bottom of the purchasing window with dishwashing liquid for 10 PLN is likely to discourage the customer rather than encourage further purchases.
The same customer will be tempted to buy dishwashing liquid for 8 PLN if they receive something in return. To propose an attractive offer, the system must be perfectly situationally aware. It must know the customer, but it also must know the market and the competition’s offer.
Information about prices and availability of competitive products should therefore be included as a variable in the recommendation models. Predictive models should also contain a multitude of regular external information, such as the day of the week, month, season, and many important situational variables such as weather forecasts and consumer optimism indices. The system should know the trends and fashions in the market. All this information is processed as variables in the realm of digital mathematical models.
Clustering the Customer Population
Technically, it is relatively possible to predict the behaviors of each customer accurately. Buyers behave in a repetitive manner. However, no one tries to treat customers individually, as this solution would be inefficient. The goal of the clustering process, which is a more effective form of simple grouping, is to find customers who have very similar behaviors. A customer has individual habits and routines, and their behavior is similar to that of others. The system attempts to assign individuals to a cluster of people very similar in specific features. They become a kind of digital twins, acting very similarly, sharing the same phobias, preferences, purchasing frequencies, and sensitivity to advertisements. A population consisting of 100,000 customers can thus be divided into 5,000 clusters. People assigned to specific clusters will be treated differently by the sales system. For each of these groups, a different form of incentive and persuasion will be applied. Excellent results come from finding individuals within a cluster who deviate from the others and who, thanks to knowledge of the traits of other individuals in the cluster, can be easily persuaded to make larger or more efficient purchases.
Transposition of Divisions
Assigning customers to clusters can occur according to various variables. Initially, an RFM approach (Recency, Frequency, and Monetary) can be applied, where customers are grouped based on the frequency and volume of their purchases. This way, a preliminary analysis of customers is conducted. Then, these customers can be grouped based on the assortment they choose. This way, the first set of customer clusters can be formed. Now transposition can be done, using RFM grouping to overlay assortment clusters and analyze the customer distributions in these transpositions. This way, transposition matrices are created, indicating market niches, anomalies, and areas where the sales process is unnaturally excessive.
Customers can be assigned to various sets of clusters, and within each of these sets, correlations or non-linear relationships can be analyzed with other simple divisions such as seasonality, multiplicity, place of residence, gender, or day of the week. We will understand that the ability to analyze customer behaviors is practically endless.
Similarly, products can also be grouped; they can be clustered based on sales frequency, turnover, or seasonality. Alongside simple groupings, complex partitioning of product populations can be created. Product and customer clusters can be matched, creating infinite patterns of behavior. The most important thing is to find the repeatability of customer behavior patterns in relation to related products. The ability to eliminate excessive information is crucial here.
Simple E-commerce System in Practice
Let’s assume a customer appears on the e-commerce platform. This customer is one of 140 customers grouped in one cluster. In this cluster, customers behave similarly. This customer has placed a hammer in their cart. The system immediately finds a second item related to it. This is a screwdriver. It turns out that in this cluster, customers who chose a screwdriver usually also bought a hammer, and if they chose a hammer, they also bought a screwdriver. Among the 140 customers, 38 made this choice in the past. The system estimates that the customer who selected a hammer on the e-commerce platform has a 24% probability of shortly adding a screwdriver to their cart. At the same time, the system knows that there is a 76% chance that this customer will not add a screwdriver to their cart. The high likelihood of purchasing a screwdriver causes the system to display the recommendation. The customer sees several screwdrivers on the shopping platform. We do not know if, without this advertisement, the customer would have purchased a screwdriver. It should be assumed that customers in this cluster, if they do not receive advertising about screwdrivers, will likely not buy one. However, the applied recommendation drastically increases the likelihood of purchasing this tool. This is how a system based on two important tools works, namely customer clustering and the creation of so-called recommendation pairs.
Recommendation Pairs
Products sold in pairs or trios can be searched without relying on customer clusters. If someone buys a dish sponge, they are likely also going to buy dishwashing liquid. For some reason, however, some customers do this, while others buy only sponges. This may be due to the fact that the sponges are not intended for washing dishes but, for example, for cleaning car rims.
Or perhaps sponges and dishwashing liquid are bought more often by women, while sponges without dishwashing liquid are bought more often by men. This is a typical behavior pattern where the customer’s belonging to a specific group, cluster, plays an important role. This simple example indicates that it is important for recommendation systems to use pairs based on clusters. Such individual patterns are found thousands of times. Fortunately, tools search for and select them en masse.
Analysis of Incomplete Purchases in a Cluster
As mentioned earlier, customers assigned to the same cluster exhibit a specific set of behaviors. For example, let’s take a cluster of people who purchase roofing-related articles. The algorithms consider the frequency and value of purchased items and the type of customer (for example, small business). A substantial group of customers is classified as small contractor roofers. Now it is possible to analyze the completeness of their purchases. Some customers buy within certain categories offered by the store, such as insulation materials, gutters, tiles, bituminous coverings, and adhesives.
Let’s assume that the completeness analysis showed that 27% of customers in this cluster buy bituminous coverings but do not purchase adhesives. However, the technology for creating bituminous coverings requires the use of adhesives. It follows that a significant portion of customers does not buy adhesives from our store but purchases them elsewhere. This is an example of identifying a market niche.
There is nothing left but to offer this excellent adhesive deal exclusively to the 27% of customers from the roofing cluster. It is crucial for customers to change their habits and start buying adhesives in our store.
Analysis of Interrupted Habits
Grouping customers into purchasing clusters facilitates the detection of changes in their habits. Let’s assume we have a cluster of plumbers. Seasonally, small hydrofor pumps were purchased in this cluster. The seasonality resulted from the fact that, in spring, it often turned out that small hydrofor pumps installed in recreational gardens needed replacement. However, at some point, it was found that hydrofor pumps were not being sold in the spring within the plumbers’ cluster.
If we had analyzed the level of hydrofor pump sales without clusters, we might not have detected this anomaly. Hydrofor pumps can be purchased by various customer groups. However, we are most interested in plumbers. Thanks to the detected anomaly, it is possible to investigate the situation and propose better conditions so that plumbers do not buy hydrofor pumps from competitors. This action will not be automatically executed by the recommendation system, but the system allows for such actions by sales managers.
How to Retain Customers in the Store?
Until now, my considerations have focused on increasing the purchasing efficiency of customers. One customer purchasing a hammer may, thanks to a well-displayed advertisement, also buy a screwdriver. A woman buying dishwashing liquid receives recommendations for sponge purchases.
Thanks to these and thousands of similarly effectively utilized patterns, the overall average purchasing efficiency of customers can increase by 30-40%.
They will also feel an improvement in the shopping experience because a well-fitted suggestion is usually well-received by customers.
However, an increase in the purchasing efficiency of customers may not compensate for a decline in sales due to the departure of some customers. Customers usually leave for some reason, often due to a poorly fitted treatment format or offers from other stores. The reason may also be unrelated to the store’s operations. To avoid losses resulting from our store’s operations, another recommendation system must be created, focused solely on customer retention.
Here too, we must carry out clustering and divide customers into groups that consist of digital twins. Then it is necessary to initiate survival tests.
The survival test is a method developed long ago for medical facilities, aimed at statistically analyzing the survival time of patients with a certain probability concerning the time based on currently applied medical treatments. Various treatments were tested this way, and based on that, tools calculated the probability of future patient deaths. Thus, applying treatment for specific patient groups predicted, for example, that they had an 80% chance of surviving the first year, a 52% chance of surviving the second year, and a 10% chance of surviving the third year. This tool was quickly adapted by commercial companies. The concept of a patient’s death was replaced with the concept of a customer leaving the store.
With this method, it is easy to determine the likelihood of retaining a customer for the upcoming years. Various incentives can be added to the algorithm, such as loyalty cards, discounts, periodic individual discounts, and information on the effect of these techniques on customer retention. This relatively simple approach can drastically reduce the likelihood of customer departure. Interestingly, such a system can also serve for the ongoing management of customers. It can be a handy advisor for sales representatives talking to particular customers.
Analysis of Purchase Cancellations
An important element preceding the decision for the customer’s approach from the e-commerce platform is the purchase cancellation rate. A customer may decide to leave the store because they have not found a suitable offer for an extended period. And such a decision will also not be detected by the survival test.
To create a purchase cancellation rate, it is necessary to reconfigure the data collection system. As mentioned earlier, the recommendation system is built based on the sales register. The sales register consists of transactions that were concluded with the customer within a certain period and for certain products. If a customer enters the online store and browses different pages, this behavior will not be found in the sales register.
The only way to analyze customer behavior is through the history of browsing the online store’s pages. This requires the application of specialized tracking tools, but this endeavor is cost-effective because it effectively indicates customer fluctuations and their purchasing determination.
How to Build a Recommendation System?
The most crucial aspect in building a recommendation system is to identify the goal. Fortunately, usually, all companies want the same thing, which is an increase in customer purchasing efficiency. At the same time, e-commerce stores most often want to identify the factors contributing to customer departures. Both of these goals can easily be reconciled based on simple principles for building a recommendation system.
The most important elements are data, which must be complete, reliable, and comprehensive. It is impossible to create an effective recommendation system based on a small amount of data. A sales transaction record containing at least several hundred thousand operations should be linked to other databases obtained from external sources.
The next step is the proper use of data clustering methods. Often, with many variables, the PCA algorithm (Principal Component Analysis) is used. This is a method of combining many features into 3 or 4 consolidated component features. Partial features are combined into main features based on the strength of mutual covariances among features.
Let’s imagine information about a customer and try to count how many features they can have. This may include residential address, number of transactions per month, number of different product categories, payment method, and dozens of other variables. With such data clouds, it is impossible to effectively assign a customer to a cluster. That is why various tools are used to reduce the number of variables through reduction or grouping into large variables using the PCA algorithm.
Survival analysis, like tools that increase customer purchasing efficiency, is a method simple to create and manage. At the same time, the methodology is considered tedious because there is a vast amount of interactions and external changes that require experience from the researcher. A very important skill is maintaining the correct research priorities, involving the elimination of inefficient patterns and expanding effective ones. This requires high self-discipline and patience.
First Prototype, Then Implementation
In creating a recommendation system, a prototype is first built, usually written in Python and based on the language’s libraries. The prototype is tested and refined in a programming environment. Unfortunately, the prototype is not suitable for use in the sales process. This solution must be implemented in a production environment. This is the responsibility of the data engineer, who implements the program in a cloud environment or in a local environment that supports sales systems.
Batch and Streaming Systems
Recommendation systems can be based on historical records. The prototype is built on a certain history that changes over time. Therefore, the system must be updated. Someone who previously bought yogurts suddenly began buying kefirs. Someone who once bought a hammer and screwdriver is unlikely to want to repeat their decisions. However, we know that a customer who once bought a hammer and screwdriver will, after some time, want to buy a drill. These examples illustrate the necessity of periodically updating the recommendation system. The more frequently this update occurs, the better. Ideally, it is when the update is conducted continuously. Therefore, it is possible to build a recommendation system based on a constant flow of information in the form of data streams and process them into the recommendation system.
Summary
Everyone knows that a well-formulated, tailor-made offer increases customer purchasing efficiency and additionally enhances customer loyalty to the store. The problem is that to tailor an offer for a customer, we must know a lot about them and use tools that will ensure this knowledge is utilized effectively.
To achieve such an endeavor, it is necessary to organize data and the processes for their processing and utilization in analysis perfectly.
The process is difficult but necessary because if we do not do it, the competition will. A recommendation system on an e-commerce platform can be an excellent source of competitive advantage, allowing for greater sales efficiency without incurring substantial expenses on general advertising or excessive, broad promotions, thus securing higher profits that ensure the sustainability and increasing value of the e-commerce business.
Wojciech Moszczyński
Wojciech Moszczyński is a graduate of the Department of Econometrics and Statistics at Nicolaus Copernicus University in Toruń, specializing in econometrics, finance, data science, and management accounting. He specializes in optimizing production and logistics processes. He has been involved in researching the development and application of artificial intelligence for years. He has been engaged in popularizing machine learning and data science in business environments.