Recommendation systems are used to optimize economic activity in the areas of sales and costs. In online sales, there are two entities that are subjected to recommendation systems: customers and goods.
When speaking about creating recommendations, we can also speak about creating tools that assist with strategic decisions. This concerns analyses of sales channels, directions of development, and the elimination of areas that burden an enterprise. Strategic recommendations of this kind, however, do not fit the subject that I would like to discuss today.
Thus, the subjects of a basic recommendation system are the customer and the product.
W-MOSZCZYNSKI-2024-8-39What is the purpose of recommendation systems for customers of e-commerce platforms?
In principle, two objectives can be distinguished, and they should be discussed separately.
First, the aim is for the customer to buy more. In a fairly simple and inexpensive way, recommendation systems can significantly increase the volume of purchases made by individual customers.
Second, the aim is to increase the customer-retention rate. The objective here is to prevent customers from leaving stores. It is obvious that it is much easier to lose a customer than to acquire a new one. This belief gives rise to the concern with doing everything possible to retain customers.
Both objectives—increasing purchasing capacity and retaining customers—are achieved by different, very dissimilar tools. I mentioned that these objectives should be considered separately. If we focus on retaining customers, we should not strive to increase their revenue efficiency.
On the one hand, this is true, because such an approach is consistent with the iron rule of optimization, which states that only one thing can be optimized, rather than many things at once. On the other hand, it is inconsistent with intuition and experience. If, by means of recommendation systems, we have caused a customer to begin buying more than before, this was most probably connected with the customer being very satisfied with the relationship with the store. A customer who is satisfied with that relationship will remain with the store for longer. Of course, a counterargument can always be raised: customers can, after all, be talked into promotions and things that they do not need or want. Customers who are consciously or unconsciously manipulated usually lose trust and leave. This is a good argument showing that a recommendation system must operate according to honest rules. It then has a positive effect not only on revenue optimization, but also on the growth of customer retention in the store.
A recommendation system must predict the future
A recommendation system is based on historical events. Its operation is founded on a list of transactions made by customers in the past. The true purpose of a recommendation system, however, is to predict what the customer will do. Predicting the future is the basis of optimization. This optimization concerns both the current purchasing process and the store’s future supply of products.1
When serving an individual e-commerce customer, a recommendation system can predict which products to propose in order to increase the chances of an additional sale. At the same time, a recommendation system can help predict future demand for particular products.
Effective prediction of the future is therefore the foundation of an effective recommendation system.
1 This refers to replenishment systems, which predict demand for individual products in advance and initiate the process of ordering them at the appropriate time.
What is the purpose of creating recommendation systems for goods?
A certain inaccuracy should be pointed out here. A recommendation system, as its name indicates, serves to recommend something—to suggest something. Can anything be suggested to goods? One can, however, optimize the structure of those goods and observe tendencies and processes using statistical tools. This makes it possible to optimize the structure of goods in the store, send appropriate product ranges to places where they are badly needed, and remove them from places where they are not very popular. A product recommendation system makes it possible to optimize the sales process and obtain high profits or reduce costs.
The principal entity to which a recommendation system is directed is the customer, because it is the customer who makes decisions and is, to some extent, an unstable element that must be observed from a predictive perspective.
A recommendation system analyzes collisions between customer behavior and a kind of “behavior” of goods. Every customer has a unique behavior pattern that changes very rarely. Goods have certain periods of popularity and also have certain connections with one another. Customers themselves validate goods and the connections between them. The role of data science is to detect these relationships between customers and goods.
The most important tool in recommendation systems is the grouping of entities into clusters. Both customers and goods are grouped in this way. Objects that are similar in terms of specified characteristics are grouped into clusters. It is extremely important to identify those characteristics that matter to the sales process. The description above may seem somewhat convoluted and difficult to understand, so below I will try to explain, step by step, how recommendation systems are created.
Internal data sources for e-commerce recommendation systems
The primary source of data in e-commerce sales systems is the sales register, together with the customer logins recorded in it. The purchase history of individual customers is tracked and modeled by data analysts.
Similar registers occur in retail sales, where customers use loyalty cards. According to research that I conducted a year ago, approximately 70% of customers shopping in Poland’s largest retail chains entered loyalty-card codes during their transactions. Customers are effectively encouraged to register in the store’s system. Retail chains such as Biedronka use a range of promotions. Some of them are available exclusively to people who have registered their customer card. Let us return, however, to recommendation systems for e-commerce platforms.
Another source of data for e-commerce recommendation systems is the customer’s static data. These data are obtained when the customer account is created and are not present in the sales register.
Anonymous customers
Some customers making purchases do not register with online stores. Sometimes, for some reason, customers do not reveal themselves in the system.
Transactions therefore appear without customer ID information. These transactions nevertheless contain a range of information that allows systems to match the customer’s data to the customer’s ID number with ease. This includes a credit-card number, telephone number, and the address to which the customer wishes the purchased product to be delivered. We may also mention the unique address of the device from which purchases are made. A customer identified from metadata, who did not wish to reveal themselves, can be assigned to transactions by simple applications operating as part of the e-commerce system.
It should be remembered that customers who notice that intelligent recommendation systems can identify them even though they did not disclose their data may lose trust in the store, and this may prompt them to leave.
Multiple-user customers
Multiple-user accounts arise when one customer account is used by many people. For example, a manufacturing company has one ID code in an online store, and purchases for all of the company’s departments are made there. Hydraulic pipes and valves, mattresses, power tools and paints are purchased. Worse still, purchases are often made by different people who have different behavior patterns. Employees may also make private purchases using the company account.
In general, multiple-user accounts are an element that disrupts the effectiveness of e-commerce systems. They should be eliminated in the same way that outliers are removed in modeling. Unfortunately, multiple-user accounts are usually accounts with very high turnover. One way of dealing with such accounts is to encourage the company to create multiple accounts in exchange for discounts and promotions for specialization.
External data sources
In addition to information collected in the databases of the online store, external data are also important. Unfortunately, the customer does not make decisions solely on the basis of what they see in the store. Customers base a large part of their decisions on external information. If someone previously saw a cleaning liquid for PLN 5 in another online store, they will not be tempted to buy dishwashing liquid for PLN 8. A recommendation advertisement appearing at the bottom of the shopping window for dishwashing liquid priced at PLN 10 will discourage the customer more than encourage further purchases.
The same customer will be tempted to buy dishwashing liquid for PLN 8 if they receive something in return. To propose an attractive offer, the system must have an excellent understanding of the situation. It must know the customer, but it must also know the market and the competitors’ offerings.
Information about the prices and availability of competing goods should therefore be included as a variable in recommendation models. Predictive models should also contain a great deal of regular external information, such as the day of the week, month and season, as well as many important situational variables, such as the weather forecast and the consumer confidence index. The system should know the trends and fashions in the market. All of this information is processed as variables in the world of digital mathematical models.
Clustering the customer population
Technically, it is possible to predict the behavior of every customer with relative precision. Buyers behave in repeatable ways. Nobody tries to treat customers individually, however, because such a solution would be inefficient. The purpose of clustering—a more effective form of simple grouping—is to find customers whose behavior is very similar. A customer has individual customs and habits, but their behavior resembles that of other people. The system attempts to assign to a cluster people who are very similar in terms of specified characteristics. They become a kind of digital twin: they act very similarly and have the same phobias, preferences, purchase frequencies and sensitivity to advertising. A population of 100,000 customers can therefore be divided into 5,000 clusters. People assigned to particular clusters will be treated differently by the sales system. A different form of encouragement and persuasion will be used for each of these groups. Excellent results can be achieved by finding, within a single cluster, people who differ from the others and who—thanks to knowledge of the characteristics of the other people in the cluster—can easily be encouraged to make larger or more effective purchases.
Transpositions of divisions
Customers can be assigned to clusters according to various variables. Initially, the RFM2 (Recency, Frequency and Monetary) approach can be applied, in which customers are grouped according to the frequency and value of the purchases they make. In this way, a preliminary customer analysis is performed. The customers can then be grouped according to the product range they selected. This can create the first set of customer clusters. A transposition can now be performed: using RFM grouping, we can superimpose product-range clusters and analyze the concentrations of customers in the individual transpositions. This produces transposition matrices that indicate market niches, anomalies and places in which the sales process is unnaturally excessive.
Customers can be assigned to various sets of clusters, and in each of these sets they can be analyzed in terms of correlation or nonlinear relationships3 with other simple divisions, such as seasonality, multiplicity, place of residence, gender or day of the week. We can see that the possibilities for analyzing customer behavior are practically infinite.
Goods can similarly be grouped according to sales frequency, turnover or seasonality. In addition to simple groupings, complex clustering of the population of goods can be created. Clusters of goods and customers can be compared with one another, creating an infinite number of behavior patterns. The most important thing is to find the recurrence of customer behavior patterns in relation to interconnected goods. The ability to eliminate excess information is very important here.
2 “Mastering E-commerce Product Recommendations in Python. Using RFM Scores and TF-IDF Scores for Product Recommendations,” Sadrach Pierre, Ph.D. (https://medium.com/datafabrica/mastering-e-commerce-product-recommendations-in-python-7c12a4bf0c2c)
3 The relationship between RFM grouping and a set of product-range clusters is not a linear relationship, because RFM is considered in terms of discrete—that is, class—data.
Let us assume that a customer appears on an e-commerce platform. The customer is one of 140 customers grouped in a single cluster. The customers in this cluster behave in more or less similar ways. The customer placed a hammer in the basket. The system immediately finds a second item connected with it: a screwdriver. It turned out that, in this cluster, customers who selected a screwdriver usually also bought a hammer, and if they selected a hammer, they also bought a screwdriver. Of the 140 customers, 38 had made precisely this choice in the past. The system therefore estimates that there is a 24% probability that a customer who selected a hammer on the e-commerce platform will shortly put a screwdriver in the basket. At the same time, the system knows that there is a 76% probability that this customer will not put a screwdriver in the basket.4 The high probability of buying a screwdriver causes the system to display a recommendation. The customer sees several screwdrivers on the purchasing platform’s banner. We do not know whether the customer would have bought a screwdriver without this advertising information. It should be assumed that customers from this cluster probably will not buy one if they do not receive an advertisement for screwdrivers. The applied recommendation, however, drastically increases the probability of purchasing this tool. This is how a system based on two important tools—customer clustering and the creation of so-called recommendation pairs5—operates.
4 In general, nobody examines why this happens; it is pure statistics. It is not important to discover the real reasons for such choices.
5 “Understanding the Apriori Algorithm in Market Basket Analysis,” Ajeeth, Medium: https://medium.com/@ajeethaj99/understanding-the-apriori-algorithm-in-market-basket-analysis-bbeceba4d348
Recommendation pairs
Products sold in pairs or groups of three can be found without relying on customer clusters. Quite simply, if someone buys a dishwashing sponge, they will probably also buy dishwashing liquid. For some reason, however, only some customers do this; some people buy sponges alone. This may be because the sponges are not intended for washing dishes, but, for example, for cleaning car wheel rims.
Or perhaps women buy sponges and dishwashing liquid more often, while men buy sponges without dishwashing liquid more often. This is a typical behavior pattern in which the customer’s membership in a particular group—a cluster—plays an important role. This simple example shows that it is important to use recommendation-pair systems on the basis of clusters. We encounter thousands of individual patterns of this kind. Fortunately, tools find and select them on a mass scale.
Analysis of incomplete purchases within a cluster
As I mentioned earlier, customers assigned to a single cluster are characterized by a particular set of specific behaviors. Consider, for example, a cluster of people who buy roofing-related products. The algorithm took into account the frequency and value of the items purchased and the type of customer (for example, a small business). A sizeable group of customers was classified as small roofing contractors. It is now possible to assess the completeness of the purchases they make. Some customers buy within particular categories offered by the store, such as insulation materials, gutters, roof tiles, bituminous roofing and roofing compounds.
Let us assume that the completeness analysis showed that 27% of customers in this cluster buy bituminous roofing but do not buy roofing compounds. The technology used to install bituminous roofing, however, requires the use of roofing compounds. It follows that a large proportion of customers do not buy roofing compounds in our store, but purchase them elsewhere.6 This is an example of the discovery of a market niche.
Nothing remains but to offer an excellent deal on roofing compounds exclusively to that 27% of customers in the roofing cluster. It is important that the customers change their habits and begin buying roofing compounds in our store.
6 This niche can be detected by finding recommendation pairs, but it may turn out that nobody buys roofing compound in our store because “every roofer knows that it is cheapest to order roofing compound from the manufacturer,” in which case the algorithm will not detect this relationship.
Analysis of disrupted habits
Grouping customers into purchasing clusters makes it easier to detect changes in their habits.
Let us assume that we have a cluster of plumbers. Small pressure-booster pumps were purchased seasonally within this cluster. The seasonality resulted from the fact that, in spring, it often turned out that small pressure-booster pumps installed at holiday plots had to be replaced. At some point, however, it turned out that pressure-booster pumps were not being sold in spring within the plumbers’ cluster.
If we analyzed the level of pressure-booster-pump sales without clusters, we might not detect this anomaly. Pressure-booster pumps may be purchased by different customer groups. We are most concerned with plumbers, however. Thanks to the detected anomaly, it is possible to investigate the situation and propose new and better terms so that plumbers do not buy pressure-booster pumps from competitors. This action will admittedly not be performed automatically by the recommendation system, but the recommendation system enables managers to carry it out.
How to retain customers in the store
So far, my discussion has focused on increasing customers’ purchasing efficiency. Thanks to an accurately displayed advertisement, one customer buying a hammer could additionally buy a screwdriver. A woman buying dishwashing liquid received a recommendation to buy sponges.
Thanks to these and thousands of similarly and effectively used patterns, customers’ total average purchasing efficiency may increase by 30–40%.
They will also experience improved shopping comfort, because a well-matched suggestion is usually welcomed by customers.
An increase in customers’ purchasing efficiency may nevertheless fail to compensate for the decline in sales resulting from some customers leaving. Customers usually leave for some reason; a poorly matched form of treatment or the offerings of other stores usually contribute to this. The reason may also be unrelated to the store’s activities. To avoid departures resulting from the operation of our store, another recommendation system must be created—one focused exclusively on retaining customers.
Here, too, we must perform clustering and divide customers into groups bringing together digital twins. It is then necessary to run survival tests.
A survival test7 is a method developed long ago for medical facilities. Its purpose was the statistical analysis, in relation to time, of patients’ survival time with a specified probability, based on the medical procedures currently used. Different therapies were tested in this way, and on the basis of these tools the probability of patients’ future deaths was calculated. Thus, when a therapy was applied to particular groups of patients, it was predicted that they had, for example, an 80% chance of surviving the first year, a 52% chance of surviving the second year, and a 10% chance of surviving the third year. This tool was quickly adapted by commercial companies. The concept of a patient’s death was replaced with the concept of a customer leaving a store.
This method makes it easy to determine the probability of retaining a customer over the coming years. Various stimuli, such as loyalty cards, discounts and periodic individual reductions, can be added to the algorithm, producing information about the effect that the use of these techniques will have on customer retention. This relatively easy and simple method can drastically reduce the probability of a customer leaving. Interestingly, such a system may also be used for day-to-day customer service. It may serve as a handy adviser to sales representatives speaking with particular customers.
7 “Data Science concept — Time-to-event Analysis (Survival Analysis),” Kiel Dang, Medium: https://medium.com/@kirudang/data-science-concept-time-to-event-analysis-survival-analysis-b2b4a4c1828a
Analysis of purchase abandonment
An important element preceding a customer’s decision to leave an e-commerce platform is the purchase-abandonment indicator. A customer may decide to leave the store because, over a longer period, they could not find an offer suitable for them. Such a decision will not be detected by a survival test either.
To create a purchase-abandonment indicator, the data-collection system must be reconfigured. As I mentioned above, a recommendation system is created on the basis of the sales register. The sales register consists of transactions concluded with the customer at a specified time and for specified goods. If a customer enters an online store and wanders around its various pages, this behavior will not be found in the sales register.
The only way to analyze the customer’s behavior is the history of browsing the online store’s pages. This requires specialized tracking tools, but the undertaking is worthwhile because it effectively indicates the customer’s hesitation and determination to make a purchase.8
8 “Everything You Need to Know About Tracking Consumer Behavior Online,” Illia Lahunou, Verfacto: https://www.verfacto.com/blog/data-driven-marketing/tracking-consumer-behavior-online/
How to build a recommendation system?
The most important thing when building a recommendation system is to specify its objective. Fortunately, companies usually all want the same thing: an increase in customers’ purchasing efficiency. At the same time, e-commerce stores most often want the factors that contribute to customers leaving to be identified. These two objectives can easily be reconciled by following simple principles for building a recommendation system.
The most important thing is the data, which must be complete, reliable and extensive. It is not possible to create an effective recommendation system on the basis of a small quantity of data. A sales-transaction register containing at least several hundred thousand operations should be combined with other databases obtained from external sources.
The next step is the appropriate use of data-clustering methods. When there are a great many variables, the PCA (Principal Component Analysis) algorithm is often used. This is a method for combining many characteristics into three or four consolidated component characteristics. Component characteristics are combined into principal characteristics on the basis of the strength of mutual covariances between the characteristics.9
Let us imagine the information concerning a customer and try to count how many characteristics that customer may have. These may include their residential address, number of transactions per month, number of different product categories, form of payment and dozens of other variables. With such a mountain of data, a customer cannot be assigned effectively to a cluster. For this reason, various tools are used to reduce the number of variables by reducing them or grouping them into larger variables with the PCA algorithm.
Survival analysis, like the tools that increase customers’ purchasing efficiency, is a tool that is simple to create and operate. At the same time, the methodology is regarded as laborious because an enormous number of interactions, threads and external changes arise, and these require experience from the researcher. The ability to maintain the correct task priorities is very important: eliminating ineffective patterns and expanding effective ones. This requires considerable self-discipline and patience.
9 “Segmentation of a population containing very many features. Pca analysis and clustering by k-means metod,” Wojciech Moszczyński, Medium: https://wtm695450085.medium.com/segmentation-of-a-population-containing-very-many-features-928ca8ac5112
First a prototype, then implementation
When creating a recommendation system, a prototype is built first. It is usually written in Python and based on that language’s libraries. The prototype is tested and refined in a development environment. Unfortunately, the prototype is not suitable for use in the sales process. The solution must be implemented in the production environment. This is done by a data engineer, who deploys the program in a cloud environment or in a local environment that supports the sales systems.
Batch and streaming systems
Recommendation systems may be based on historical registers. The prototype is built on a certain history that changes over time.10 The system must therefore be updated. Someone who previously bought yogurts suddenly begins buying kefirs. Someone who once bought a hammer and a screwdriver is unlikely to want to repeat those decisions. We know, however, that after some time a customer who once bought a hammer and a screwdriver will want to buy a drill. These examples show that the recommendation system must be updated periodically. The more frequently the update is performed, the better. In certain specific situations, the best approach is for updating to be performed continuously. It is therefore possible to build a recommendation system based on the continuous delivery of information in the form of a data stream and its processing within the recommendation system.11
10 We are unlikely to be speaking here of trained predictive models, because recommendation systems usually take the form of unsupervised systems—that is, systems not tested on test samples.
11 “Microservices in Python: Kafka and Django,” Manish Sharma, Medium: https://medium.com/@mansha99/microservices-using-django-and-kafka-3776e8592ef3
Summary
Everyone knows that a properly formulated, tailor-made offer increases a customer’s purchasing efficiency and additionally strengthens the customer’s attachment to the store. The problem is that, to adapt an offer to a customer, we must know a great deal about that customer and use tools that will ensure that this knowledge is used effectively.
For such an undertaking to be possible, the data and the process of processing and using them in the analysis must be organized perfectly.
The process is difficult but necessary, because if we do not do it, our competitors will. A recommendation system on an e-commerce platform can be an excellent source of competitive advantage. Without incurring greater expenditure on general advertising or the excessive, broad use of promotions, we achieve greater sales efficiency and therefore higher profits, safeguarding the durability and increasing the value of the e-commerce business being operated.
Wojciech Moszczyński — graduate of the Department of Econometrics and Statistics of Nicolaus Copernicus University in Toruń; specialist in econometrics, finance, data science, and management accounting. He specializes in the optimization of production and logistics processes. He conducts research in the area of the development and application of artificial intelligence. For years he has been engaged in the popularization of machine learning and data science in business environments.
Dodaj komentarz