Recommendations in the AWS Cloud – Amazon Personalize
Amazon Personalize is a managed service which, based on data about user interactions with products, generates personalized recommendations. It integrates with other AWS services – e.g., S3 for data storage, AWS Glue/Athena for data preparation, or DynamoDB and Lambda/ECS for providing results in the application.
Reference scheme of personalization on AWS (Retail Personalization).
Amazon Personalize automates the training of ML models on data delivered by the client and makes the results available through a simple API. Its advantage is that we do not have to implement algorithms ourselves – it is enough to properly load the data (interaction history, product catalog, possibly user profiles), and the service itself will handle training the model and making recommendations available. Amazon Personalize originates from the experience of creating a recommendation system in the Amazon.com store and provides, among others, the following practical examples:
-
Recommended for you (general personalization),
-
Customers who bought X also bought Y (complementary product recommendation),
-
More like this (similar to the viewed item),
-
as well as trend lists or user segmentation.
How does Amazon Personalize work?
Amazon Personalize mainly uses data about user interactions with products – e.g., browsing, adding to cart, purchase, rating, etc. (so-called item interaction data).
This data may come both from historical logs (batch) and be sent in real-time (e.g., from an application via API). Additionally, product metadata (catalog: e.g., categories, prices) and user metadata (e.g., age, gender, loyalty segment) can be provided, which some Personalize algorithms can also use in modeling. Unlike the Azure approach, here we do not have to choose the ML algorithm directly – instead, we choose the so-called recipe or use case, that is, a predefined modeling scheme:
-
Recipe User-Personalization (in the default use case Retail) trains a model that takes into account user interaction sequences (based on RNN neural networks),
-
Recipe Similar-Items creates a product similarity model based on co-occurrence,
-
Recipe Personalized Ranking reranks a list of items for a specific user, etc.
It is also possible to create a so-called custom solution with your own configuration, but usually ready-made recipes tailored to the business scenario are used.
Example of use in an online flour store
Step one: creating data
To use Amazon Personalize, we first define data schemas and create the so-called Dataset Group in the service. Three datasets will be needed.
-
History of user interactions with products (interactions dataset). In the simplest view, this is a table (CSV/JSON) with the columns: user_id, item_id, timestamp and optionally event_type and event_value. For a flour store, events may be, e.g., Purchase, View (viewing a product page), AddToCart, etc. Each row is one user action on a specific product at a given time. It is required to have a sufficient number of such data (AWS recommends >1000 users and >50k interactions for meaningful results). Our sales transaction database will provide part of these interactions (purchases), but it would also be worth including other behaviors (if available) – although in a wholesale food store purchases will be the main signal. These interactions are imported into Personalize – most often by uploading a file to the S3 data bucket and then pointing it to the data importing service.
-
Product catalog (items dataset). This is an optional set containing information about products. For our flour store, we could include attributes such as type of grain, country of origin, whether the product is gluten-free, etc. In Amazon Personalize, the item schema requires a unique ITEM_ID and optionally metadata fields (strings, numbers or categories). This information can help the model better understand relationships between products (e.g., recommend products from the same category) and better handle the so-called cold start (introducing a new product without many interactions).
-
Dataset with static user data (users dataset): also optional. Here we could include, for example, the client’s age, gender, type of business activity, or country of residence. Amazon Personalize allows defining several such user attributes, which can be taken into account by the algorithm (e.g., segmentation). Not all recipes use this data, but for creating user segments they may be useful. In our case, this static data is in the customer database, so we can import it. However, it is worth noting that advanced, non-standard features such as “tendency to click banners” do not immediately have an obvious place in this schema – we could encode them, e.g., as a numeric user attribute (on a defined scale) and import into the Users dataset. Amazon Personalize will probably treat them like any other categorical or numeric feature and potentially use them during model training.
Note: the main emphasis in AWS recipes is placed on user interactions with products. Of course, we can read them from application logs.
W-Moszczynski - System rekomendacji cz2
Step two: choosing the appropriate AWS recipe and starting training
When the data is prepared and loaded into S3 Personalize, we create the so-called Solution and choose an algorithm (recipe).
For personalized general recommendations, we choose, e.g., the recipe aws-user-personalization. Then we initiate model training – Personalize automatically calibrates the model based on the provided data. One can provide hyperparameters, but it is not required; one can operate on default settings. Training takes from several minutes to several hours. After completion, we receive a Solution Version – a trained model, which we can use in two ways:
-
In real time using Campaign. When launching a campaign, we reserve resources in AWS that host our model and allow real-time queries. Then we can call the Personalize GetRecommendations API by providing, e.g., userId (for the “for you” recommendations) or itemId (for similar item recommendations, if we used the SIMS recipe). In response, we get a list of recommended item_id along with probability indicators. In our scenario, we do not want to generate live recommendations at every user entry, because we have assumed weekly updates, therefore a campaign is not necessary – instead we will use batch mode.
-
In batch files (Batch inference job). Amazon Personalize allows running a task that will generate recommendations for all or a selected group of users at once and save the results to a file on S3. This is ideal for applications such as personalized mailings or periodic refreshing of recommendations in the database. We can schedule such a task once a week after updating the model. The service will generate, e.g., a CSV file containing for each user a list of N recommended products. Such a file can then be imported into our store’s database and used to serve recommendations. In this way, we maintain offline mode – for a week we display a fixed set of recommendations to the user, and every week we update them based on the latest transactional data.
AWS has ready-made solutions facilitating the automation of this process. There is, for example, a solution Maintaining Personalized Experiences with ML, which uses AWS Step Functions to orchestrate the entire pipeline: regular import of new data, training of a new version of the model, campaign updates and launching batch jobs. This can be adapted to a weekly schedule. As for integration of results with the application – AWS often shows an architecture where AppSync API (GraphQL) or Lambda fetches recommendations from Personalize and combines them with data from the database, e.g., to return full product information. In our weekly batch approach, however, it is enough to import ready-made results into the store’s database or display them statically from a file.
Summary and differences compared to Azure
Amazon Personalize is a black-box solution – it greatly simplifies implementation but limits our insight into the model. We do not have to write ML code, but, for example, we do not know exactly whether and how strongly our non-standard user attributes from the store were used. We have a limited set of algorithms to choose from.
In the context of customer behavioral features: in Personalize we can import them as additional data, but we do not have a guarantee that the chosen model will use them – most likely it will, but in some non-transparent way.
Azure gives full control – here we hand over the controls to Amazon. On the other hand, Personalize easily integrates with the AWS infrastructure (S3, Lambda, DynamoDB, Pinpoint for emails, etc.) and is designed for scale and updating as new data flows in. For example, one can send events in streaming mode and the AWS model will take them into account almost immediately thanks to incremental mechanisms. In our flour store project, we do not need streaming, but if traffic increased and more dynamic personalization were required, Personalize offers this configuration option.
Personalize charges for model training time and for queries – with 20k clients and weekly updates it should be fairly predictable. One can also use the free tier for the first two months.
Example of effect: after implementing Personalize, on the store’s main page we could display a “Recommended for You” section with a list of, e.g., 5 products, i.e., different types of flour that Personalize considers most interesting for a given logged-in client based on their purchase history and similarity to other clients. On the pages of specific products, we could add a “Customers who bought this product also bought:” section, using a separate SIMS (Similar Items) model. The whole system would be updated every week – e.g., on Monday. Personalize would retrain the model with new data from the entire previous week, and from Tuesday clients would see refreshed recommendations. Such an approach meets the requirement that the system does not operate strictly in real-time, while still using the advanced AWS ML infrastructure.
Wojciech Moszczyński – graduate of the Department of Econometrics and Statistics at Nicolaus Copernicus University in Toruń, specialist in econometrics, finance, data science and management accounting. He specializes in the optimization of production and logistics processes. He conducts research in the field of development and application of artificial intelligence. For years he has been involved in the popularization of machine learning and data science in business environments.
Dodaj komentarz