Prediction in the Age of Big Data: The Science Behind Recommendation Systems


Recommendation systems have become a primary way to discover relevant information from vast amounts of data. Netflix uses algorithms to make movie recommendations. Amazon uses them to tailor product recommendations. And the list goes on.   

But as the volume, variety, and velocity of data continues to increase, customers’ preferences are only going to become more complex and difficult to compute. Familiar approaches may become outdated, even obsolete, as people change the way they make decisions, exhibit preferences or take actions. 

How can data scientists extract meaningful insights and accurately predict customers’ preferences in the age of big data? Below are three simple steps to help ensure success.

Step 1: Define the Problem
A recommendation system cannot do its job properly without the right data to derive insights from. Therefore, prior to designing the system, data scientists must first figure out what problem they are looking to solve. Once that question is determined, data needs to become more apparent. 

For example, an apparel retailer’s problem may be figuring out which products to show customers when they are visiting the e-commerce site. In this case, the retailer will not only need data about the specific customer, their likes and dislikes, etc., but also how they compare to others. That data can be found through a customer’s browsing history, past purchases, and other online or mobile activity.  Data requirements will, of course, change if other variables, such as operational constraints, budgets, and supplier schedules, need to be taken into account before choosing which products to showcase. Put simply: the question of what data to collect depends on what challenge organizations are looking to solve. 

Step 2: Work with the Dataset
Once the right data is gathered, the next step toward building a successful recommendation system is to develop a systematic approach for processing and filtering data at scale. There are three types of filtering: collaborative, user-based, and a hybrid approach. Many companies, such as Netflix, use a hybrid approach. Other successful brands, including Amazon and Spotify, rely on a collaborative filtering model, which takes into account the users browsing history, likes, purchases and rating before providing a recommendation.

For example, a movie-streaming service lets users rate movies they've seen. In order to generate personalized recommendations, the algorithm will first look at how other user ratings overlap and then assign similarity scores. Those similarity scores would then be weighted as ratings and are aggregated to predict an individual’s response to a particular movie. In general, the more information a collaborative-filtering algorithm has about users' preferences, the more accurate its predictions will be. This also requires being equipped with tools and technologies that can handle high dimensional datasets.  

Step 3: Convert Insights to Actions
Another key step toward building a successful recommendation system is having access to a skilled team of data scientists and statisticians who can apply appropriate tools and identify appropriate statistical methods as well as behavioral models to develop data-processing algorithms. They can then design human-friendly interfaces that can collect useful data and subsequently deliver decisions.

In order to know which model or approach to implement, it is important to understand a) how modern use cases work and b) the limitations of certain systems. Professional education programs can fill this need by exposing data scientists to concepts outside their own domain and allowing them to interact with practitioners of other disciplines across industries. 

There are a variety of approaches available to build recommendation systems – but it isn’t always necessary for data scientists to start from scratch when building their own. The vast array of publicly available Recommenders (in all sorts of programming environments) can be used as a foundation. Some examples are: RecommenderLab for R, Graphlab-Create for Python, and Apache Spark's Recommendation module

If there is one thing that’s clear, it is that businesses no longer need to rely on intuition when it comes to choosing which products to spotlight for customers. Recommendation systems can now effectively be used to accurately determine which products customers are most likely to buy. And the success of modern companies like Amazon and Netflix is proof using a recommendation system can have a big impact on business. Prediction is going to get more important as big data and technologies evolve. Data scientists must move quickly to acquire the skills and knowledge necessary to keep pace so they can innovate as well, and improve the performance of their own organizations.   

Devavrat Shah, co-director of the Data Science: Data to Insights course, is a professor in MIT’s Department of Electrical Engineering and Computer Science, director of the SDSC, and a core faculty member at the IDSS. He is also a member of MIT’s Laboratory for Information and Decision Systems (LIDS) and the Operations Research Center (ORC). 

Edited by Ken Briodagh

Related Articles

DoD Selects Semtech, Ripl for 3D Asset Tracking

By: Maurice Nagle    12/3/2020

Semtech Corporation announced the U.S. DoD selected Ripl Networks to deploy an IP mesh, 3D tracking solution for foreign guest and fleet assets at nav…

Read More

Waylay Pushes Industry 4.0 Forward

By: Maurice Nagle    12/1/2020

IoT automation and analytics software company Waylay announced the arrival of the industry's first zero-code provisioning portal creating digital twin…

Read More

More Yield, Less Maintenance, and Greater Value: The Promise of Connected Field Service

By: Arti Loftus    12/1/2020

A new project is underway in Berlin, Germany, at the Werner-von-Siemens Centre designed to address the repair of massive gas turbine blades which are …

Read More

A Big Moment for IoT: 12 Billion Existing Connections Surpass non-IoT Connections for the First Time

By: Arti Loftus    11/30/2020

The Internet of Things (IoT) has seen an astronomical increase in use throughout 2020, more so as the Covid-19 pandemic accelerates the need for inter…

Read More

Requirements for the New Era of IoT Security

By: Special Guest    11/20/2020

IoT security must be powerful enough to protect global enterprises and carrier networks, yet light enough to be embedded on an IC. Only approaches tha…

Read More