The Role of In-Memory Computing and HTAP in High Performance IoT Architectures

By Special Guest
Nikita Ivanov, CTO and Co-founder, GridGain
December 27, 2016

By 2020, Gartner expects over 20 billion devices to be connected to the Internet of Things (IoT). According to the “Cisco Global Cloud Index: Forecast and Methodology, 2015–2020”, the total amount of data created by devices, driven by IoT, will reach 600 ZB per year by 2020, up from 145 ZB per year in 2015. To deliver on the promise of IoT, successful IoT platforms will need a data architecture that can deliver the performance and scale to deliver real-time insights on vast amounts of data. One of the most promising technologies to address these challenges is in-memory computing-based Hybrid Transactional and Analytical Processing (HTAP).

IoT Use Cases
Consider just a handful of the IoT use cases that demand fast data processing and the resulting real-time insights to drive actions:

  • Manufacturers are using sensors to track production, streamline industrial processes, and monitor inventory in industries ranging from food to logistics
  • In logistics, self-driving vehicles will produce and consume massive amounts of data while determining optimal routes and minimizing fuel usage
  • In transportation, both ride-sharing companies and taxi drivers are using apps to broadcast anonymous information about their travel, becoming part of a broader effort to implement intelligent traffic control
  • Healthcare organizations are implementing both hospital and residential-based patient monitoring, as well as environmental monitoring
  • Connected home products include security devices, thermostats, refrigerators, robots, and garden sensors

In most of these use cases, the analysis of IoT data must occur in real or near-real time to drive appropriate action. If analytics are not processed in time to prevent a patient emergency or food spoilage, then the benefits of IoT can be minimal.

In-Memory Hybrid Transactional/Analytical Processing for IoT
An infrastructure capable of the speed and scale required for IoT can be built on hybrid transactional and analytical processing solutions that use a single data store for transactions and analytics. Until recently, in-memory HTAP was expensive. Users were also typically concerned about the impact slow running queries could have on the ability to process transactions in real-time. Today, however, in-memory HTAP solutions are increasingly affordable thanks to reductions in the cost of memory. In addition, analytical processing, often based on streaming technology, greatly reduced concerns about analytics impacting operations. The simplified architecture of HTAP solutions also offers considerable cost savings potential compared to two side-by-side solutions – one for OLTP and one for OLAP.

There are three current approaches to IoT platforms: proprietary, cloud-based and open source. The distinctions across these solutions mirror the distinctions across other solution categories.

These solutions may be purpose-built and address a specific use case well but may come with high costs and result in vendor lock-in. Examples include GE Predix, the Bosch IoT Suite, or PTC ThingWorx.

These solutions can be cost-effective and easy to implement for certain IoT applications. However, they can have rigid application and device integration interfaces and are unable to provide real-time analytics so are more commonly used for non-real-time IoT applications. Examples include AWS IoT and Azure IoT Suite.

Open Source
These solutions can be very flexible because the source code can be customized. They can also be cost effective and result in no vendor lock-in. Some of these solutions offer the in-memory capabilities necessary for real-time IoT applications. Examples include the SMACK stack (Apache® Kafka™, Mesos™, Akka™, Cassandra™ and Spark™) and Apache® Ignite™.

In-Memory Computing Platforms for IoT
The basic elements required for an in-memory HTAP solution for Internet of Things applications include:

  • An in-memory data grid for distributed data storage and scalability
  • An in-memory compute grid for high performance parallel processing of queries
  • In-memory streaming processing to allow analysis of incoming data in real-time
  • An in-memory SQL grid that provides in-memory database capabilities including support for SQL communications through ODBC/JDBC and may provide geospatial support for location-based IoT use cases

These interconnected modules can form a single, easily managed layer running in memory for the type of speed and scalability needed to deal with the torrent of data flowing from IoT devices. By enabling these core pieces of the IoT architecture to be implemented in an integrated, scalable, high performant, in-memory computing platform, IoT applications can manage the fast data processing required for next-generation IoT solutions.

Edited by Ken Briodagh

Related Articles

Opti is Using Particle to Stop Floods Before They Start

By: Ken Briodagh    3/23/2018

More and more of the world's population is moving into cities, making it necessary for cities to grow, and making it harder for rainwater to absorb in…

Read More

9 Things to Consider When Choosing Cloud Backup Providers

By: Special Guest    3/22/2018

The advent of cloud computing offers businesses many advantages regarding both data security and efficiency.

Read More

You Have to Be Agile with Today's First Responder Legacy

By: Carl Ford    3/21/2018

I recently attended the International Wireless Communications Expo [IWCE] event and was impressed with the commitment to adopt FirstNet, by the vendor…

Read More

IDC Government Insights Names Smart Cities North America Award Finalists

By: Ken Briodagh    3/20/2018

26 cities across 34 different projects and 16 states recognized for success in leveraging emerging technology and innovation to support Smart City ini…

Read More

Digital Transformation and IoT: Making the Connection

By: Special Guest    3/20/2018

Data is an essential commodity. It is so valuable because, when managed and analyzed properly, it provides the insights that businesses need to innova…

Read More