Understanding Real-time AI

Recently, I had the opportunity to sit down with Matt Shaffer, VP of Artificial Intelligence and Co-founder at RIOS Intelligent Machines. We discussed real-time AI and what it means for the manufacturing sector. I asked Matt some questions about this interesting and fast-moving area of technology. Here’s what Matt had to say … 

What Is real-time AI?

Imagine you’re dealing with a ton of data streaming in real-time from camera feeds and depth streams. It’s not just static images; it’s a constant flow of information and difficult to manage. But that’s where computer vision uses deep learning steps. It’s a game-changer, but surprisingly, it’s a relatively recent development. Until recently, it was all about analyzing images individually rather than tackling video streams head-on.

Traditionally, processing a video stream using computer vision has meant breaking it down into individual images and crunching data frame by frame. But things are evolving. With advancements in streaming technology and algorithms, we can now apply AI more directly to video feeds. However, there’s still a lag in seamlessly integrating video and image analysis. We’re getting there, but there’s yet to be an off-the-shelf solution.

And that’s where the idea of real-time AI comes into play. We’re living in a world that demands instant responses. We’re talking about rich data streams from cameras with HD resolutions and high frame rates. We need to make sense of this data in a way that’s intuitive for us and our customers. After all, AI can be a powerful user interface, making complex tasks as easy as texting a friend.

Take large language models, for instance. They understand human language better than ever before, making interactions with computers feel more natural. Looking ahead, there’s a massive opportunity to leverage AI to create better user experiences across various interfaces, including text and multi-modal setups.

Now, let’s talk about applications. Using cutting-edge computer vision to improve industrial processes is becoming more common. Whether guiding robot arms in manufacturing or ensuring quality control on the production line, AI can revolutionize how we do things. But keep in mind it’s not just about fancy tech; it’s about involving end-users in the process.

Preferences can be more subjective than you might expect when it comes to tasks like quality control, for example. What’s acceptable to one person might not be passable to another. That’s why designing user interfaces that speak directly to the end-users is crucial. Allowing them to control AI behavior intuitively ensures alignment between what they want the system to do and what it actually does. 

By empowering users to guide automation based on their preferences, we’re not just streamlining processes but revolutionizing how industries think about automation. 

How can real-time AI be used in manufacturing?

When it comes to tackling challenges in industries like quality control, event monitoring, and automation, there are a number of considerations

 In applications like quality control, it can be difficult to develop qualitative metrics, especially when there is a level of subjectivity. Event monitoring is another challenge where capturing and responding to real-time events can prevent bottlenecks, downtime, and injuries, but it is often hard to define programmatically. A human can usually tell you right away what to look for, but translating this into a program requires significant expertise along with trial and error testing.  Finally, there’s the realm where automation meets the real world, and we have mechanical systems or robots taking action. In these systems, we need to figure out where things are, what they are, and how to interact with them. Localizing objects and features in space with the accuracy necessary to perform actions reliably is critical to industrial processes that rely on vision for perception.  

Take the beverage industry, for example, where standardized processes with minimal defects have been tuned to operate with high precision. In this world, localization—knowing exactly where things are so that objects can be moved and reorganized—is the primary concern. On the other hand, if you are in the lumber industry, you have a completely different set of challenges that contribute to process variability, like dealing with very organic materials that are not uniform in size and shape or behave unpredictably during transport. We want our systems to easily handle any object without needing a Ph.D. in Computer Science to make it happen. That’s where our real-time AI platform, Mission Control, comes into play. Mission Control goes beyond just recognizing objects; it enables a manufacturer to control how those objects are handled.

Think about it like this: Imagine being able to glance at a video feed and not only spot the objects you need to work with but also decide how to handle them. Do you want to grab them? Where do you want to put them? It’s all about making the process as intuitive as possible so you can focus on getting stuff done without getting bogged down in the details.

Tell me more about Mission Control

Mission Control is the software interface to our real-time AI platform. Most AI platforms are built for machine learning engineers or data scientists and require a non-trivial amount of oversight. We wanted to build something that abstracted away as much of the repetitive processes used to build and deploy models as possible and instead made them immediately useful. We wanted to build a UI that allows anyone to intuitively instruct a model and see the results on their real-time video streams.  

Now, here’s where it gets interesting. Unlike the robotic systems you typically see in place today, Mission Control goes above and beyond. We’re talking about capabilities that can spot when things get clustered, notice if an object has a blemish, or even detect defects you didn’t even know were there. It’s all about identifying edge cases, especially in unstructured, organic environments, and making sure that your lines don’t go down when something bad happens.

Getting this level of insight isn’t easy with existing robotic platforms. With today’s systems, you can’t just snap your fingers and voilà, it’s done. Instead, you need data, and lots of it. And that data needs to contain specific details that tell the AI what’s in the images. And let’s not forget the whole training process—it’s like boot camp for AI, requiring a ton of know-how in data science and machine learning.

With Mission Control, we’re removing all of the complexity—making it as simple as possible for manufacturers to take control of their AI systems. Ultimately, it’s all about getting accurate metrics based on what the AI sees.

And here’s where it gets really good. These models are invaluable – they can help manufacturers understand where the bottlenecks are in their industrial processes. Imagine slapping a camera on something, observing it like a hawk, and gaining insights into where things could be smoother. This helps you dial in on operational efficiency – which means great profits.

Can this technology be applied in brownfield situations? Or is this just for new installations?

Yes. Mission Control has been designed to work in both situations. Regarding engineering solutions, there are greenfield and brownfield opportunities. Greenfield is typically easier to work with, as you have much more freedom as you design your systems. But brownfield is where things get really interesting.

Brownfield opportunities are like the industry’s seasoned pros—they’ve been around the block for a while, doing things their way. But here’s the kicker: there’s so much room for improvement. That’s where real-time AI can be a real game changer, offering low-cost systems to pinpoint areas ripe for optimization.

How did you and the RIOS team gain the expertise to build Mission Control?

These ideas didn’t just pop out of thin air—they’ve been marinating for a good five years, simmering in conversations with many old and new customers. From walking the factory floors to chatting with folks, we’ve been on a mission to uncover pain points and find solutions that enable the manufacturing industry to improve significantly.

Many companies are itching to enter the world of AI, but let’s face it, it’s not exactly a walk in the park. Building a top-notch team to tackle machine learning is easier said than done. We’re all about making AI accessible to everyone, offering a scalable platform that packs a punch without breaking the bank. It’s like having your own personal team of data scientists at a fraction of the cost.