Custom Vision Enables the Future of Recycling

Recently, when our team participated in an internal Microsoft Hackathon for Smart and Sustainable Building Solutions, we realized we had an opportunity to tie in a very exciting project we had been working on with an NFL Football Stadium organization. We decided to focus our efforts on a mutual area of interest - in driving sustainability and recycling efforts within the stadium complex – especially during NFL games.

 

With that in mind, we ran with the concept of Re-Cycling Gamification, and how we could engage the sports fans to help drive re-cycling and sustainability efforts within the stadium complex.   Before we get into our solution, let’s take a moment to understand the difference between Single-Stream vs Multi-Stream Re-cycling.

 

Single-Stream Re-Cycling:

Single stream refers to the process of processing “mixed” recycling materials, including glass, paper, plastic, and metal.  In this process, everything is combined into one stream of recyclables and will require extensive sorting and processing before the materials eventually wind-up in the correct categories. Most consumers today are familiar with this concept – and participate in weekly household re-cycling efforts.

 

For most recycling facilities, this means they are required to pick up re-cycle bins from consumers’ homes or offices.  Today, this is seen as a “preferred” method for consumers - who don’t have the time or the desire to sort by category of material.

 

Once collected, the mixed materials are brought to the material recycling facility, where they are then pre-sorted by facility employees and loaded onto industrial vibration equipment that is designed to further separate the materials by weight and size.

 

Most recycling programs in the United States are now single stream.

 

Between 2005 and 2014, these programs went from covering 29 percent of American communities to 80 percent, according to a survey conducted by the American Forest and Paper Association.

 

The popularity of single-stream re-cycling has become “mainstream” today – especially given that is convenient and a full 66 percent of people surveyed by a Harris Poll last October said that they wouldn’t recycle at all - if it wasn’t easy to do.

• Some experts have credited single stream re-cycling with large increases in the amount of material recycled in the U.S. today.
• Studies have shown that people choose to put more items out on the curb for recycling when they have a single-sort system.
• The growth of single-stream recycling tracks mirrors the growth of recycling overall in this country.

 

But, Re-cycling Efforts Today - They are a Changing…

 

ENTER: Multi-Stream Re-Cycling

Multi-stream recycling is a collection method in which waste generators (humans) are required to separate their recyclables into multiple separate bins at the time of disposal.

 

The specific bin categories may vary depending upon the types of materials that are being targeted for the re-cycling efforts. Typical categories might include Paper, Plastic, Aluminum, Metal, or Compostable Waste.

 

This appeared to be a great idea for several other reasons:

• The stadium concession (meal & beverage) containers are all made of highly recycle-able materials.
  • Example: A recent deal with Coors Light + Ball Corporation to use Infinitely Recyclable Aluminum Cups within the stadium.
• The fans inside the stadium complex represent a true “captive audience” experience:
  • Attendees are gathered together in the stands for a single purpose.
  • Availability of private, secure, wireless network throughout the complex.
    • Potential for Mobile Phone Applications within the Stadium.
  • Potential for links into the Stadium Scoreboard
    • To really amplify the re-cycling competition within the complex!
  • Potential to construct additional “Proof-of-Concept” (POC) scenarios + equipment (i.e. Motion-Activated IP Cameras) to facilitate the recycling and sustainability gamification efforts. 
• The stadium complex presented a unique challenge and scenario for several reasons:
  • There is no better captive audience than highly competitive sports fans!
  • If we could channel the fans’ natural competitive nature…
    • We could educate them on sustainability while providing entertainment.
    • We could leverage their highly capable mobile app skills.
    • We could channel their voracious appetite for consumables with a unique approach to encouraging re-cycling efforts.
      • This could be seen as a grass-roots education and training effort for consumers about single-stream recycling and move towards multi-stream recycling efforts.

The Concept:

The premise for this solution was rather simple:

• Capture images of items being disposed into a recycle bin – real-time.
• Train a Custom Vision Model to recognize and predict the different types of re-cycled materials being thrown into the bin.
• Use those predictions to drive a real-time, Power BI dashboard – that would be available on BOTH fans mobile devices & the Stadium Scoreboard!

Figure 1 – Team Vision for Recycling Gamification

 

The team gathered various training images that were used as the basis for the initial iterations of the custom vision model and were pre-sorted by the following categories – so that the images could be tagged appropriately:

 

The Technical Implementation:

The technical Proof-of-Concept “POC” solution was implemented using the following Microsoft Azure services:

 

 

High-Level Architecture:

The graphic below illustrates the high-level architecture for the solution. This project was constructed during the hackathon and has a working demo to help prove the viability as well.

 

 

Figure 2 – High-Level Architecture.

 

Technical Overview

• The process starts when a motion-activated IP Camera detects an item being thrown into a re-cycling bin.
• Still images of the item are captured and up-loaded (in real-time) to an Azure Storage blob container. This creates a blob created event in Azure.
• The blob created event triggers an Azure Logic App to “wake-up” with the arrival of each new image that is saved to Azure blob storage.
• This triggers a Logic App workflow to process each image:
  1. Retrieve the URL of the uploaded image from blob storage.
  2. Compose a rest API request - using the URL - to perform a real-time Machine Learning prediction on the image.
     1.     The Machine Learning prediction model was previously created using the Custom Vision service.
  3. Process the prediction results:
     1.     If the prediction for a re-cycle category is => 85% THEN
        1. Compose and send an EventHub message.

• A Message is placed on an Azure Event Hub reflecting an updated prediction result.
• An Azure Stream Analytics job then processes the incoming event hub message and outputs it directly to a real-Time Power BI streaming dataset.
• Power BI then displays the Re-cycling dashboard via web and mobile layouts.
  1. With any luck, we will be able to project not the stadium scoreboard to help encourage participation and friendly competition for valuable prizes.        

 

 

Figure 3 – Sample re-cycling dashboard.

Compelling Facts:

As part of the team’s efforts to raise awareness around sustainability and re-cycling efforts, they gathered an interesting set of “Did You Know” facts to help provide “Edutainment” for the fans to help drive sustainability efforts within the stadium complex. 

Below are sample “Did You Know” facts:

Figure 4– Sample “Did You Know” Fact #1.

Figure 5 – Sample “Did You Know” Fact #2.

Figure 6 – Sample “Did You Know” Fact #3.

 

Key Takeaways

• The team joined forces and skillsets to build on the work we’ve done with the customer, and in the process, learned more about how data and analytics can be used to advance environmental sustainability goals such as multi-stream recycling.
• We as individuals each share a common passion around caring for our environment. Participating in this hackathon project accelerated our understanding of developing solutions for sustainability that can scale.
• This project could be described as a “perfect storm” – where AI technology meets passion for sustainability, and the outcomes are endless. As is mostly the case these days, the technology IS available, it just requires those with a purpose and a passion to leverage it in the best ways possible.
• We all have our own lessons to learn about sustainability and recycling, and what we have learned with single-stream re-cycling was just the beginning!
• Multi-Stream re-cycling is the future, and it will require all of us to pause, and focus – even for just a few seconds – before we toss our trash into the bins, because when it comes to recycling, pre-sorting is the key to the future.