AI-driven IoT Shopping Assistant Based on LattePanda and ChatGPT

Projects

HARDWARE LIST

1 WIZnet W5300 TOE SHIELD

1 STM32 Nucleo-144 NUCLEO-F439ZI

1 DFRobot GM77 Barcode and QR Code Scanning Module

View Product

1 DFRobot Fermion: 1.51” OLED Transparent Display

View Product

1 DFRobot Tiny (Embedded) Thermal Printer

View Product

1 LattePanda 3 Delta 864

View Product

1 DFRobot 8.9” 1920x1200 IPS Touch Display

View Product

1 Anycubic Kobra 2

1 Keyes 10mm RGB LED Module (140C05)

3 Button (6x6)

1 Xiaomi 20000 mAh 3 Pro Type-C Power Bank

1 USB Buck-Boost Converter Board

1 Breadboard

1 Ethernet Cable

1 Jumper Wires

In light of recent developments in machine learning and artificial intelligence, various brands aimed to implement AI-based solutions to improve the shopping experience and increase their profit margin by tailoring unique suggestions and advertisements depending on the targeted user's profile and preferences. Especially e-commerce companies, platforms, and marketplaces benefited from AI algorithms to autonomously identify the most optimal ad placements for individual customers.

Even though there are some pioneering methods to combine physical store shopping with AI-based e-commerce features, such as Amazon Go cashierless convenience stores, these self-checkout stations are expensive investments for local stores since they require renovating (remodeling) store layouts or paying monthly fees to cloud services. Furthermore, local businesses may need to construct data sets for niche market products since AI algorithms require broad data sets for methods based on computer vision, sensor fusion, and deep learning.

After scrutinizing the recent research papers and the granted patents on e-commerce methods, I noticed there are nearly no appliances focusing on merging physical store shopping with AI-based solutions without altering existing conditions for local businesses. Therefore, I decided to build a budget-friendly and accessible AIoT shopping assistant utilizing the most common and recognizable product identification method — barcodes — to bring AI-based e-commerce features to physical store shopping.

Since I wanted to build a shopping assistant providing a wholesome physical shopping experience with the most prominent e-commerce features, I decided to develop a full-fledged e-commerce web application from scratch in PHP, HTML, JavaScript, CSS, and MySQL. Since I developed this complementing web application to bridge the gap between physical store shopping and e-commerce, it is capable of communicating with the shopping assistant (device), obtaining product information from barcodes, and generating AI-based recommendations consecutively.

The complementing e-commerce web application allows the customer to:

Create a user account identified with the unique 12-digit token
Obtain the generated account verification QR code via an HTML email, including the unique user token
Add or remove product information via barcodes scanned by the shopping assistant
Display the current product list in the cart and the total cart price
Inspect ChatGPT-powered recommendations for each product in the cart
Place an order by making a payment via credit/debit card
Get the payment confirmation QR code via an HTML email
Check purchased items from previous orders

As suitable for local businesses, I decided to install an Apache HTTP Server (XAMPP) on LattePanda 3 Delta, which also has a MariaDB database, to host my web application. Since I focused on creating a low-budget but state-of-the-art AIoT shopping assistant, I decided to employ the OpenAI API to generate ChatGPT-powered recommendations related to the obtained product information by barcodes via the Open Food Facts JSON API. I also employed Brevo's Email API to send HTML emails directly from localhost to inform the customer of the generated unique verification QR codes.

After developing my e-commerce web application successfully, I started to work on building the shopping assistant (device). Since Wi-Fi and Bluetooth transmissions may not be suitable options for a shopping assistant requiring as minimal latency as possible to communicate with the e-commerce application, I decided to utilize WIZnet's W5300 TOE SHIELD providing high-speed bus communication and reliable Ethernet transmission speeds. Since W5300 TOE SHIELD cannot perform without an STM32 Nucleo-144 board, I decided to utilize the NUCLEO-F439ZI development board and program it in MicroPython.

Since I needed the shopping assistant to scan barcodes and QR codes simultaneously to recognize customers, activate/deactivate carts, and send the scanned product barcode with the selected command (add/remove) to the e-commerce web application, I connected a GM77 barcode and QR code scanner to NUCLEO-F439ZI. Also, I connected an SSD1309 transparent display and an RGB LED so as to inform the customer of the device status. Furthermore, I added a tiny (embedded) thermal printer to directly notify the customer when the customer requests a device status report or scans the payment confirmation QR code. In that regard, I wanted to enrich the correlation between physical store shopping and e-commerce.

Lastly, to make the shopping assistant as robust and sturdy as possible while being employed as a self-checkout station, I designed a checkout counter-themed case, a removable top cover with snap-fit joints, and handles for moving the shopping assistant if necessary.

So, this is my project in a nutshell 😃