教程第3部分：帶有OV7670攝像頭模塊的TinyML

描述

這是我在 TensorFlow 下使用 Google Summer of Code (GSoC) 的第二個項目。互聯(lián)網(wǎng)上沒有合適的文檔來構(gòu)建自定義圖像識別 TinyML 模型，所以我的 GSoC 導師 Paul Ruiz 建議我應該嘗試解決它。下面介紹了如何構(gòu)建圖像識別 TinyML 應用程序。快樂修補！

項目背后的理念：

我想解決變量較少的問題，因為有關(guān)如何使用相機模塊和處理其數(shù)據(jù)的文檔不是很好。我選擇構(gòu)建一個 MNIST TinyML 模型，因為在這種情況下，我不需要擔心訓練數(shù)據(jù)集，它可以讓我專注于項目的基本部分來啟動和運行。但是，既然我已經(jīng)弄清楚了構(gòu)建自定義圖像識別項目的所有部分，我已經(jīng)記錄了如何使用相機模塊收集訓練數(shù)據(jù)集。

博客的主題/基調(diào)？

我想警告你，這個博客可能會有點難以理解。對此有一個正確的解釋：使用基于加速度計的應用程序，只需在串行監(jiān)視器或繪圖儀上打印出一個軸的加速度計值，就可以很容易地進行完整性檢查。相比之下，對圖像識別應用程序進行健全性檢查至少要麻煩 10 倍，因為無法實時可視化檢查一段代碼是否正在執(zhí)行所需的操作。

一些評論

由于單元測試的復雜性，這個博客可能有點難以理解。我想通過讀者的反饋來解決解釋中的任何空白。因此，請在下面評論您對嵌入式系統(tǒng)圖像識別相關(guān)的任何疑問和問題。

TinyML 是否有意義？

我建議您閱讀TinyML 書的作者 Pete Warden 撰寫的這篇精彩文章，以了解為什么在微控制器上運行機器學習模型是有意義的，并且是機器學習的未來。

即使 TinyML 有意義，圖像識別在 TinyML 上有意義嗎？

我們將在這里使用的 OV7670 相機的完整 VGA（640×480 分辨率）輸出對于當前的 TinyML 應用程序來說太大了。uTensor 通過使用 28×28 圖像的 MNIST 運行筆跡檢測。TensorFlow Lite for Microcontrollers 示例中的人員檢測示例使用 96×96，這綽綽有余。即使是最先進的“Big ML”應用程序通常也只使用 320×320 的圖像。總之，在微型微控制器上運行圖像識別應用程序很有意義

本教程簡而言之：

• 集成攝像頭和 LCD 模塊
• 構(gòu)建 MNIST TinyML 模型
• 測試 TinyML 模型
• 結(jié)論

8.集成攝像頭和液晶模組

8.a TFT+OV7670：顯示測試

本小節(jié)的 Github 鏈接

代碼說明：

tft.fillScreen(ST77XX_BLACK);

這行代碼用黑色填充屏幕。

for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
      tft.drawPixel(i,j,ST77XX_GREEN);
      delay(0);
    }
  }
  delay(1000);

這些代碼行繪制了一個 28x28 的綠色塊。

草圖：

#include     // Core graphics library
#include  // Hardware-specific library for ST7735
#include 


  // For the breakout board, you can use any 2 or 3 pins.
  // These pins will also work for the 1.8" TFT shield.
  #define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6

// For 1.44" and 1.8" TFT with ST7735 use:
Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);
#include 

uint16_t pixels[176 *144];
uint16_t color;
uint8_t red, blue, green;

void setup(void) {
  Serial.begin(9600);

  // Use this initializer if using a 1.8" TFT screen:
  tft.initR(INITR_BLACKTAB); 
  delay(1000);// Init ST7735S chip, black tab
  Serial.println(F("Initialized"));
 
  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }
  Serial.println(F("Initialized"));
  


  // large block of text
  tft.fillScreen(ST77XX_BLACK);
  
  
}

void loop() {
  for(int i =0; i<28;i++){
    for(int j =0;j<28;j++){
      tft.drawPixel(i,j,ST77XX_GREEN);
      delay(0);
    }
  }
  delay(1000);
 tft.fillScreen(ST77XX_BLACK);
}

8.b TFT + OV7670：靜態(tài)圖像測試

本小節(jié)的 Github 鏈接。

現(xiàn)在我們已經(jīng)知道如何顯示綠色塊，讓我們從存儲的 HEX 值數(shù)組中顯示圖像。

代碼說明：

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,.....0x95B5,0xB6B5,0xB6B5};

該數(shù)組存儲所有像素的十六進制值。

for(int i =0; i<50;i++){
    for(int j =0;j<50;j++){
       pixel = pixels[176*i +j];
       tft.drawPixel(i,j,pixel);
}
}

這些代碼行通過陣列循環(huán)并在 TFT LCD 顯示器上繪制圖像。

草圖：

#include     // Core graphics library
#include  // Hardware-specific library for ST7735
#include 

  #define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6


Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,.....0x95B5,0xB6B5,0xB6B5};

uint16_t color, pixel;
uint8_t red, blue, green;

void setup() {
  Serial.begin(9600);
  while (!Serial)
    delay(10);
 // Serial.print(F("Hello! ST77xx TFT Test"));
  tft.initR(INITR_BLACKTAB); 
  delay(1000);
  // large block of text
  tft.fillScreen(ST77XX_BLACK);
  
  
}

void loop() {
  for(int i =0; i<50;i++){
    for(int j =0;j<50;j++){
       pixel = pixels[176*i +j];
        red   = ((pixel >> 11) & 0x1f) << 3;
        green = ((pixel >> 5) & 0x3f) << 2; 
       blue  = ((pixel >> 0) & 0x1f) << 3; 
      color = tft.color565(red+50, green+50, blue+50);
      Serial.print(red);
      Serial.print(", ");
      Serial.print(green);
      Serial.print(", ");
      Serial.println(blue); 
      tft.drawPixel(i,j,color);
      delay(0);
    }
  }
  delay(100);
  //tft.fillScreen(ST77XX_BLACK);
}

8.c TFT + OV7670：Liveimagetest

本小節(jié)的 Github 鏈接。

現(xiàn)在我們已經(jīng)知道如何從存儲的 HEX 值數(shù)組中顯示圖像，讓我們擴展它以顯示實時圖像。

代碼說明：

uint16_t pixels[176 * 144];

這行代碼聲明了一個數(shù)組來存儲從相機捕獲的圖像。

Camera.readFrame(pixels);

這行代碼從相機讀取一幀并將其存儲在像素數(shù)組中。

for (int i = 0; i < 112; i++) 
{
   for(int j = 0; j < 112; j++)
   {
      uint16_t pixel = pixels[176*i +j];
      tft.drawPixel(i,j,pixel);
   }
}

這些代碼行遍歷像素陣列并在 TFT LCD 顯示器上繪制圖像。

草圖：

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include 
#include     // Core graphics library
#include  // Hardware-specific library for ST7735
#include 

#define TFT_CS        A7
  #define TFT_RST        7 // Or set to -1 and connect to Arduino RESET pin
  #define TFT_DC         A6


Adafruit_ST7735 tft = Adafruit_ST7735(TFT_CS, TFT_DC, TFT_RST);

uint16_t pixels[176 * 144];
void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();

  tft.initR(INITR_BLACKTAB); 
  delay(1000);
  
  if (!Camera.begin(QCIF, RGB565, 1)) {
    Serial.println("Failed to initialize camera!");
    while (1);
  }

  Serial.println("Camera settings:");
  Serial.print("\twidth = ");
  Serial.println(Camera.width());
  Serial.print("\theight = ");
  Serial.println(Camera.height());
  Serial.print("\tbits per pixel = ");
  Serial.println(Camera.bitsPerPixel());
  Serial.println();

  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
    Serial.println("Reading frame");
    Serial.println();
    Camera.readFrame(pixels);
    tft.fillScreen(ST77XX_BLACK);
    for (int i = 0; i < 112; i++) {
      for(int j = 0; j < 112; j++){
       uint16_t pixel = pixels[176*i +j];
       int red   = ((pixel >> 11) & 0x1f) << 3;
       int green = ((pixel >> 5) & 0x3f) << 2; 
       int blue  = ((pixel >> 0) & 0x1f) << 3; 
      
      Serial.println(red);
      Serial.println(green);
      Serial.println(blue);
      tft.drawPixel(i,j,pixels[176*i+j]);
      }
    }
    delay(2000);
}

9. 構(gòu)建 MNIST TinyML 模型

注意：如果您是 TensorFlow 或 TinyML 的新手，我強烈建議您通讀這篇TinyML 簡介博客，以便在深入了解本節(jié)之前很好地掌握這些概念。

現(xiàn)在我們已經(jīng)讓相機和顯示器工作并集成了，讓我們構(gòu)建機器學習模型來識別數(shù)字。

9.a 探索 Colab 筆記本

本小節(jié)的 Github 鏈接。

代碼說明：

import numpy as np                   # advanced math library
import matplotlib.pyplot as plt      # MATLAB like plotting routines
import random                        # for generating random numbers

from keras.datasets import mnist     # MNIST dataset is included in Keras
from keras.models import Sequential  # Model type to be used

from keras.layers.core import Dense, Dropout, Activation # Types of layers to be used in our model
from keras.utils import np_utils                         # NumPy related tools
import tensorflow as tf

tf.config.run_functions_eagerly(True)

這些代碼行導入必要的庫來構(gòu)建和可視化我們的模型。

(X_train, y_train), (X_test, y_test) = mnist.load_data()
print("X_train shape", X_train.shape)
print("y_train shape", y_train.shape)
print("X_test shape", X_test.shape)
print("y_test shape", y_test.shape)

>>X_train shape (60000, 28, 28)
>>y_train shape (60000,)
>>X_test shape (10000, 28, 28)
>>y_test shape (10000,)

這些代碼行加載 MNIST 測試并將圖像訓練到正確的變量中。

plt.rcParams['figure.figsize'] = (9,9) # Make the figures a bit bigger

for i in range(9):
    plt.subplot(3,3,i+1)
    num = random.randint(0, len(X_train))
    plt.imshow(X_train[num], cmap='gray', interpolation='none')
    plt.title("Class {}".format(y_train[num]))
    
plt.tight_layout()

這些代碼行從 MNIST 數(shù)據(jù)集的訓練數(shù)據(jù)中可視化了九個不同的圖像。

def matprint(mat, fmt="g"):
    col_maxes = [max([len(("{:"+fmt+"}").format(x)) for x in col]) for col in mat.T]
    for x in mat:
        for i, y in enumerate(x):
            print(("{:"+str(col_maxes[i])+fmt+"}").format(y), end=",")
        print("")
     
matprint(X_train[num])

這些代碼行將訓練數(shù)據(jù)中的隨機圖像顯示為值數(shù)組。

from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D, GlobalAveragePooling2D, Flatten
from keras.layers import BatchNormalization

這些代碼行導入必要的層來構(gòu)建機器學習模型。

# Again, do some formatting
# Except we do not flatten each image into a 784-length vector because we want to perform convolutions first

X_train = X_train.reshape(60000, 28, 28, 1) #add an additional dimension to represent the single-channel
X_test = X_test.reshape(10000, 28, 28, 1)

X_train = X_train.astype('float32')         # change integers to 32-bit floating point numbers
X_test = X_test.astype('float32')

#X_train /= 255                              # normalize each value for each pixel for the entire vector for each input
#X_test /= 255

print("Training matrix shape", X_train.shape)
print("Testing matrix shape", X_test.shape)

這些代碼行對訓練和測試數(shù)據(jù)進行預處理以使其正常工作，例如：標準化、Float64 到 Float32 的轉(zhuǎn)換和重塑。

# one-hot format classes

nb_classes = 10 # number of unique digits

Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

這些代碼行 one-hot 對訓練和測試圖像的標簽進行編碼。

from keras.layers.convolutional import DepthwiseConv2D
from keras.backend import relu
from keras.activations import softmax
model = Sequential()                                 # Linear stacking of layers

model.add(DepthwiseConv2D((3,3),input_shape=(28,28,1)))
# Convolution Layer 1
model.add(Conv2D(2, (3, 3)))      # 2 different 3x3 kernels -- so 2 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
convLayer1 = Activation('relu')                     # activation
model.add(convLayer1)

# Convolution Layer 2
model.add(Conv2D(2, (3, 3)))                        # 2 different 3x3 kernels -- so 2 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
model.add(Activation('relu'))                        # activation
convLayer2 = MaxPooling2D(pool_size=(2,2))          # Pool the max values over a 2x2 kernel
model.add(convLayer2)

# Convolution Layer 3
model.add(Conv2D(4,(3, 3)))                         # 4 different 3x3 kernels -- so 4 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
convLayer3 = Activation('relu')                     # activation
model.add(convLayer3)

# Convolution Layer 4
model.add(Conv2D(4, (3, 3)))                        # 4 different 3x3 kernels -- so 64 feature maps
model.add(BatchNormalization(axis=-1))               # normalize each feature map before activation
model.add(Activation('relu'))                        # activation
convLayer4 = MaxPooling2D(pool_size=(2,2))          # Pool the max values over a 2x2 kernel
model.add(convLayer4)

model.add(Flatten())                                 
model.add(Dense(5,activation = relu))                           
model.add(Dense(10, activation = softmax))

這些代碼行定義了機器學習模型中的實際層。

model.summary()

這行代碼向用戶顯示有關(guān)模型架構(gòu)的信息。

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

這行代碼定義了機器學習模型在訓練時要使用的損失、優(yōu)化器和其他指標。

history = model.fit(X_train,Y_train, steps_per_epoch=60000//128, epochs=3, verbose=1, 
                    validation_data=(X_test,Y_test))

這行代碼訓練機器學習模型。

score = model.evaluate(X_test, Y_test)
print('Test score:', score[0])
print('Test accuracy:', score[1])

這些代碼行評估機器學習模型并將準確性和分數(shù)打印給用戶。

!apt-get -qq install xxd
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Save the model to disk
open("gesture_model.tflite", "wb").write(tflite_model)
  
import os
basic_model_size = os.path.getsize("gesture_model.tflite")
print("Model is %d bytes" % basic_model_size)

這些代碼行將 TensorFlow 模型轉(zhuǎn)換為 TensorFlow Lite 模型。

!echo "const unsigned char model[] = {" > /content/model.h
!cat gesture_model.tflite | xxd -i      >> /content/model.h
!echo "};"                              >> /content/model.h

import os
model_h_size = os.path.getsize("model.h")
print(f"Header file, model.h, is {model_h_size:,} bytes.")
print("\nOpen the side panel (refresh if needed). Double click model.h to download the file.")

這些代碼行將 TensorFlow Lite 模型轉(zhuǎn)換為 C 文件以供微控制器使用。

現(xiàn)在我們已經(jīng)構(gòu)建并訓練了我們的模型，我們現(xiàn)在需要弄清楚如何將數(shù)據(jù)從相機發(fā)送到 TinyML 模型。

相機的輸出尺寸為 176x144，模型的輸入尺寸為 28x28。我們可以嘗試兩種方法：

• 1. 從相機的輸出中裁剪出 28x28 的圖像。
• 2. 從相機的輸出中裁剪出 128x128 的圖像并將其重新整形為 28x28

我們將在以下部分嘗試這兩種方法。

9.b OV7670：作物測試

本小節(jié)的 Github 鏈接。

代碼說明：

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,....,0xB5AD,0x95B5,0xB6B5,0xB6B5};

這行代碼將 HEX 值存儲在大小為 176X144 的數(shù)組中。

for(int a = 0; a< 112; a++)
   {
    for(int b = 0; b< 112; b++)
    {
      Serial.print( pixels[176*a +b]);
      Serial.print(", ");
    }
    Serial.println("");
   }

這些代碼行遍歷數(shù)組并打印出圖像的前 28x28 像素。

草圖：

/*
  OV767X - Camera Test Pattern

  This sketch waits for the letter 'c' on the Serial Monitor,
  it then reads a frame from the OmniVision OV7670 camera and 
  prints the data to the Serial Monitor as a hex string.

  The website https://rawpixels.net - can be used the visualize the data:
    width: 176
    height: 144
    RGB565
    Little Endian

  Circuit:
    - Arduino Nano 33 BLE board
    - OV7670 camera module:
      - 3.3 connected to 3.3
      - GND connected GND
      - SIOC connected to A5
      - SIOD connected to A4
      - VSYNC connected to 8
      - HREF connected to A1
      - PCLK connected to A0
      - XCLK connected to 9
      - D7 connected to 4
      - D6 connected to 6
      - D5 connected to 5
      - D4 connected to 3
      - D3 connected to 2
      - D2 connected to 0 / RX
      - D1 connected to 1 / TX
      - D0 connected to 10

  This example code is in the public domain.
*/

#include <Arduino_OV767X.h>

uint16_t pixels[176 * 144]= {0x0D2A,0xED29,0xED29,0xED29,....,0xB5AD,0x95B5,0xB6B5,0xB6B5};
int arr1[28*28];
void setup() {
  Serial.begin(9600);
  while (!Serial);

  Serial.println("OV767X Camera Capture");
  Serial.println();
  Serial.println("Send the 'c' character to read a frame ...");
  Serial.println();
}

void loop() {
  if (Serial.read() == 'c') {
    
    for(int a =0; a< 112; a++)
   {
    for(int b =0; b< 112; b++)
    {
      Serial.print( pixels[176*a +b]);
      Serial.print(", ");
    }
    Serial.println("");
   }
   Serial.println("");
  }
}

9.c OV7670：重塑測試

本小節(jié)的 Github 鏈接。

代碼說明：

for(int i=0; i < 28; i++){
    for(int j=0; j< 28; j++){
      int sum =0;

     for(int k =0; k<4;k++){
      for(int l =0; l<4; l++){
        sum += arr[4*(112*i+j) + 112 * k + l];
      }
     }
      sum = sum /16;
      arr1[i*28+j] = sum;
      Serial.print(sum);
     Serial.print(", ");
    }
    Serial.println("");
  }

Serial.println("");

這些代碼行使用 4x4 池化內(nèi)核，步長為 1，遍歷 112x112 2D 數(shù)組以輸出 28x28 圖像。

草圖：

#include "num.h"
float arr1[28 * 28];
int filterWidth = 4;
int filterheight = 4;
void setup() {
  Serial.begin(9600);
  
  
}

void loop() {
  // put your main code here, to run repeatedly:
  for(int i=0; i < 28; i++){
    for(int j=0; j< 28; j++){
      int sum =0;

     for(int k =0; k<4;k++){
      for(int l =0; l<4; l++){
        sum += arr[4*(112*i+j) + 112 * k + l];
      }
     }
      sum = sum /16;
      arr1[i*28+j] = sum;
      Serial.print(sum);
     Serial.print(", ");
    }
    Serial.println("");
  }

Serial.println("");
 
}

9.d 探索 reshape.ipynb Colab 筆記本

本小節(jié)的 Github 鏈接。

代碼說明：

from skimage.transform import resize
t = number28.reshape(28,28)
print(t.shape)
number112 = resize(t, (112, 112))
print(number112.dtype)
"""
for i in range(0,112):
  for j in range(0,112):
    if number112[i][j] < 10e-20:
      number112[i][j] = 0
      /
"""
for i in range(0,112):
  for j in range(0,112):
      number112[i][j] = number112[i][j] * 10e+19
      if number112[i][j] < 10:
       number112[i][j] = 0
np.amax(number112)
number112 = number112 /12
np.amax(number112)
plt.imshow(number112, cmap='gray', interpolation='none')
plt.imshow(t, cmap='gray', interpolation='none')

這些代碼行將 28X28 MNIST 圖像放大為 112X112 圖像。

def matprint(mat, fmt="g"):
    col_maxes = [max([len(("{:"+fmt+"}").format(x)) for x in col]) for col in mat.T]
    for x in mat:
        for i, y in enumerate(x):
            print(("{:"+str(col_maxes[i])+fmt+"}").format(y), end=",")
        print("")
matprint(number112)

這些代碼行打印放大的 112X112 MNIST 圖像。

number28new = number28new.reshape(28,28)
for i in range(0,28):
  for j in range(0,28):
      if number28new[i][j] < 35:
       number28new[i][j] = 0
plt.imshow(number28new, cmap='gray', interpolation='none')

這些代碼行打印重新整形的 28x28 MNIST 圖像。

10. 測試 TinyML 模型

現(xiàn)在我們已經(jīng)構(gòu)建了一個 TinyML 模型并測試了將數(shù)據(jù)輸入模型的兩種方法，是時候?qū)?TinyML 模型集成到主應用程序中了。

10.a TinyML 模型：輸入測試

本小節(jié)的 Github 鏈接。

我們?nèi)绾伟l(fā)送輸入數(shù)據(jù)？

對于基于時間序列的模型，很明顯發(fā)送輸入數(shù)據(jù)是因為輸入張量是一維數(shù)組。

//1D input tensor
tflInputTensor->data.f[i * 3 + 0] = (ax + 8.0) / 16.0;
tflInputTensor->data.f[i * 3 + 1] = (ay + 8.0) / 16.0;
tflInputTensor->data.f[i * 3 + 2] = (az + 8.0) / 16.0;

我不知道如何將數(shù)據(jù)發(fā)送到這個特定的 TinyML 模型，所以我設(shè)計了一個測試。

我將 MNIST 訓練數(shù)據(jù)的數(shù)組值存儲到 num.h 文件中。

數(shù)小時：

float num[784] = {0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 49,143,223,196,149, 73,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,126,228,252,257,252,248,242,193, 67,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,176,247,254,213,156,149,175,236,256,204, 53,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,119,246,248,156,  0,  0,  0,  0, 69,216,259,221, 50,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,166,246,160,  0,  0,  0,  0,  0,  0,107,225,259,177,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,115,229,234, 86,  0,  0,  0,  0,  0,  0,  0,142,252,209,0,0,0,0,0,0,
0,0,0,0,0,0,107,223,230,214,237,192, 50,  0,  0,  0,  0,  0,  0,124,245,186,0,0,0,0,0,0,
0,0,0,0,0,0,201,251,147, 44, 95,154,127,  0,  0,  0,  0,  0,116,224,235, 91,0,0,0,0,0,0,
0,0,0,0,0,0,192,254,178, 89,  0,  0,  0,  0,  0,  0,  0, 80,224,242,149,  0,0,0,0,0,0,0,
0,0,0,0,0,0, 67,227,256,244,190, 94,  0,  0,  0,  0, 82,218,248,163,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0, 91,210,245,244,227,184, 90,  0, 84,219,256,188, 38,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0, 76,114,127,201,234,231,244,263,218, 74,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 47,205,266,273,250, 92,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 44,227,264,260,253,145,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,173,251,208,158,218,239,163,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,120,245,236, 92,  0, 52,201,227, 98,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,184,256,192,  0,  0,  0, 65,205,213, 40,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,192,250,146,  0,  0,  0, 70,206,225, 42,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,173,247,232,186,181,199,240,249,178,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0, 39,140,201,226,230,232,233,184, 65,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0};

模型.h：

//MPU6050_model.ino
#include 
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
#include "model.h"
#include "num.h"


const tflite::Model*  tflModel = nullptr; 
tflite::ErrorReporter*  tflErrorReporter = nullptr; 
TfLiteTensor* tflInputTensor = nullptr;  
TfLiteTensor* tflOutputTensor = nullptr; 
tflite::MicroInterpreter* tflInterpreter = nullptr; 

constexpr int tensorArenaSize = 140 * 1024; 
uint8_t tensorArena[tensorArenaSize];
float out[10];

void setup() {

  Serial.begin(115200);
  while (!Serial)
    delay(10);
     
  
  static tflite::MicroErrorReporter micro_error_reporter; 
  tflErrorReporter = µ_error_reporter;

   tflModel = tflite::GetModel(model);
   if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
   TF_LITE_REPORT_ERROR(tflErrorReporter,
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        tflModel->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  static tflite::MicroMutableOpResolver<6> micro_op_resolver;
  micro_op_resolver.AddMaxPool2D();
  micro_op_resolver.AddConv2D();
  micro_op_resolver.AddDepthwiseConv2D();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();

  static tflite::MicroInterpreter static_interpreter(tflModel, micro_op_resolver, tensorArena, tensorArenaSize, tflErrorReporter);
  tflInterpreter = &static_interpreter;

  TfLiteStatus allocate_status = tflInterpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    TF_LITE_REPORT_ERROR(tflErrorReporter, "AllocateTensors() failed");
    return;
  }
  tflInputTensor = tflInterpreter->input(0);
  

}

void test_func(){
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j] / 255;
    } 
  }
   for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
 Serial.println("");
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
  Serial.println("");
  for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]-num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
  */
  
}
void loop() {
  
}

代碼說明：輸入張量：

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j] / 255;
    } 
  }

為了測試輸入張量，我使用 2D 循環(huán)發(fā)送輸入數(shù)據(jù)。

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
Serial.println("");

然后，我打印出存儲在 TinyML 模型的輸入張量中的值。

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }
Serial.println("");

然后我打印出存儲在循環(huán)中的實際值。

for(int i = 0; i < 28; i++){
  for(int j =0; j < 28; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]-num[28*i+j]);
      Serial.print(", ");
    } 
      Serial.println("");
  }

最后，我打印出存儲在輸入張量中的值與存儲在數(shù)組中的值之間的差異。如果所有差異都為零，則輸入已按照我們想要的方式正確存儲在輸入張量中。

10.b TinyML 模型：模型測試

本小節(jié)的 Github 鏈接。

現(xiàn)在我們已經(jīng)弄清楚了如何發(fā)送輸入數(shù)據(jù)，是時候測試模型了。這段代碼與上面的代碼相同，只是調(diào)用了推理并打印了輸出張量。

num.h 文件：

float num[784] = {0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 49,143,223,196,149, 73,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,126,228,252,257,252,248,242,193, 67,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,176,247,254,213,156,149,175,236,256,204, 53,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,119,246,248,156,  0,  0,  0,  0, 69,216,259,221, 50,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,166,246,160,  0,  0,  0,  0,  0,  0,107,225,259,177,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,115,229,234, 86,  0,  0,  0,  0,  0,  0,  0,142,252,209,0,0,0,0,0,0,
0,0,0,0,0,0,107,223,230,214,237,192, 50,  0,  0,  0,  0,  0,  0,124,245,186,0,0,0,0,0,0,
0,0,0,0,0,0,201,251,147, 44, 95,154,127,  0,  0,  0,  0,  0,116,224,235, 91,0,0,0,0,0,0,
0,0,0,0,0,0,192,254,178, 89,  0,  0,  0,  0,  0,  0,  0, 80,224,242,149,  0,0,0,0,0,0,0,
0,0,0,0,0,0, 67,227,256,244,190, 94,  0,  0,  0,  0, 82,218,248,163,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0, 91,210,245,244,227,184, 90,  0, 84,219,256,188, 38,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0, 76,114,127,201,234,231,244,263,218, 74,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 47,205,266,273,250, 92,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0, 44,227,264,260,253,145,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,173,251,208,158,218,239,163,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,120,245,236, 92,  0, 52,201,227, 98,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,184,256,192,  0,  0,  0, 65,205,213, 40,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,192,250,146,  0,  0,  0, 70,206,225, 42,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,173,247,232,186,181,199,240,249,178,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0, 39,140,201,226,230,232,233,184, 65,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0,
0,0,0,0,0,0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,0,0,0,0,0,0

草圖：

//MPU6050_model.ino
#include 
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h"
#include "model.h"
#include "num.h"


const tflite::Model*  tflModel = nullptr; 
tflite::ErrorReporter*  tflErrorReporter = nullptr; 
TfLiteTensor* tflInputTensor = nullptr;  
TfLiteTensor* tflOutputTensor = nullptr; 
tflite::MicroInterpreter* tflInterpreter = nullptr; 

constexpr int tensorArenaSize = 140 * 1024; 
uint8_t tensorArena[tensorArenaSize];
float out[10];

void setup() {

  Serial.begin(115200);
  while (!Serial)
    delay(10);
     
  
  static tflite::MicroErrorReporter micro_error_reporter; 
  tflErrorReporter = µ_error_reporter;

   tflModel = tflite::GetModel(model);
   if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
   TF_LITE_REPORT_ERROR(tflErrorReporter,
        "Model provided is schema version %d not equal "
        "to supported version %d.",
        tflModel->version(), TFLITE_SCHEMA_VERSION);
    return;
  }

  static tflite::MicroMutableOpResolver<6> micro_op_resolver;
  micro_op_resolver.AddMaxPool2D();
  micro_op_resolver.AddConv2D();
  micro_op_resolver.AddDepthwiseConv2D();
  micro_op_resolver.AddFullyConnected();
  micro_op_resolver.AddReshape();
  micro_op_resolver.AddSoftmax();

  static tflite::MicroInterpreter static_interpreter(tflModel, micro_op_resolver, tensorArena, tensorArenaSize, tflErrorReporter);
  tflInterpreter = &static_interpreter;

  TfLiteStatus allocate_status = tflInterpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    TF_LITE_REPORT_ERROR(tflErrorReporter, "AllocateTensors() failed");
    return;
  }
  tflInputTensor = tflInterpreter->input(0);
  

}


void loop() {
  
    
   for(int i = 0; i < 28; i++){
    for(int j =0; j < 28; j++){
      tflInterpreter->input(0)->data.f[28*i+j] = num[28*i+j]/255.0;
    } 
  }
  /*
   for(int i = 0; i < 1; i++){
    for(int j =0; j < 1; j++){
      Serial.print(tflInterpreter->input(0)->data.f[28*i+j]);
      Serial.print(", ");
    } 

  }
  */
  
  TfLiteStatus invokeStatus = tflInterpreter->Invoke();
  out[0] = tflInterpreter->output(0)->data.f[0];
  out[1] = tflInterpreter->output(0)->data.f[1];
  out[2] = tflInterpreter->output(0)->data.f[2];
  out[3] = tflInterpreter->output(0)->data.f[3];
  out[4] = tflInterpreter->output(0)->data.f[4];
  out[5] = tflInterpreter->output(0)->data.f[5];
  out[6] = tflInterpreter->output(0)->data.f[6];
  out[7] = tflInterpreter->output(0)->data.f[7];
  out[8] = tflInterpreter->output(0)->data.f[8];
  out[9] = tflInterpreter->output(0)->data.f[9];
  
  float maxVal = out[0];
  int maxIndex = 0;
  for(int k =0; k < 10;k++){
    if (out[k] > maxVal) {
         maxVal = out[k];
         maxIndex = k;
      } 
  }
  Serial.print("Number ");
  Serial.print(maxIndex);
  Serial.println(" detected");
  Serial.print("Confidence: ");
  Serial.println(maxVal);
  
  Serial.print(out[0]);
  Serial.print(",");
  Serial.print(out[1]);
  Serial.print(",");
  Serial.print(out[2]);
  Serial.print(",");
  Serial.print(out[3]);
  Serial.print(",");
  Serial.print(out[4]);
  Serial.print(",");
  Serial.print(out[5]);
  Serial.print(",");
  Serial.print(out[6]);
  Serial.print(",");
  Serial.print(out[7]);
  Serial.print(",");
  Serial.print(out[8]);
  Serial.print(",");
  Serial.println(out[9]);
  
}

代碼說明：

TfLiteStatus invokeStatus = tflInterpreter->Invoke();

這些代碼行對存儲在輸入張量中的數(shù)據(jù)調(diào)用推理。

out[0] = tflInterpreter->output(0)->data.f[0];
  out[1] = tflInterpreter->output(0)->data.f[1];
  out[2] = tflInterpreter->output(0)->data.f[2];
  out[3] = tflInterpreter->output(0)->data.f[3];
  out[4] = tflInterpreter->output(0)->data.f[4];
  out[5] = tflInterpreter->output(0)->data.f[5];
  out[6] = tflInterpreter->output(0)->data.f[6];
  out[7] = tflInterpreter->output(0)->data.f[7];
  out[8] = tflInterpreter->output(0)->data.f[8];
  out[9] = tflInterpreter->output(0)->data.f[9];

這些代碼行訪問輸出張量并將它們存儲在一個數(shù)組中以供以后使用。

float maxVal = out[0];
int maxIndex = 0;
for(int k =0; k < 10;k++){
  if (out[k] > maxVal) {
      maxVal = out[k];
      maxIndex = k;
    } 
  }
Serial.print("Number ");
Serial.print(maxIndex);
Serial.println(" detected");
Serial.print("Confidence: ");
Serial.println(maxVal);

這幾行代碼打印出輸出值最高的類的類索引和置信度值。

Serial.print(out[0]);
  Serial.print(",");
  Serial.print(out[1]);
  Serial.print(",");
  Serial.print(out[2]);
  Serial.print(",");
  Serial.print(out[3]);
  Serial.print(",");
  Serial.print(out[4]);
  Serial.print(",");
  Serial.print(out[5]);
  Serial.print(",");
  Serial.print(out[6]);
  Serial.print(",");
  Serial.print(out[7]);
  Serial.print(",");
  Serial.print(out[8]);
  Serial.print(",");
  Serial.println(out[9]);

這些代碼行打印出每個單獨類的置信度值。

10.c MNIST：測試圖像

本小節(jié)的 Github 鏈接。

您可以使用這些轉(zhuǎn)換為 NumPy 數(shù)組的圖像來測試您的 MNIST TinyML 模型。

結(jié)論

我感謝我的 GSoC 導師 Paul Ruiz 在整個項目中對我的指導！

?