A first project in machine learning and electronics
Much of this project was made possible by the creators of the Tensorflow ModelZoo models, the wiring diagrams, data formatting, and model setup guides that helped me learn and adapt the material to suit my project.
Brief Overview of the Project:
The object-detecting Nerf gun uses a machine learning model and electronics to pull the trigger automatically when the camera on the gun detects a person, requiring the holder only to reload the gun and aim it. This project consisted mainly of two parts, one being the machine learning model that runs the detection of human faces, as well as the code that was built around the model to incorporate it into the larger project, as well as to configure the data. The other part is the physical electronics made up of an arduino that takes in signals from the model, a breadboard to manage the electrical wiring and power supply, 9 volt power supply, a small motor capable of pulling the trigger of a Nerf gun, and a camera to allow the Nerf gun to use the model in real time. The electrical parts are mounted on the nerf gun such that they do not obstruct the normal functions of the gun.
Model and Code Breakdown:
The machine learning model is the ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8 Tensorflow ModelZoo model that comes somewhat pretrained on a large dataset, and can be further trained on your own data, this was taken from the Tensorflow ModelZoo Github. The model was trained on data I took from the “Labled Faces in the Wild” dataset from the University of Massachusetts Amherst, which contains images of faces of several different backgrounds . I used this dataset to train the model on several characteristic labels I developed with the labelimage software. Before training could begin, the model required the data to be formatted to the right size, and the label boxes from labelimg to be converted to a single csv file. Tfrecords were also required, as well as a labelmap to handle the data while in training, these were generated by scripts I took and modified to fit my purposes. The config of the model then had to be changed to fit the specifications and paths of my image and label data before training.
To run any programs, I was working through Anaconda prompt, using it to install modules and packages in a virtual environment for running my programs with the correct libraries, while some libraries like protobuf required more direct installation. The model was trained over a few hours on the data, not using the GPU of the computer due to having an AMD graphics card and was saved from a checkpoint in the model’s training to create a saved model that can be easily used to run the detection script off of new images. From there, the runFromWebcam script accesses an external webcam to run the detection script every frame, and return a prediction when confidence is high in a certain class being detected. This script calls the arduino script when the conditions set by the program are met for what class should trigger a response, opening a pin and allowing current to flow through a switch and turn the motor, pulling the trigger on the Nerf gun.
Electronics and Hardware Breakdown:
The hardware is very straightforward and simplistic, with tape holding all of the electronics to the Nerf gun, and wires attached haphazardly to several of the components. The arduino that recives a signal from the machine learning model via USB switches a pin from low to high when this occurs. This 5v electrical signal is enough to mostly “open” a MOSFET transistor. The MOSFET switch system is necessary because the motor requires more than the maximum 5v from the arduino. To set this system up, a breadboard with two separate power busses is established, with one side running 5v from the arduino, and the other running 9v from an external power supply, both sharing a common ground connection. When the MOSFET transistor receives the 5v signal from the arduino, it opens, allowing the 9v from the external power supply to run to the motor and make it spin with enough torque to pull tight a string mounted to both sides of the trigger, pulling it and firing the Nerf gun. The entire signal process after the arduino recices a signal from a computer running the mahine learning model is controlled by a script within the arduino. The camera mounted on the Nerf gun is connected directly to computer to be taken as input for assesment by the machine learning model.
Current Complications:
The machine learning model was trained on an unbalanced dataset that contained a relatively small amount of data, however, this could be easily resolved by gathering more data and retraining the model.
Challenges in Development:
The machine learning model was the first part of the project that I tackled, and I did this with very little experience in the field beforehand. Because of this, I decided to use a premade model and follow several tutorials online to fully setup the model. However, this was not straightforward as many tutorials and libraries were highly dependent on version, age, and tensorflow and dependencies in particular had several hardware requirements that my machine did not meet due to having predominantly AMD hardware. Furthermore, in the setup of virtual environments through Anaconda, there would be innumerable conflicts in installing libraries as well as faliures to download and import sucessfully whatsoever with some libraries, namely keras and some other tensorflow dependencies, although conflicts were not limited to these, especially when taking into account versions. These conflicts took up the majority of my time in working on making the model operational, and required an extensive amount of problem solving, web searching, and trial and error to resolve. The formatting of data was also difficult, as I tried to use several different methods of getting the face data organized and labled in the first place. Because of this, my data organization had many errors and required me to redo it to make it usable in the model, meaning new tfrecords, new csv files, and new reformatting was required. Ensuring paths were all correct for training and running the model was also an issue if I made any errors or changes, I would have to check my unorganized files for the correct path. After these issues were settled, the model was able to train smoothly, but slowly because the model was only able to use my CPU, rather than my much more suited but still AMD and incompatible GPU.
In the electrical portion of the project, I also suffered several delays due to lack of experience with electrical systems. Several circut diagrams were taken and used from external sources, and some were mapped for the arduino breadboard which made matters far easier. However, the main complication with this part of the project was making the motor generate enough torque to pull the trigger of the Nerf gun. Initially, I was using a small motor and a gun with a stiff trigger, this proved impractical, as the motor could not generate nearly enough torque to pull the stiff trigger. The next attempted approach was using an incredibly large drill motor that is capable of producing an insane amount of torque, but the current required to make it turnover under the load of a trigger was too great for the arduino to safely handle, as the 20v power supply it required to move the motor could irreparably damage the arduino and the MOSFET. Finally, using the smaller motor, alongside a power supply for a much safer 9v instead of the arduino’s base 5v, the motor was able to pull a trigger that had a lower tension, and so was able to be used for the trigger-pulling application. Lastly, external USB wiring and taping came with some minor challenges, but most things on this side were eventually resolved.