I had mentioned in my last post that I will add AI in my project next. Here is an update on this project. The video below has the details. Under that is a summary of the changes done. All of these are running on Nvidia Jetson nano Orin, which is the cheapest in the jetson family. 1K CUDA cores, 8GB shared ram and a powerhouse for such a small footprint.
- Added 4 AI models for inference. All pre-trained, no training involved yet.
- Piper, Whisper for text to speech, speech to text. Although jetson keeps falling back on espeak.
- Yolo for vision processing.
- Tiny llama for language processing. It's an SLM, a small language model rather than an LLM, with about 1.1 Billion parameters. To contrast, chat GPT 5 is rumored to have trillions of parameters.
- My UI does have a training mode for learning from actual mowing sessions but I am hesitant to turn my backyard into a training ground.
- The agentic flow yields about 4 seconds of latency, which is really good for an SLM running on a small machine. It started at about 20 seconds and took quite some effort to tune it down to 4 seconds.
- it does hallucinate when asked complex questions. But I intend to use it only for very simple navigational commands, augmented by deterministic code and safeguards before it executes navigation.
- Some of the challenges that can really slow you down on an edge AI device like this are drivers and port assignment.
- By default Nvidia ships the unit with no OS, and all GPIO ports disabled. They are pre-assigned for specific protocols but still need to be enabled via jetson-io. it will look something like this.
- Most aftermarket peripherals come with drivers that need to be installed separately. Relay board, Arduino, NRF transceiver etc had to be enabled using CH340/341 drivers because they use proprietary Chinese chips.