As mentioned in my previous post, my three-month effort to get to an overlay-based userspace that could run arbitrary commands from the filesystem yielded several things I needed to improve on.
The first thing on my list was transitioning the filesystem from exFAT to FAT32. This served three purposes:
Regarding the third issue, I discovered that Microsoft® has patents on some parts of the exFAT specification, despite the fact that the spec is public. I was not aware of that at the time I built my driver and I don’t want to run into any trouble.
The next system I want to target the OS for is the Agon Light 2, which runs MOS by default. MOS only supports FAT32, so moving to FAT32 has the added benefit that MOS executables and NanoOs executables can live together on the same SD card as I’m playing around with this.
Unfortunately, my first pass at a new driver didn’t do very much toward achieving goal #2. I had to refine and optimize things quite a bit. Writing a filesystem driver for a standard filesystem is not something I’m terribly interested in, so I used AI to help me with that. Its first pass of a file handle structure included an array of 256 bytes to accommodate the maximum length of a filename. This not consumed an unacceptable amount of memory. I had to change it so that the filename was a pointer to dynamic memory that was only large enough to accommodate whatever the actual name of the file was.
The next thing on my list was simplifying and consolidating my HAL implementations. I really have three HALs that I maintain: One for SAMD21-based Arduinos®, one for the Arduino® Nano Every, and one for the POSIX simulator. Unfortunately, there are several boards that use the SAMD21 microcontroller and I was kind of lazy with the way I supported them. When I moved from the Arduino® Nano 33 IoT to the Adafruit® Feather M0, I just copied the file over and changed some of the defines. I then repeated the process for the Seeed® Xiao, the Adafruit® Trinket M0, and the Adafruit® Itsy Bitsy M0. So I wound up with five copies of identical logic with just some defines changed for the GPIOs used.
I factored out all of the common code into one .cpp file for the SAMD21-based, Arduino-compatible boards. I then converted all of the other files to .c files that just pass in some configuration options for the GPIOs and a few other things. So, now I’m down to just that file plus the one for the Arduino® Nano Every that are C++ files in my codebase. Everything else is straight C.
After that, I went back and refactored my Tasks. I mentioned in Preemptive Multitasking - Take 2 that, “I may do some renaming of ‘process’ items to be ‘task’ items.” I did that shortly after that post. In the course of the work I did for the overlays, I realized that had been a mistake. In my renaming, I had labeled the actual Coroutine part of the task a “TaskHandle”. I always disliked that because it seemed pretty vague and meaningless. I realized in the overlay work that a task handle was really a thread. I also realized that the way I had gone about doing that renaming/rework prevented me from ever having multi-threaded processes. The overlay work caused me to extend the role of a “Task” way beyond what the role of a thread is. So, I dutifly renamed my “TaskHandle” to “Thread” and my “Task” back to a “Process” again. That made things much simpler and easier to follow.
Another problem I was having was the fragmentation that was being caused by calling a function in a different overlay. The issue was that calling a function in an overlay required allocating dynamic memory to make copies of the path to the overlay, the name of the overlay, and the function within the overlay. I also had to temporarily allocate space for the full path to the overlay file and then free it again. So, each call to a new overlay function punched four (4) holes in memory. This was causing pretty severe fragmentation. So, I changed the algorithm to account for all the space needed for all the different things up front, do one large allocation, then copy the pieces it needed into different sections. That drastically reduced fragmentation and improved the reliability of the overlay system.
I also realized that my Console process was contributing to memory fragmentation during my overlay work. At the very beginning of my OS work, I had four console buffers that were maintained in the Console process’s state. One buffer was allocated per process so that the console could accumulate input for it. When a process needed to do a print to the console, it checked one out, filled the buffer, then sent it back. I decided at some point that the console only needed to maintain buffers for sending processes input and that it could just allocate buffers from dynamic memory when a process needed one for output. You can probably see where this is going: I was punching a new hole in dynamic memory every time a process did any output. I decided that my original idea was better and extended the number of console buffers back to four. While I was at it, I improved the tracking of the buffers so that I could clearly track which process owned which buffer.
At this point, I had to stop and do some debugging. I kept running into weird crashes after running a few overlay processes. I’ll spare you the details on this one because this turned out to be the most circuitous bug I’ve ever tracked down. I traced it through three libraries and countless memory allocations and deallocations and it took me several days of effort. The issue turned out to be that I had forgotten to initialize the prev pointer in a file handle when opening a file. Because of this, there were some stale references to freed file handles in the open files linked list. That eventually turned into overwriting memory that had been allocated for something else. Once I got that fixed, the system was stable!! I could reliably pipe data between three concurrent commands as many times as I liked!! HOORAY!!!
There did turn out to be another issue with piped commands, though. With the way that NanoOs works to conserve process slots, the last command in the chain is exec’d instead of spawned. Within the scheduler, that was terminating the main thread of the process and starting it fresh with a new command. In a piped situation, that was problematic because what I saw was that messages piped to that process early in the command chain’s life weren’t being processed. The reason was that terminating the thread was destroying the message queue. So, I added an option on terminate to allow for the message queue to be kept. That FINALLY allowed piped data to work end to end.
I was slowly leaking memory, though. After a bit of debugging, I realized that the only thing that could possibly be leaking was the envp variables. I eventually tracked that down to not freeing them properly at the beginning of an exec. Fixed that and finally was able to make use of all of the memory in the system.
I discovered that I was getting a crash when piping too many commands and failing to launch all processes. The issue turned out to be that, when the main process restarted, all of its memory was freed first. Because the last link in the chain had failed, the file descriptor never got assigned to the correct process and the memory was freed but the pointer to the file descriptor wasn’t updated. The fix for that was to close the last file descriptor if the command failed to launch. Once that was done, no more segfaults.
My next goal for NanoOs is to reduce the compiled binary size. One of the ways I’m thinking of doing that is by moving the filesystem to an overlay process. But, the current overlay mechanism uses the filesystem for the overlay lookup. My thought on how to move the filesystem itself into an overlay is to have a second overlay lookup mechanism that’s block based. But the current function uses strings for the path to the overlay directory and the name of the overlay file. That obviously wasn’t going to work. So, I converted the existing lookups to take void* arguments and updated the userspace code to match. I changed the “overlay directory” argument to be an “overlay namespace” argument. For block-based lookups, this will be an integer block device ID cast to a void*. The overlay argument will be the numerical index of the overlay block, also cast to a void*.
The need to look up block devices by numerical ID revealed two big flaws in my HAL model: (1) The HAL did nothing to abstract block devices and (2) the way I had originally envisioned numerical IDs working was flawed.
Let me start with the second issue first. I had originally thought that numerical IDs for devices would run from zero to the last one on the system. I realied in the course of doing this that that’s not a valid assumption to make. I had already kind of hit on this in some preliminary work I did for the UART on the Agon Light 2 where the first UART was not capable of being a console. It occurred to me that what I was really going to have was a maximum number of devices that the platform supported and a bitmask of which ones were actually online. Most of my HAL subsystems were oriented the old way and they were all wrong. So, I spent about a week painstakingly going through one subsystem at a time and changing things over to a bitmask system.
When I finally got all that done, I started looking at having the HAL provide some kind of interface into block devices. One of my tenents for my HAL is that a HAL does not manage state. Anything that manages state belongs in a process. Block devices have state, so the interface to my SD card has always been through a process. So, I didn’t want the HAL to become responsible for directly managing block devices, but it did need to provide some kind of interface. I decided that, in addition to exposing the bitmap, I would expose a function that would send a message to a designated block device process. That seemed like the right level of abstraction to me. That allows for the block device “overlay namespace” to be directly mapped to a block device ID. At that point, I could finally write the block-based lookup function (although I don’t have anything to use it with yet).
One of the other things I did during this effort was to improve how bulk reads and writes were done on the SD card. Initially, my logic only read or wrote one block at a time. Doing multiple blocks meant sending the preamble for each individual command, doing the operation, then ending the command for each block. This is obviously inefficient. I changed it so that my driver process made use of SD card commands to read and write multiple blocks at a time. That made things much faster. In particular, it made running commands with the overlays perform better than I had originally hoped for. I feel like I now have a very scalable system with a decent user experience.
Another thing I’m considering to try and conserve flash space is getting rid of all the strings in the binary. In order to do that, I’ll have to have a logger process that can turn the pointers back into strings in the console, which will consume some RAM. I went through an exercise of reducing the existing RAM usage as much as possible to accommodate another process. I wasn’t sure how much space it would really save, but I decided that I needed to do it regardless since this is a real technique that’s used in the industry. I worked to make a makefile that would create both a build with strings and a build without. (The build with strings is so that I can load and run that directly while I’m debugging.) I was surprised to find that the build without strings was almost 11 KB smaller than the version with strings, so this is definitely worth doing. I guess I’ll have to write about that in a future post.
So, at this point, I have a pretty stable system and some clear paths forward. I’m just itching to start the port to the Agon Light 2 but the eZ80 only has 128 KB of on-board flash and I want to make sure the binary is as small possible before I start that work. So, as always, more stuff to do before I get to the stuff I REALLY want to be doing but that’s just the name of the game. I’ll get there, but one thing at a time.
To be continued…