RSoC: improving drivers and kernel - part 2 (largely io_uring)

By 4lDO2 on

Introduction

This week has initially been mostly minor bug fixes for the redox_syscall and kernel parts. I began the week by trying to get pcid to properly do all of its scheme logic, which it hasn’t previously done (its IPC is currently, only based on passing command line arguments, or pipes). This meant that the kernel could no longer simply process the syscalls immediately (which I managed to do with non-blocking syscalls such as SYS_OPEN and SYS_CLOSE) by invoking the scheme functions directly from the kernel. So for the FilesUpdate opcode, I then tinkered a bit with the built-in event queues in the kernel, by adding a method to register interest of a context that will block on the event, and by allowing non-blocking polls of the event queues. Everything seemed to work fine, until:

The overestimated bug

At the very moment I added context::switch to the io_uring_enter syscall impl, and ran it, the entire system became blocked. While it would make perfect sense for pcid itself to block, since that was the process calling io_uring_enter, the rest of the system just… froze. Since I had experienced a similar freeze when working with interrupts in the drivers (which were deadlock related), I unconciously assumed that it had something to do with faulty event queues or a faulty scheduler, also due to a more recent freeze seen by me and jD91mZM2, where ahcid blocked and the entire system froze.

After about four days of debugging, as I had kept failing to find the bug in the kernel that caused this, I saw that init actually blocked on every process it spawned, and simply assumed that they would eventually fork (or finish, like pcid did previously). So, the solution was four lines of code that forked pcid after its initialization, but before it began to block on scheme requests. That said, I still think the scheduler could get some extra care…

Progress!

After I had solved the bug, I experimented with writing an executor for this new kernel API for async I/O, which can be found here. Since Rust futures are based on the readiness model, where futures are polled once, and when they return Poll::Pending, they tell their reactor to begin using their waker, and to eventually notify them. This is very easy to implement for readiness-based I/O, since they only have to poll, and then add their waker to an epoll loop. However, since io_uring is completion based, there has to be some state management going on, since the io_uring doesn’t remember what a syscall did when it’s complete.

Inspired by withoutboats’s blog post, I came up with an executor that either associated syscalls with a unique tag that’s increment upon every new future, and then let the reactor (which can be integrated or not into the executor) wake up the corresponding future when it receives a completion entry, or stored Weak references as a raw pointer directly in the user data of each submission/completion entry. The former is meant for the scenario when you don’t trust the scheme you are using, while the latter allows the producer to modify pointers, which is obviously unsafe, and should only be allowed when the kernel is the producer.

TODO

Even though I was delayed by a few days due to this extremely simple bug, I did manage to get pcid to properly operate on a scheme socket, solely by using my executor and io_uring! Still, everything isn’t completely async yet within the kernel (FilesUpdate is though!), but it would probably not be that hard to get Redox’s event queues to properly use a dedicated Waker to make the rest of the syscalls asynchronous. I also plan on getting the rest of the Rust async ecosystem to work asynchronously with this, but for me, it’s a higher priority to get this interface to the drivers first.

So last week’s TODO still applies here: getting the pcid <=> xhcid <=> usbscsisd communication to be completely based on io_uring.