Hi everyone, I'm Rob. I think most of you know me, or at least know of my from longwinding rambling and/or annoying opinions on the bug tracker, but just in case...
I'm based in Melbourne, Australia, which is why I'm barely awake while you're just getting ready for your after-lunch nap _yesterday_.
I live and work on the traditional lands of the Boonwurrung people.
I've worked full-time on OpenZFS since 2022, and at TrueNAS since 2026. For the 30+ years before that, I've done all sorts, up and down the stack, so I've seen a bit of everything, which is useful!
So we love to talk about "userspace ZFS", and I get asked about it almost every week in the user calls (which you should attend, btw). And I always say, "well everything I do is working towards it, sorta".
Because, I'm not interested in userspace ZFS _as such_.
I'm far more interested in working towards an OpenZFS that can used on or in or with anything that has a need to store things.
To me "userspace" is mostly just a placeholder for "not in the kernel".
Or at least, outside of these two kernels.
Or maybe outside of these kernels or operating systems, that already have a ZFS of some sort, either downstream or adjacent. Though bringing them all into the fold would be nice.
There's also a cottage industry of other operating systems that would be interesting to put ZFS on.
But still, these are all kind of the same: ZFS as the filesystem for regular applications.
Incidentally, that's where I put FUSE. It's not uninteresting, and there are some useful things it could enable, but ultimately it's more of the same.
Because there's a lot more storage out there that is not especially interested in the OS-supplied filesystem, beyond the fact that it's there by default.
The "big three" SQL databases all have interesting and rich backend storage APIs. What would it look like for ZFS to be connected directly to those, rather than as we do now, with careful tuning required to minimise the filesystem overheads?
There's other kinds of databases around. Lucene stores search indexes, basically massive tables of terms -> documents.
Prometheus is a time-series database for service metrics & monitoring, storing, querying and performing computations on millions of raw numbers - think of it like a native store for spreadsheets.
Cyrus is an IMAP mail server. I ran it for millions of users for over a decade. Surprising fact about IMAP: its data and access control model rival the POSIX filesystem for richness and complexity. Cyrus in particular works very hard to get the filesystem out of the way, and in doing so has flushed out serious bugs in Linux, ext4 and OpenZFS.
What about a web server? What if your webroot was a dataset, and your "publish" was to just push a replication stream into it?
Ok, I said I wasn't interested in "more of the same", but these are filesystem options for VMs. If you're running a VM host but don't actually need to see the host filesystem there, why can't you just run a virtio-fs "server" on the host, with ZFS inside it, talking to the raw disk. Why route it all through the host filesystem and kernel when actually we just want to expose raw memory regions into the VM space?
Not unrelated, why not ZFS in the earliest parts of computer startup?
And then there's our classic "remote storage" options, and the newcomer. All of these have high-quality application servers with pluggable backend storage APIs. Why can't we just spin up a VM or container, give it the underlying block devices, run a conventional application server and hand those things out to the network with all the power of ZFS under there.
And then all the tooling we've talked about over the years: backup receivers, recovery tools, offline scrub/repair, etc.
Now to be clear, I'm not saying any of these are good ideas or natural fits for ZFS. But I don't think we've explored the space nearly enough, and I think a big part of that is that the OpenZFS codebase is kinda difficult to work with in a bunch of ways.
And hey, it doesn't hurt to dream a little. There's a lot of things to store, and we have some very good storage tech. Maybe it shouldn't matter how you access it.
So here's an idealised or aspirational diagram of ZFS. It has three places where it connects to its host environment. But if you've spent much time in the code, you'll know that there's a bunch of special cases that don't quite fit neatly into one of these boxes, and even the ones that do have an obvious home often have weird bits entangled with something else.
XXX second slide, same boxes but pulled apart with harder lines between
The goal that I've set for myself is to clearly define these lines, by whatever means. Some of that is just moving things around in the source tree a bit, or removing unused code. Some of it fixing existing warts, or finishing wiring a concept all the way through the tree. For some its requiring developing entirely new subsystems.
XXX third slide, no outer boxes at all, just the "portable core"
Because this is what I actually want. I want there to be clearly defined "thing" that is the "core" of ZFS, that can be connected up to any storage system that has these three concepts - data, control, and services, so the developers of that system can present it to the users in the way that makes most sense for them
I've been working on this in some way for a few years, but this year I really focused on the goal and started working on the things we'd need. No time for everything, but here's a few of the more interesting ones.
So of course, if I'm doing things, how do I know I'm doing them right?
So you know ZTS. Big ball of shell scripts that poke at ZFS in various ways to make sure it does the right thing.
For many kinds of change, this is enough, but for many other types of change, its actually quite complicated to exercise a change to something very core to ZFS, because its usually not exposed directly; instead, you have to take particular actions that you know will tickle the internals in particular way to produce some effect, and then look for the effect.
This is integration testing, or functional testing, or black-box testing. It's very nice, and serves us well! But it can't test the fine details of the internals.
Also, kind of not helpful if we want to make sure the core itself works, without those connective pieces, or with ones that look radically different.
So we need something that can test the core subsystems directly, often in isolation. You know this. It's called...
No new ideas around here, I swear!
So I wrote a unit testing framework, and it's now integrated into upstream. If you have a git clone on your laptop, you can type `make unit` right now and it'll run the tests we have.
This is simplest suite we have, just to show you the output.
This is what a simple test looks like. You'll see "munit" in there; that's from the underlying test framework this is built on. Things like `unit_eq()` are all ours, and it's designed to be incredibly simple and obvious to use, and have lots of nice helpers and affordances for ZFS testing.
Incidentally, thats the reason to do our own - most C unit testing frameworks are awful to use because they require you to write code that isn't much like anything else you do, or go through weird contortions to make your code testable, or other nonsense. I wanted tests to feel like ZFS code, and maybe even be fun to write!
A less trivial example is the ZAP test suite. This is calling directly into the ZAP APIs.
Oh yeah, that's the other thing. If I want to test ZAPs, then I don't want to need all those other annoying things that it would usually require, like a real pool under there somewhere, or a whole working DMU, or whatever. Because I want to make sure the thing works in isolation, not based on behaviours of other subsystems.
And so, the framework has "mocks" of key objects and data structures. Because, internally the ZAP code just operates on a dnode, and actually, just getting its DMU buffer pointer and reading and writing into that memory.
So, a "mock dnode" just has a chunk of memory under it, and a "mock tx" actually currently does nothing. But the types work, and so we can just test the code we're interested in and let the framework take care of the rest.
We get nice things too. Code coverage isn't everything, but it helps a lot for some kinds of work. The framework just takes care of it.
Actually my slide presenter doesn't easily show that it's also has nice colours, so here's a screenshot of the previous output. It's designed so you can get a very quick visual check on how "good" your coverage is.
Of course, it's just lcov under there, so if you like the HTML output...
You can have that too. I just really wanted a quick text output that you can see right there, rather than having to regenerate a whole HTML thing every time, which is particularly a pain if you're working on a remote machine.
So, right now we have a small handful of test suites. Only a couple are mine, most were done by Christos Longros in the first couple of weeks after the framework weren't in, and I'm very grateful to him for pushing it; there's some nice features in there (like type-aware comparisons!) that wouldn't be without him finding problems.
Interestingly, LLMs are incredibly good at being told to write a comprehensive test suite for a specific subsystem. You tell them to under no circumstances modify the original code, and they really like the coverage data. So I've got a bunch that Claude has generated that are waiting for me to review, rewrite, document, etc. Still work, but 80% already done!
And I've got some actual real ones, or extensions to existing ones, in progress.
Of course only a couple of t
I wrote a big describing how it all works, but have a look. It's not hard, that's the point!
Moving on, lets look at host services.
Mostly, I'm talking about the SPL; the Solaris Porting Layer.
Except of course that it probably hasn't looked much like Solaris for a while.
And so it's not really about porting things, not exactly.
And layer, well, at least the tentacles are in pretty deep to call it that.
So it's not a great name.
But that's ok, because the abbreviation has lots of room for us to rename it to something that better describes its function without having to do more than update some code comments.
But the name isn't really important. What we do need to do is pin down exactly what the API isand what the semantics are.
And it turns out, filling out the details of `libspl`, the userspace/POSIX SPL library, has been a really good way to identify a lot of the moving parts.
The POSIX implementations of things used to live directly in `zfs_context.h`, behind a `_KERNEL` gate. So about a year ago Brian and I moved them. Beyond general code cleanliness, this also gave us a chance to see what the "core" services are, that every environment would have to implement in some way.
We've also removed some userspace-specific variants of facilities provided by libspl. These are mostly used by libs and CLI tools, but the benefit is that we now don't have two unrelated and different implementations of the same thing, so it's less to maintain and less for a ZFS generalist to understand.
So an SPL mutex is a bit more than just a lock. It also tracks the current owner.
Until last week, the implementation looked like this.
As it turns out, this isn't actually legal POSIX, because the underlying layout of `pthread_t` is unknown. POSIX does not require it to fit in a register, and it is allowed to be a composite type, like a struct. So, we can't actually assign NULL to it, because it might not be a pointer.
Turns out there isn't even a sentinal value for "no thread" (until C23 anyway), so there's nothing else we can usefully assign to it anyway.
So I thought, no big deal, I'll just do what we do in kernel code, and keep a kthread_t* pointer and use curthread for the owher, and the NULL pointer for "no owner".
And that even kinda looks right, except...
Actually makes it worse. The cast to `uintptr_t` still won't work if its a struct, and even if it does fit, there's nothing that says that two `pthread_t`s are comparable by memory layout. Only `pthread_self()`.
And it turns out its even more complicated, because `pthread_t` doesn't have the same lifetime as `kthread_t`, so isn't a suitable proxy. I'm definitely not going into all of this now, but anyway, last week I rewrote SPL kthreads, and now ownership is tracked properly for owned resources.
All of this is to say that the SPL isn't _just_ mapping the ZFS name for a thing to its nearest host-provided counterpart. There's also specific semantic requirements, most of which are undocumented. So writing those down, in code, comments and/or tests is something I'm slowly picking away at.
But there's another more complicated case of host services, where there might be more than one available, and we need to choose one.
ZFS has a lot of these. Anywhere we've got some data transform, in fact. And we have multiple implementations of many of them, if for no other reason than we may have dedicated hardware available that a different implementation can exploit.
It turns out, if you ignore the specific algorithm and even the type, and just look at the way we use them, we have the same basic requirements for all of these.
They all have the same interface, at least within their type. We mostly aren't doing our own implementations, either getting them elsewhere or using something provided by the host. The operator may want to select which implementation is used, or we can test them and use the "best" one. And there's some way to find out what's available and what's selected.
For the crypto algorithms, we have two implementations we can use.
There's the ICP, which is basically all the crypto stuff we ship ourselves. This is what we use for Linux and in userspace.
In the FreeBSD kernel, we use whatever the kernel's crypto framework provides.
Actually, a couple of weeks ago we got an adapter for the Linux kernel's crypto framework. It's very very new though, and a little complicated to access, but it's very interesting indeed!
Regardless, this is a weird situation. We ship multiple implementations, and we can access the host implementation, but what if, say, on FreeBSD, an ICP implementation could outperform a kernel one? Or what if we want to add a new algorithm, that isn't supported by a host or platform framework at all? Why can't we choose the best of what's available?
$ perl -E 'printf "%-22s%s\n", ($_ =~ m#/([^/]+)$#), do { local @ARGV = ($_); <> } for sort @ARGV' /sys/module/zfs/parameters/*_impl
We have some runtime implementation selectors, but not for everything. Also you'll notice that fletcher4 doesn't have a "cycle" option.
This is actually all weird because the code that drives this selection is repeated in all these modules, usually slightly modified each time.
We also have some stats for the microbenchmarks that we run to choose the "best" version of each. Here's the fletcher4 stats, showing its result for the two fletcher4 variants. Note that "fastest" could produce a different result for each, but recall that there's only one selector available.
It's even more insane for raidz math. One selector for ten different largely-unrelated algorithms.
Also this is all the stats we have for any of these options. Again, the stats code for each of these is very custom, for particular reason.
Theres a company called wolfSSL, that sell a FIPS-certified crypto module exactly like this.
They also ship a huge patch to OpenZFS to make it use it. It rips out the entirety of the ICP, and replaces it with calls into wolfCrypt instead.
This is very cool, and can put ZFS into more hands. It also means those people have to re-patch OpenZFS on every release. It is also a particular kind of punishment for the wolfSSL people, that this is pretty much the only way to do it.
There's more to it than what I've shown here, not least that it isn't just about crypto. This is compression, checksums, and every other thing where we might possibly want to update an implementation, or connect up to whatever the host environment provides.
We cannot and should not try to do everything. We should make it easy to bring in all these interesting things.
So I wrote a new subsystem for ZFS, called zalgo. It's "ZFS algorithms" or that eldritch Unicode horror, your choice. There's a prototype in PR 19129 right now.
These are logs from the prototype. At startup, different implementations of things register with the hub. So we see three ICP algorithms being added, and then an OpenSSL provider registers for those as well (yeah, I wrote an OpenSSL provider for this, just to show it).
It registers various fletcher4 implementations, both for `fletcher4` and `fletcher4-byteswap`. They are _different_ algorithms; there's no reason to lump them under the same selector and microbenchmarks.
At the end, the microbenchmarks are run for each type and subtype. You'll see here that the AVX2 fletcher4 implementations beats "scalar" (just the default C version) by some margin, so it gets selected. Similarly, OpenSSL AES-CCM beats out ICP AES-CCM, and gets selected. ICP GCM wins, so nothing gets changed.
In the Linux kernel, it looks the same, except there's no OpenSSL provider there, so the ICP gets selected.
Here I ported the ICP to the FreeBSD kernel, and here we see them both loading. I've commented out the load for the FreeBSD HMAC-SHA512, which is why the ICP version is the only one. That was deliberate, for an interop test. Works great!
There's a nice kstat to show what's currently available, the relative performance on the microbenchmarks, and the current state of things.
I should note, though this says "ICP", in a real thing it wouldn't be, it would just be the OpenZFS-supplied implementations registered directly (all the different instruction set extensions). Just hitting the ICP direct was much easier for a prototype.
Same kstat for FreeBSD.
Theres a lot more to this than what I've shown you, and if you're interested in this I'd love your feedback there.
Ok so, when you want to tell ZFS to do something, you run a `zfs` or `zpool` command, and eventually it ends up doing a call into the kernel. Basically, a syscall, but as we know in Unixlikes, the way we get those is to open a device, and send it a custom thing.
Oh yeah never mind that, that's just `zfs_cmd_t`, it's seen some shit.
So we wanna tell ZFS to do five barrel rolls. Which is a thing a filesystem can do.
We create an empty nvlist for the request and the response, and add our count parameter to it. If you haven't seen nvlists, they're basically an in-memory typed key-value object. Think Python dictionary.
Then we make the call. Pretty much what we saw before. We have to fill in a bit more zfs_cmd_t than this, but not much more. It really doesn't matter right now I swear.
And then we unpack the results and do something with them and see just how wicked and rad those barrel rolls were!
But what if we wanted to do this?
Ok, that's not hard to arrange exactly; you'd put a HTTP listener on that host, and it'd take the JSON off the wire, convert it back to an nvlist and do the ioctl.
Except if you're doing web calls, you're not doing them in C.
Really, you're hoping to do something more like this.
(and I don't really know much Python but that's ok, any Python heads thinking "well actually" right now, don't worry, because you're definitely right and this is for you)
Because how do you know what the args and returns are here? How do you actually build this API. And then, how do you repeat this for every language and every enviroment.
And you know the answer to this already too. It's documentation! Even better, documentation a machine can read and use to generate the API, that is, a schema.
So we already sort of have a data description for many (not all) of the ioctls.
If you've looked in the ioctl code before, you've seen this. If you squint, this is just a table. The name, what to call, security config, scope, some rules, etc. With a tiny bit of regex, that's just JSON or TOML or whatever and might be the start of a schema.
We even have these mini schema shaped things that we use to validate the args coming in, that describe types and options and things. So we're sorta kinda there.
But it's also kinda half-baked. Like, "props" there is an nvlist. But an nvlist also has keys and types, so what validates its contents?
But then we get into the ioctl implementation, and we have a ton of work to do. We unpack the args, basically restating the names, types, and how optional they are.
Further down, we actually discover more schema-type things. have hardcoded policy: if we're creating a zvol, the "props" arg is actually not optional, and it has to have a uint64 key called "volsize".
And then we just pass two whole nvlists which could have anything in it straight down into some other function somewhere else in the system. Who knows what it needs, or what it does. We do this a lot, and I do not like it at all.
But also it's sucks for anything that might want to embed ZFS, that already has a control system. Do we really want it to create an nvlist in memory, just to pass it down to a function that is going to take it apart again? That's just a crappy function call.
So there's a lot here, and I've been working on it from multiple angles. But the schema stuff has been particular interesting because it would help libzfs a lot, or a libzfs replacement.
So I wrote a marshalling layer. Which is basically like serialise/deserialise, but also presents it in a way that makes the most sense for the language?
So this is the entry point for scrub, and its unpacking everything by hand, like before.
Instead, we can do this. This creates the struct with the named fields, and then this `NVM_UNMARSHAL` call will take the nvlist and unpack it into the object. There's an inverse, `NVM_MARSHAL`, which will go in the other direction.
This is the simple case. I haven't wired it all the way through yet, mostly because of all those other functions that want to take a raw nvlist that need to be untangled first. But it can handle all the weird cases.
Instead, we can do this. This creates the struct with the named fields, and then this `NVM_UNMARSHAL` call will take the nvlist and unpack it into the object. There's an inverse, `NVM_MARSHAL`, which will go in the other direction.
This is the simple case. I haven't wired it all the way through yet, mostly because of all those other functions that want to take a raw nvlist that need to be untangled first. But it can handle all the weird cases.
Lightning overview of what we can do here.
It'll take any nvlist scalar type and make it a regular C scalar. You can say its required, or optional, or optional with a non-zero default.
nvlist treats strings and nvlist members as scalars, so we do too.
It'll take any nvlist scalar type and make it a regular C scalar. You can say its required, or optional, or optional with a non-zero default.
Incidentally, there's magic so that any allocations required during unmarshaling are charged to the source nvlist, so everything is cleaned up properly when you free the nvlist.
All the nvlist array types are handled too. There's two forms, one for arbitrary number of elements, and one for known or fixed. For the arbitrary length, it records the number of elements found. For the fixed version, it won't, and also it'll enforce it - unmarshaling will the array does not have exactly ten elements.
The weird "key with no value" boolean type is modeled as a flag.
Then we get into the weird and cool stuff.
It's pretty common in nvlists to model a list of of things as a sub-nvlist
with some arbitrary number of elements. We know this in other languages as a
map or dictionary. So we can say, this is a map of uint64. Inside the nvlist,
it will be an nvlist full of UINT64 pairs. On the C side, we make a simple
"pair" struct, and you get a regular array of them.
Similarly, we have a set, for the nvlist of those "wierd boolean" things, ie
flags.
The best part. Recursive unmarshalling, enforcing the rules all the way.
So I think that's a pretty cool tool for us to use internally anyway, but going back to the start, it forces us to write down everything that goes into our nvlists. And once we have programattic access to it, we can advertise our own capabilities, or use it to generate a client library, or whatever else.