Special vdev for metadata and small blocks on raidz not just mirror

[rcfa]

Maybe I’m wrong, but I’m under the impression that currently special vdevs for metadata and small blocks/files can only be of type stripe or mirror. Someone elsewhere explained that historically the feature was implemented to improve draid performance, and of course, if one specifically adds SSDs to a spinning disk array to improve performance, one would use the fastest possible way of doing that, and that’s of course not a raidzN setup. Perfectly reasonable.

However, I come from a completely different angle: I have a SSD RAIDZ2 pool and a spinning disk RAIDZ2 pool, and there is no plan to add additional devices, for space and cost reasons. However, since in both cases metadata is stored on a RAIDZ vdev anyway, it still would speed up things if the spinning disks had their metadata on the SSDs. Similarly, having two pools and having to manually decide which data goes where brings additional overhead.
If I could create a pool where both the SSDs and the spinning disks were RAIDZ2, but the SSDs were used as special vdev for small files/blocks and metadata, then most file ops would be as fast as an SSD RAIDZ2 even on the spinning disks, and I could organize the file system logically, rather than by what files I want on the spinners vs. on the SSDs.

So, coming from that angle, where everything is RAIDZ anyway, and it’s just a question if things end up on slow or fast hardware, having a special vdev as raidz2 suddenly does make sense.

Am I missing something here? If not, how big would be the effort to actually get this implemented, given that both raidz2 logic is fully implemented and special vdev logic is implemented. It would almost seem, recommended practices aside, once a vdev is created, the zpool should be agnostic to how it’s being used. After all, nothing would stop me (except common sense) to use a mirror of spinning disk drives as special vdev for an SSD RAIDZ2 pool. 😆

[tonyhutter]

Maybe I’m wrong, but I’m under the impression that currently special vdevs for metadata and small blocks/files can only be of type stripe or mirror.

Special devices can be any type, including raidz2 or draid. They just have to match the redundancy level of the pool. You can verify the behavior by making a test pool of sparse files:

$ truncate -s 100M file{1..20}
$ sudo ./zpool create tank draid ./file{1..10} special draid ./file{11..20}
$

[amotin]

Before ZFS 2.4 special was artificially restricted to be only mirrors. It was relaxed just recently.

[rcfa]

Before ZFS 2.4? I thought we’re at 2.3.4? So is this going to be relaxed in an upcoming release? Or did you mean 2.3.4, since you used "was" rather than "will be"?

[amotin]

2.4 is now in RC1 stage, testers are welcome. Technically it would be trivial to merge it to 2.3.x, since the limitation is completely artificial, but it is quite a big user-visible functionality change.

[rcfa]

Before ZFS 2.4 special was artificially restricted to be only mirrors. It was relaxed just recently.

Since you say it was artificially restricted, was that restriction only in the options to create such pools, or in the rest of the code, too?
In other words: were I to create a pool with ZFS 2.4RC1 that were to use a RAIDZ2 special vdev, could it be mounted by current stable releases of ZFS (e.g. as being used in TrueNAS (2.3.4)), or would the pool be recognized as incompatible?

[amotin]

Yes, it was limited only on creation, so should be backwards-compatible. But same time 2.4 has few other pool features added, which are not compatible, that you may need them explicitly disabled on creation if you wish 2.3 compatibility.

[rcfa]

Another thing that’s not quite clear to me: if a special vdev is full, will it simply overflow to the regular devices, or will I experience something akin to a regular file system running out of inodes even though there is still plenty of unused disk space?
Conversely, if the regular devices are full, will things bigger than the small files limit end up on the special vdev, or will the pool be full, even though there would still be space on the special vdev?

[amotin]

Currently special vdevs overflow to normal, but not backwards. Dedup vdevs first overflow to special and then to normal.

[rcfa]

So then I could err on the side of writing too much on the special vdev, and if it turns out to fill up too quickly, I could change the parameters and use the new zfs rewrite functionality to rebalance things, right?

[amotin]

Right. But IIRC special vdev also has an additional mechanism to disable special_small_blocks once reaching certain percent of capacity, so overflow might be not too trivial to trigger, unless significantly fragmented.