Does ARC ever conflict/double-cache with Linux page cache?

[qaako]

I heard from somewhere that ZFS double caches what's already in the page cache. I'm wondering how often this occurs, and is it possible to disable the linux page cache since we have ARC? Also I think the file /proc/sys/vm/drop_caches is meant for page cache, does it affect ARC too?

[IvanVolosyuk]

Linux page cache is a smart idea that programs in memory can share memory with disk cache. Different programs can load the same shared libraries, like libc, libX11, etc. For position independent code the library in memory contain the same data as on disk, thus you can have just a single page for all copies of libc page in memory and and disk cache. I don't think you want to disable page cache and it is even possible as it fundamental to Linux memory management subsystem.

[qaako]

So does ZFS double cache what's already in the page cache?

[IvanVolosyuk]

In simplified view, a few points:

• there is a high chance for frequently used and currently running program data be available in compressed form in ARC as well as uncompressed in page cache.
• For files read from local disk using read() kernel calls (media players) - there duplication of data happens for all file-systems and should be minimal overall.
• For specific cases where applications (like databases) use their own caching - there is a way to disable OpenZFS data caching.

OpenZFS is highly tunnable file system and you can customize compression, encryption, caching, block sizes per dataset depending on your needs: performance, storage efficiency, memory usage.

[qaako]

Does /proc/sys/vm/drop_caches affect ARC too or it only affects page cache?
Also, if I were to set primarycache=none or limit zfs_arc_max to something extremely low like 20MB, is page cache still in effect for the ZFS filesystem? And would the page cache be enough to make ZFS have a similar level of caching to other filesystems such as ext4 or btrfs?

[amotin]

For regular files only data accessed via mmap() are going through the page cache, while via regular read()/write() are not. The mentioned shared libraries do use mmap(), so there may be 2 or 3 copies of data for all the instances of the library -- one always in page cache (if actually accessed by application), another possibly in ARC, and one more may be in ZFS dbuf cache. For ZVOLs page cache might be used if the device is not opened with O_DIRECT, but that is up to the kernel, ZFS has no say here.

Does /proc/sys/vm/drop_caches affect ARC too

Yes, it does. Though through a different interface than page cache, and it does not mean that ARC can always free all the memory.

or limit zfs_arc_max to something extremely low like 20MB

I would not recommend that, since not all ARC content is evictable, and you are putting it into very uncomfortable position.

is page cache still in effect for the ZFS filesystem

ZFS in no way controls the page cache. Neither ii uses it.

And would the page cache be enough to make ZFS have a similar level of caching to other filesystems such as ext4 or btrfs?

Page cache is highly integrated into the Linux kernel. Would ZFS used Solaris buffer cache, it would likely never be ported anywhere else. Respectively integration with Linux page cache would mean a total rewrite, not mentioning endless restrictions by Linux kernel developers by declaring symbols as GPL or not exposing outside at all.

[qaako]

So the page cache on ext4/btrfs does more than page cache on ZFS?
So if I set primarycache=none, it is not going to have the same amount of page caching done as on ext4/btrfs?

[amotin]

Other file systems on Linux are effectively built around the page cache. ZFS on other side while knows about the page cache to allow mmap() work properly, does not use it directly, allocating all its memory through other mechanisms. Setting primarycache=none you are disabling caching as much as possible, while still keeping read-ahead and write-back, without which ZFS either can not function or would be too slow.