GNU Hurd/ user/ Sergio Lopez

Welcome to... ... the GNU Hurd!

About Myself

Stuff I'm working on

Advisory Pageout

libpager deadlock, gnumach memory management 2.

Rationale

This work has two objectives:

  • Put a limit on the number of dirty (or better said, potentially dirty) pages that can be at one given time on Hurd. Having lots of dirty pages stresses the translators when syncing its memory objects, since they could receive a huge amount of m_o_data_return messages in a short amount of time. This is the primary cause for thread explosions.

  • When memory is scarce due to work load, try to avoid flushing anonymous memory. This is done by trying to satisfy the target of free pages releasing clean external pages (which can be considered as cache).

Trying it

I've commited this work to this branch:

https://git.savannah.gnu.org/cgit/hurd/gnumach.git/log/?h=k0ro/advisory_pageout/master

You'll also need the counterpart for user space:

http://blogs.nologin.es/slopez/files/hurd-advisory_pageout.patch

If you try it, please let me know your experience with it.

Reduce task map fragmentation

After some hours of workload over an ext2fs translator, the number of entries in its memory map (which uses to be something around 60 when clean) is increased up to a few thousand. This increases the cost of each vm_map_lookup_entry, and this is perceived from a user perspective as a significant slowdown on the services provided by that translator.

The causes for the fragmentation of the task map are:

The problem

  • When a big chunk of memory is reserved, and the protection level is changed in small fractions of it, each of those generates a new entry in the map. This happens in glibc's malloc and sysdeps/mach/hurd/brk.c.

  • Allocating memory for permanent data structures in small chunks, without putting them at an specific address. GNU Mach has the ability of coalescing similar memory entries, but they must be contiguous. As every Hurd server needs to allocate small, temporal buffers for sending and receiving data, these ones will get in the middle of the permanent chunks, preventing them for being coalesced. This is the case of glibc's hurd/hurdmalloc.c, libpager's pagemap and libthread's per thread stack (and possibly some others).

The solution

  • GNU Mach should be able of coalescing entries with similar properties after a call to vm_protect. I'm working in this direction.

  • The use of vm_allocate/mmap in translators should be avoided, except for temporal buffers for IPC operations.

Workaround

While trying to find the causes of the task map fragmentation, I developed some changes which don't provide a complete, clean solution for this problem, but they minimize it (at least, for ext2fs) without significant drawbacks. You can find them here:

memfs (Memory FS translator)

Other miscellaneous changes