GNU Hurd/ open issues/ resource management problems/ zalloc panics

Welcome to... ... the GNU Hurd!

Written by antrik / Olaf Buddenhagen, last updated: 12 Apr 2007.

The Hurd sometimes crashes with a kernel panic saying someting like: "Panic: zalloc failed: zone map exhausted".

These panics are generally caused by some kind of kernel resource exhaustion, but there are several differnt reasons for that.

It used to happen very often under heavy disk load (like large compile jobs), or in a reproducible test case by opening a large number of ports to /dev/null and then closing them all very quickly. The reason for this particular problem has been identified a while back: The multithreaded Hurd servers create a new worker thread whenever a new request (RPC) comes in while all existing threads are busy. When the server is hammered with lots of requests -- which happens both under heavy disk load, and when quickly closing many ports to one server -- it will create an absurd number of threads, causing the resource exhaustion.

The Debian hurd package contains a patch by k0ro (Sergio Lopez), which fixes this by limiting the amount of created threads in a rather simplistic but very effective manner. This patch however hasn't been included in upstream CVS so far. A more elegant solution, suitable for upstream inclusion, would be desirable.

Some panics still seem to happen in very specific situations, like the one described at https://savannah.gnu.org/bugs/?19426 . These are probably the result of bugs that cause port leaks, accidental fork bombs, or similar problems.

In principle, resource exhaustion can also happen by normal use, though this is rather unlikely in the absence of bugs or malicious programs. Nevertheless, all these problems could be avoided (or limited in effect) by introducing some limits on number of processes per user, number of threads and ports per process/user etc.

Trying to track down causes for the panics, I got some interesting results. (UPDATE: Many of my original observations were clearly related to the server thread explosion problem. To avoid confusion, I now removed these, as this is no longer an open issue.)

  • It all started with someone (probably azeem) mentioning that builing some package always crashes Hurd at the same stage of the Debian packaging process (UPDATE: Almost all of these panics when building packages were a result of the thread explosion and don't happen anymore.)
  • Someone (maybe he himself) pointed out that this stage is characterized by many processes being quickly created and destroyed
  • Someone else (probably hde) started some experimenting, to get a reproducible test case
  • He realized that just starting and killing five child processes in quick succession suffices to kill some Hurd systems
  • I tried to confirm this, but it turned out my system is more robust

As I could never reproduce the problem with a small number of quickly killed processes, I can't say whether this problem still exists. While I could reproduce such an effect with first opening and then very quickly closing many ports (which is more or less what happens when quickly killing many processes), I needed really large numbers of processes/ports for that. The thread throtteling patch fixed my test case; but it seems unlikely that killing only five processes could have caused a thread explosion, so maybe hde's observation was a different problem really...

I started various other experiments with creating child processes (fork bombs), resulting in a number of interesting observations:

  • Just forking a large number of processes crashes the Hurd reliably (not surprising)
  • The number of processes at which the panic occurs is very constant (typicallly +-2) under stable conditions, as long as forking doesn't happen too fast
  • The exact number depends on various conditions:
    • Run directly from the Mach console, it's around 1040 on my machine (given enough RAM); however, it drops to 940 when started through a raw ssh session, and to 990 when run under screen through ssh (TODO: check number of ports open per process depending on how it is started) UPDATE: In a later test, I got somewhat larger numbers (don't remember exactly, but well above 1000), but still very constant between successive runs. Not sure what effected this change.
    • It doesn't depend on whether normal user or root
    • With only 128 MiB of RAM, the numbers drop slightly (like 100 less or so); no further change between 256 and 384 MiB
    • Lowering zone_map_size in mach/kern/zalloc.c reduces the numbers (quite exactly half from 8 MiB to 4 MiB)
    • There seems to be some saturation near 16 MiB however: The difference between 8 MiB and 16 MiB is significantly smaller
    • Also, with 8 MiB or 4 MiB, the difference between console/ssh/screen becomes much more apparent (500 vs. 800, 250 vs. 400)
    • With more than 16 MiB, Mach doesn't even boot
  • Creating the processes very fast results in a sooner and less predictable crash (TODO: Check whether this is still the case with thread throtteling?)
  • Creating processes recursively (fork only one child which forks the next one etc.) results in faster crash
  • rpcinfo shows that child processes have more ports open by default, which is very likely the reason for the above observation
  • Opening many ports from a few processes doesn't usually cause a system crash; there are only lots of open() failures and translator faults once some limit is reached... Seems the zalloc-full condition is better caught on open() than on fork() (TODO: investigate this further, with different memory sizes, different zone_map_size, different kinds of resources using zalloc etc.)
  • After opening/leaking lots of ports to /dev/null (32768 it seems), the NULL translator somehow becomes disfunctional, and a new instance is started

While most of these Observations clearly show an exhaustion of kernel memory which is not surprising, some of the oddities seem to indicate problems that might deserve further investigation.

IRC, freenode, #hurd, 2012-04-01

<mel__> antrik: i just found
  http://www.gnu.org/software/hurd/open_issues/resource_management_problems/zalloc_panics.html
  -- that is from 2007. is this still the current status?
<youpi> mel__: probably
<mcsim> mel__: gnumach has no more zalloc allocator, so I doubt if it could
  be a problem.

gnumach memory management.

<youpi> mcsim: but it still has an allocator
<youpi> which can run out of resources
<mcsim> AFAIR, now there is no such limit.
<youpi> err, there is
<youpi> the size of your RAM :)
<mcsim> In zalloc appearing of this message didn't depend of available size
  of free ram.
<youpi> then update the description, but I'm still getting allocation
  errors, when userland makes crazy things like creating millions of tasks
<mcsim> At least it could appear when there still was free memory
<youpi> and that's not surprising
<youpi> sure, I know that *some* limits have been removed, but there
  weren't so many, and I have seen cases where it's simply mach running out
  of memory
<youpi> also, we have a limited amount of virtual addressing space
<antrik> mel__: this writeup is outdated in several regards. *some* of the
  observations might still be relevant, but nothing that seems
  particularily important
<antrik> the zalloc panics have pretty much disappeared after the default
  zalloc zone size has been considerably extended (which was not possible
  before because of some bug)
<mel__> i see
<antrik> but as mcsim pointed out, with the new allocator not relying on a
  fixed-sized zalloc zone at all, they are even less likely, and should
  happen only if all memory is exhausted
<antrik> I guess this outdated report can just be dropped
<mcsim> I think, that now it is problem rather of absence of OOM-killer or
  resource manager
<antrik> mcsim: right :-)
<antrik> (and we have separate articles about that)