From: Larry Woodman <lwoodman@redhat.com> Date: Wed, 3 Oct 2007 07:44:03 -0400 Subject: [mm] oom: prevent from killing several processes Message-id: 1191411844.8155.12.camel@dhcp83-56.boston.redhat.com O-Subject: [RHEL5-U2 patch] prevent systems that are "trying" to OOM kill from hanging and/or killing several processes. Bugzilla: 392351 RHEL5 version of this patch: Often when processes must be OOM killed they are "stuck" down inside try_to_free_pages() trying to reclaim memory themselves. When this happens the system hangs and/or starts killing lots of other processes. The attached patch fixes this by bailing out of try_to_free_pages() and __alloc_pages() if the calling process has been OOM killed. Acked-by: Pete Zaitcev <zaitcev@redhat.com> Acked-by: Peter Zijlstra <pzijlstr@redhat.com> diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3798e77..b03e362 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -994,6 +994,8 @@ nofail_alloc: goto nopage; rebalance: + if (test_thread_flag(TIF_MEMDIE)) + goto nopage; cond_resched(); /* We now go into synchronous reclaim */ diff --git a/mm/vmscan.c b/mm/vmscan.c index d50542f..bdd76ee 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -906,6 +906,8 @@ static unsigned long shrink_zone(int priority, struct zone *zone, nr_inactive = 0; while (nr_active || nr_inactive) { + if (test_thread_flag(TIF_MEMDIE)) + return 0; if (nr_active) { nr_to_scan = min(nr_active, (unsigned long)sc->swap_cluster_max); @@ -954,6 +956,8 @@ static unsigned long shrink_zones(int priority, struct zone **zones, for (i = 0; zones[i] != NULL; i++) { struct zone *zone = zones[i]; + if (test_thread_flag(TIF_MEMDIE)) + return 0; if (!populated_zone(zone)) continue; @@ -1014,6 +1018,8 @@ unsigned long try_to_free_pages(struct zone **zones, gfp_t gfp_mask) } for (priority = DEF_PRIORITY; priority >= 0; priority--) { + if (test_thread_flag(TIF_MEMDIE)) + goto out; sc.nr_scanned = 0; if (!priority) disable_swap_token();