From: Jeff Layton <jlayton@redhat.com> Date: Thu, 4 Jun 2009 10:23:12 -0400 Subject: [net] sunrpc: remove flush_workqueue from xs_connect Message-id: 1244125392-4954-1-git-send-email-jlayton@redhat.com O-Subject: [RHEL5.5 PATCH] BZ#495059: sunrpc: remove unneeded flush_workqueue from xs_connect Bugzilla: 495059 Posting this as a RHEL5.5 patch, but it might be nice to get this into 5.4 if there's time... Frank Filz at IBM ran across this deadlock in a -128.el5 kernel. The NFSv4 renewal thread was hung here: 10.0.11.43-re D 0000000000000000 12512 14208 515 13543 (L-TLB) Call Trace: [C0000000017475B0] [C000000000565D28] 0xc000000000565d28 (unreliable) [C000000001747780] [C000000000010ACC] .__switch_to+0x124/0x148 [C000000001747810] [C0000000003C6454] .schedule+0xbfc/0xdb0 [C000000001747920] [C000000000081DE0] .flush_cpu_workqueue+0xbc/0x140 [C0000000017479E0] [C000000000081EFC] .flush_workqueue+0x98/0xe0 [C000000001747A70] [D000000000D2545C] .xs_connect+0x108/0x128 [sunrpc] [C000000001747B00] [D000000000D240FC] .xprt_connect+0x194/0x1bc [sunrpc] [C000000001747BA0] [D000000000D21F34] .call_connect+0xa0/0xc0 [sunrpc] [C000000001747C30] [D000000000D2841C] .__rpc_execute+0xec/0x344 [sunrpc] [C000000001747CD0] [D000000000D20E84] .rpc_call_sync+0xac/0x100 [sunrpc] [C000000001747D70] [D000000000F3D18C] .nfs4_proc_renew+0x68/0xe0 [nfs] [C000000001747E20] [D000000000F4CBD8] .reclaimer+0x80/0x3a4 [nfs] [C000000001747EE0] [C000000000087248] .kthread+0x128/0x178 [C000000001747F90] [C0000000000274C4] .kernel_thread+0x4c/0x68 ...and there's a hung events thread that looks like this: events/5 D 0000000000000000 9792 31 1 32 30 (L-TLB) Call Trace: [C0000001DFDB77D0] [C0000001D9430C00] 0xc0000001d9430c00 (unreliable) [C0000001DFDB79A0] [C000000000010ACC] .__switch_to+0x124/0x148 [C0000001DFDB7A30] [C0000000003C6454] .schedule+0xbfc/0xdb0 [C0000001DFDB7B40] [C0000000003C8C10] .rwsem_down_read_failed+0x27c/0x2c4 [C0000001DFDB7C10] [C00000000008B06C] .down_read+0x44/0x5c [C0000001DFDB7CA0] [D000000000F4DED8] .nfs4_renew_state+0x3c/0x270 [nfs] [C0000001DFDB7D50] [C0000000000819D0] .run_workqueue+0xdc/0x168 [C0000001DFDB7DF0] [C00000000008273C] .worker_thread+0x12c/0x19c [C0000001DFDB7EE0] [C000000000087248] .kthread+0x128/0x178 [C0000001DFDB7F90] [C0000000000274C4] .kernel_thread+0x4c/0x68 This makes all access to any NFS mounts come to a stop. The renewal thread is holding the semaphore that the events thread needs to complete its work. It called flush_workqueue but that can't complete until the work on events/5 is done. I believe the flush_workqueue() call in xs_connect() is no longer needed and should have been removed as part of the fix for BZ#448754. Since that code no longer queues work to the events workqueue, there should be no need to flush it. This is a difficult to reproduce deadlock so I can't confirm that this fixes the problem, but I have run this patch in my test kernels for a month or two and haven't seen any ill effects. Signed-off-by: Jeff Layton <jlayton@redhat.com> diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index 2b694e1..a46af3b 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -1275,10 +1275,6 @@ static void xs_connect(struct rpc_task *task) } else { dprintk("RPC: xs_connect scheduled xprt %p\n", xprt); queue_work(rpciod_workqueue, &xprt->connect_worker); - - /* flush_scheduled_work can sleep... */ - if (!RPC_IS_ASYNC(task)) - flush_scheduled_work(); } }