[ 8390.456752] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.21@tcp1 Q 0 ffff81007a708000 172.16.3.21 [ 8390.473288] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.21@tcp1 ip 172.16.3.21:988 [ 8390.488590] LustreError: A timeout occurred sending data to 172.16.3.22; the network or that node may be down. [ 8390.490967] BUG: soft lockup detected on CPU#0! [ 8390.490970] CPU 0: [ 8390.490972] Modules linked in: nfsd exportfs osc llite lov mdc ksocklnd ptlrpc lnet obdclass lvfs libcfs bluesmoke_k8 bluesmoke_mc mptspi mptscsih mptbase bcm5700 w83627hf e [ 8390.490988] Pid: 4653, comm: nfsd Not tainted 2.6.15.7 #1 [ 8390.490990] RIP: 0010:[] {get_num_ver+296} [ 8390.490997] RSP: 0018:ffff810056a5b668 EFLAGS: 00000246 [ 8390.491000] RAX: 0000000000000000 RBX: ffffffff8012a0fe RCX: 0000000000000000 [ 8390.491003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff81001a264030 [ 8390.491005] RBP: ffff810056a5ba98 R08: 00000000ffffffff R09: 0000000000000000 [ 8390.491008] R10: ffff81001a264000 R11: ffff810056a5bc58 R12: 0000000000000008 [ 8390.491011] R13: ffff810056a5b778 R14: 0000000000000001 R15: ffff81007a082c00 [ 8390.491015] FS: 00002aaaaae00680(0000) GS:ffffffff80469800(0000) knlGS:00000000556a82a0 [ 8390.491017] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 8390.491020] CR2: 00002aaaab459000 CR3: 0000000010502000 CR4: 00000000000006e0 [ 8390.491022] [ 8390.491022] Call Trace:{get_num_ver+296} {fix_nodes+2039} [ 8390.491034] {reiserfs_paste_into_item+272} {reiserfs_add_entry+727} [ 8390.491071] {__up_wakeup+53} {reiserfs_create+262} [ 8390.491090] {vfs_create+182} {:nfsd:nfsd_create_v3+791} [ 8390.491115] {:nfsd:nfsd3_proc_create+308} {:nfsd:nfsd_dispatch+219} [ 8390.491140] {svc_process+1011} {default_wake_function+0} [ 8390.491149] {:nfsd:nfsd+0} {:nfsd:nfsd+454} [ 8390.491169] {child_rip+8} {:nfsd:nfsd+0} [ 8390.491182] {:nfsd:nfsd+0} {child_rip+0} [ 8390.491197] [ 8390.725330] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.22@tcp1 Q 0 ffff81003b09c000 172.16.3.22 [ 8390.741871] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.22@tcp1 ip 172.16.3.22:988 [ 8390.757169] LustreError: A timeout occurred sending data to 172.16.3.23; the network or that node may be down. [ 8390.769147] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.23@tcp1 Q 0 ffff81003b204000 172.16.3.23 [ 8390.785697] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.23@tcp1 ip 172.16.3.23:988 [ 8390.800993] LustreError: A timeout occurred sending data to 172.16.3.24; the network or that node may be down. [ 8390.812984] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.24@tcp1 Q 0 ffff81003b200000 172.16.3.24 [ 8390.829520] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.24@tcp1 ip 172.16.3.24:988 [ 8390.844848] Lustre: 1793:0:(router.c:184:lnet_notify()) Upcall: NID 172.16.3.21@tcp1 is dead [ 8390.855288] Lustre: 9:0:(linux-debug.c:96:libcfs_run_upcall()) Invoked portals upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,172.16.3.21@tcp1,down,1152188191 [ 8394.440878] LustreError: A timeout occurred sending data to 172.16.3.253; the network or that node may be down. [ 8394.452970] LustreError: 1793:0:(socklnd_cb.c:1945:ksocknal_find_timed_out_conn()) Timed out TX to 12345-172.16.3.253@tcp1 Q 0 ffff81007a6d0000 172.16.3.253 [ 8394.469717] LustreError: 1793:0:(socklnd_cb.c:1977:ksocknal_check_peer_timeouts()) Timeout out conn->12345-172.16.3.253@tcp1 ip 172.16.3.253:988 [ 8394.485234] Lustre: 1793:0:(router.c:184:lnet_notify()) Upcall: NID 172.16.3.253@tcp1 is dead [ 8394.495446] Lustre: 1793:0:(router.c:184:lnet_notify()) previously skipped 3 similar messages