2.6.23.1: mdadm/raid5 hung/d-state
am 04.11.2007 13:03:30 von Justin Piszcz
# ps auxww | grep D
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 273 0.0 0.0 0 0 ? D Oct21 14:40 [pdflush]
root 274 0.0 0.0 0 0 ? D Oct21 13:00 [pdflush]
After several days/weeks, this is the second time this has happened, while
doing regular file I/O (decompressing a file), everything on the device
went into D-state.
# mdadm -D /dev/md3
/dev/md3:
Version : 00.90.03
Creation Time : Wed Aug 22 10:38:53 2007
Raid Level : raid5
Array Size : 1318680576 (1257.59 GiB 1350.33 GB)
Used Dev Size : 146520064 (139.73 GiB 150.04 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 3
Persistence : Superblock is persistent
Update Time : Sun Nov 4 06:38:29 2007
State : active
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 1024K
UUID : e37a12d1:1b0b989a:083fb634:68e9eb49
Events : 0.4309
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
2 8 65 2 active sync /dev/sde1
3 8 81 3 active sync /dev/sdf1
4 8 97 4 active sync /dev/sdg1
5 8 113 5 active sync /dev/sdh1
6 8 129 6 active sync /dev/sdi1
7 8 145 7 active sync /dev/sdj1
8 8 161 8 active sync /dev/sdk1
9 8 177 9 active sync /dev/sdl1
If I wanted to find out what is causing this, what type of debugging would
I have to enable to track it down? Any attempt to read/write files on the
devices fails (also going into d-state). Is there any useful information
I can get currently before rebooting the machine?
# pwd
/sys/block/md3/md
# ls
array_state dev-sdj1/ rd2@ stripe_cache_active
bitmap_set_bits dev-sdk1/ rd3@ stripe_cache_size
chunk_size dev-sdl1/ rd4@ suspend_hi
component_size layout rd5@ suspend_lo
dev-sdc1/ level rd6@ sync_action
dev-sdd1/ metadata_version rd7@ sync_completed
dev-sde1/ mismatch_cnt rd8@ sync_speed
dev-sdf1/ new_dev rd9@ sync_speed_max
dev-sdg1/ raid_disks reshape_position sync_speed_min
dev-sdh1/ rd0@ resync_start
dev-sdi1/ rd1@ safe_mode_delay
# cat array_state
active-idle
# cat mismatch_cnt
0
# cat stripe_cache_active
1
# cat stripe_cache_size
16384
# cat sync_action
idle
# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid1 sdb2[1] sda2[0]
136448 blocks [2/2] [UU]
md2 : active raid1 sdb3[1] sda3[0]
129596288 blocks [2/2] [UU]
md3 : active raid5 sdl1[9] sdk1[8] sdj1[7] sdi1[6] sdh1[5] sdg1[4] sdf1[3]
sde1[2] sdd1[1] sdc1[0]
1318680576 blocks level 5, 1024k chunk, algorithm 2 [10/10]
[UUUUUUUUUU]
md0 : active raid1 sdb1[1] sda1[0]
16787776 blocks [2/2] [UU]
unused devices:
#
Justin.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.23.1: mdadm/raid5 hung/d-state (md3_raid5 stuck in endlessloop?)
am 04.11.2007 13:39:13 von Justin Piszcz
Time to reboot, before reboot:
top - 07:30:23 up 13 days, 13:33, 10 users, load average: 16.00, 15.99, 14.96
Tasks: 221 total, 7 running, 209 sleeping, 0 stopped, 5 zombie
Cpu(s): 0.0%us, 25.5%sy, 0.0%ni, 74.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8039432k total, 1744356k used, 6295076k free, 164k buffers
Swap: 16787768k total, 160k used, 16787608k free, 616960k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
688 root 15 -5 0 0 0 R 100 0.0 121:21.43 md3_raid5
273 root 20 0 0 0 0 D 0 0.0 14:40.68 pdflush
274 root 20 0 0 0 0 D 0 0.0 13:00.93 pdflush
# cat /proc/fs/xfs/stat
extent_alloc 301974 256068291 310513 240764389
abt 1900173 15346352 738568 731314
blk_map 276979807 235589732 864002 211245834 591619 513439614 0
bmbt 50717 367726 14177 11846
dir 3818065 361561 359723 975628
trans 48452 2648064 570998
ig 6034530 2074424 43153 3960106 0 3869384 460831
log 282781 10454333 3028 399803 173488
push_ail 3267594 0 1620 2611 730365 0 4476 0 10269 0
xstrat 291940 0
rw 61423078 103732605
attr 0 0 0 0
icluster 312958 97323 419837
vnodes 90721 4019823 0 1926744 3929102 3929102 3929102 0
buf 14678900 11027087 3651843 25743 760449 0 0 15775888 280425
xpc 966925905920 1047628533165 1162276949815
debug 0
# cat meminfo
MemTotal: 8039432 kB
MemFree: 6287000 kB
Buffers: 164 kB
Cached: 617072 kB
SwapCached: 0 kB
Active: 178404 kB
Inactive: 589880 kB
SwapTotal: 16787768 kB
SwapFree: 16787608 kB
Dirty: 494280 kB
Writeback: 86004 kB
AnonPages: 151240 kB
Mapped: 17092 kB
Slab: 259696 kB
SReclaimable: 170876 kB
SUnreclaim: 88820 kB
PageTables: 11448 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 20807484 kB
Committed_AS: 353536 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 15468 kB
VmallocChunk: 34359722699 kB
# echo 3 > /proc/sys/vm/drop_caches
# cat /proc/meminfo
MemTotal: 8039432 kB
MemFree: 6418352 kB
Buffers: 32 kB
Cached: 597908 kB
SwapCached: 0 kB
Active: 172028 kB
Inactive: 579808 kB
SwapTotal: 16787768 kB
SwapFree: 16787608 kB
Dirty: 494312 kB
Writeback: 86004 kB
AnonPages: 154104 kB
Mapped: 17416 kB
Slab: 144072 kB
SReclaimable: 53100 kB
SUnreclaim: 90972 kB
PageTables: 11832 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 20807484 kB
Committed_AS: 360748 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 15468 kB
VmallocChunk: 34359722699 kB
Nothing is actually happening on the device itself however.
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sde 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdh 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdi 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdj 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdk 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdl 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
6 0 160 6420244 32 600092 0 0 221 227 5 1 1 1 98 0
6 0 160 6420228 32 600120 0 0 0 0 1015 142 0 25 75 0
6 0 160 6420228 32 600120 0 0 0 0 1005 127 0 25 75 0
6 0 160 6420228 32 600120 0 0 0 41 1022 151 0 26 74 0
6 0 160 6420228 32 600120 0 0 0 0 1011 131 0 25 75 0
6 0 160 6420228 32 600120 0 0 0 0 1013 124 0 25 75 0
6 0 160 6420228 32 600120 0 0 0 0 1042 129 0 25 75 0
# uname -mr
2.6.23.1 x86_64
# cat /proc/vmstat
nr_free_pages 1598911
nr_inactive 146381
nr_active 42724
nr_anon_pages 37181
nr_mapped 4097
nr_file_pages 151975
nr_dirty 123572
nr_writeback 21501
nr_slab_reclaimable 16152
nr_slab_unreclaimable 24284
nr_page_table_pages 2823
nr_unstable 0
nr_bounce 0
nr_vmscan_write 20712
pgpgin 1015377151
pgpgout 1043634578
pswpin 0
pswpout 40
pgalloc_dma 4
pgalloc_dma32 319052932
pgalloc_normal 621945603
pgalloc_movable 0
pgfree 942598566
pgactivate 31123819
pgdeactivate 18438560
pgfault 360236898
pgmajfault 16158
pgrefill_dma 0
pgrefill_dma32 11683348
pgrefill_normal 18799274
pgrefill_movable 0
pgsteal_dma 0
pgsteal_dma32 176658679
pgsteal_normal 233628315
pgsteal_movable 0
pgscan_kswapd_dma 0
pgscan_kswapd_dma32 164181746
pgscan_kswapd_normal 217338820
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 13074075
pgscan_direct_normal 17342937
pgscan_direct_movable 0
pginodesteal 332816
slabs_scanned 12368000
kswapd_steal 380216091
kswapd_inodesteal 9858653
pageoutrun 1167045
allocstall 68454
pgrotated 40
# cat /proc/zoneinfo
Node 0, zone DMA
pages free 2601
min 3
low 3
high 4
scanned 0 (a: 11 i: 12)
spanned 4096
present 2486
nr_free_pages 2601
nr_inactive 0
nr_active 0
nr_anon_pages 0
nr_mapped 1
nr_file_pages 0
nr_dirty 0
nr_writeback 0
nr_slab_reclaimable 0
nr_slab_unreclaimable 4
nr_page_table_pages 0
nr_unstable 0
nr_bounce 0
nr_vmscan_write 0
protection: (0, 3246, 7917, 7917)
pagesets
cpu: 0 pcp: 0
count: 0
high: 0
batch: 1
cpu: 0 pcp: 1
count: 0
high: 0
batch: 1
vm stats threshold: 6
cpu: 1 pcp: 0
count: 0
high: 0
batch: 1
cpu: 1 pcp: 1
count: 0
high: 0
batch: 1
vm stats threshold: 6
cpu: 2 pcp: 0
count: 0
high: 0
batch: 1
cpu: 2 pcp: 1
count: 0
high: 0
batch: 1
vm stats threshold: 6
cpu: 3 pcp: 0
count: 0
high: 0
batch: 1
cpu: 3 pcp: 1
count: 0
high: 0
batch: 1
vm stats threshold: 6
all_unreclaimable: 1
prev_priority: 12
start_pfn: 0
Node 0, zone DMA32
pages free 699197
min 1166
low 1457
high 1749
scanned 0 (a: 14 i: 0)
spanned 1044480
present 831104
nr_free_pages 699197
nr_inactive 38507
nr_active 11855
nr_anon_pages 11228
nr_mapped 612
nr_file_pages 39127
nr_dirty 38462
nr_writeback 34
nr_slab_reclaimable 8164
nr_slab_unreclaimable 4747
nr_page_table_pages 756
nr_unstable 0
nr_bounce 0
nr_vmscan_write 6132
protection: (0, 0, 4671, 4671)
pagesets
cpu: 0 pcp: 0
count: 183
high: 186
batch: 31
cpu: 0 pcp: 1
count: 52
high: 62
batch: 15
vm stats threshold: 36
cpu: 1 pcp: 0
count: 23
high: 186
batch: 31
cpu: 1 pcp: 1
count: 14
high: 62
batch: 15
vm stats threshold: 36
cpu: 2 pcp: 0
count: 173
high: 186
batch: 31
cpu: 2 pcp: 1
count: 61
high: 62
batch: 15
vm stats threshold: 36
cpu: 3 pcp: 0
count: 95
high: 186
batch: 31
cpu: 3 pcp: 1
count: 57
high: 62
batch: 15
vm stats threshold: 36
all_unreclaimable: 0
prev_priority: 12
start_pfn: 4096
Node 0, zone Normal
pages free 897091
min 1678
low 2097
high 2517
scanned 0 (a: 29 i: 0)
spanned 1212416
present 1195840
nr_free_pages 897091
nr_inactive 107874
nr_active 30878
nr_anon_pages 25956
nr_mapped 3484
nr_file_pages 112857
nr_dirty 85110
nr_writeback 21467
nr_slab_reclaimable 7988
nr_slab_unreclaimable 19546
nr_page_table_pages 2067
nr_unstable 0
nr_bounce 0
nr_vmscan_write 14580
protection: (0, 0, 0, 0)
pagesets
cpu: 0 pcp: 0
count: 124
high: 186
batch: 31
cpu: 0 pcp: 1
count: 1
high: 62
batch: 15
vm stats threshold: 42
cpu: 1 pcp: 0
count: 68
high: 186
batch: 31
cpu: 1 pcp: 1
count: 9
high: 62
batch: 15
vm stats threshold: 42
cpu: 2 pcp: 0
count: 79
high: 186
batch: 31
cpu: 2 pcp: 1
count: 10
high: 62
batch: 15
vm stats threshold: 42
cpu: 3 pcp: 0
count: 47
high: 186
batch: 31
cpu: 3 pcp: 1
count: 60
high: 62
batch: 15
vm stats threshold: 42
all_unreclaimable: 0
prev_priority: 12
start_pfn: 1048576
On Sun, 4 Nov 2007, Justin Piszcz wrote:
> # ps auxww | grep D
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 273 0.0 0.0 0 0 ? D Oct21 14:40 [pdflush]
> root 274 0.0 0.0 0 0 ? D Oct21 13:00 [pdflush]
>
> After several days/weeks, this is the second time this has happened, while
> doing regular file I/O (decompressing a file), everything on the device went
> into D-state.
>
> # mdadm -D /dev/md3
> /dev/md3:
> Version : 00.90.03
> Creation Time : Wed Aug 22 10:38:53 2007
> Raid Level : raid5
> Array Size : 1318680576 (1257.59 GiB 1350.33 GB)
> Used Dev Size : 146520064 (139.73 GiB 150.04 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 3
> Persistence : Superblock is persistent
>
> Update Time : Sun Nov 4 06:38:29 2007
> State : active
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 1024K
>
> UUID : e37a12d1:1b0b989a:083fb634:68e9eb49
> Events : 0.4309
>
> Number Major Minor RaidDevice State
> 0 8 33 0 active sync /dev/sdc1
> 1 8 49 1 active sync /dev/sdd1
> 2 8 65 2 active sync /dev/sde1
> 3 8 81 3 active sync /dev/sdf1
> 4 8 97 4 active sync /dev/sdg1
> 5 8 113 5 active sync /dev/sdh1
> 6 8 129 6 active sync /dev/sdi1
> 7 8 145 7 active sync /dev/sdj1
> 8 8 161 8 active sync /dev/sdk1
> 9 8 177 9 active sync /dev/sdl1
>
> If I wanted to find out what is causing this, what type of debugging would I
> have to enable to track it down? Any attempt to read/write files on the
> devices fails (also going into d-state). Is there any useful information I
> can get currently before rebooting the machine?
>
> # pwd
> /sys/block/md3/md
> # ls
> array_state dev-sdj1/ rd2@ stripe_cache_active
> bitmap_set_bits dev-sdk1/ rd3@ stripe_cache_size
> chunk_size dev-sdl1/ rd4@ suspend_hi
> component_size layout rd5@ suspend_lo
> dev-sdc1/ level rd6@ sync_action
> dev-sdd1/ metadata_version rd7@ sync_completed
> dev-sde1/ mismatch_cnt rd8@ sync_speed
> dev-sdf1/ new_dev rd9@ sync_speed_max
> dev-sdg1/ raid_disks reshape_position sync_speed_min
> dev-sdh1/ rd0@ resync_start
> dev-sdi1/ rd1@ safe_mode_delay
> # cat array_state
> active-idle
> # cat mismatch_cnt
> 0
> # cat stripe_cache_active
> 1
> # cat stripe_cache_size
> 16384
> # cat sync_action
> idle
> # cat /proc/mdstat
> Personalities : [raid1] [raid6] [raid5] [raid4]
> md1 : active raid1 sdb2[1] sda2[0]
> 136448 blocks [2/2] [UU]
>
> md2 : active raid1 sdb3[1] sda3[0]
> 129596288 blocks [2/2] [UU]
>
> md3 : active raid5 sdl1[9] sdk1[8] sdj1[7] sdi1[6] sdh1[5] sdg1[4] sdf1[3]
> sde1[2] sdd1[1] sdc1[0]
> 1318680576 blocks level 5, 1024k chunk, algorithm 2 [10/10]
> [UUUUUUUUUU]
>
> md0 : active raid1 sdb1[1] sda1[0]
> 16787776 blocks [2/2] [UU]
>
> unused devices:
> #
>
> Justin.
>
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.23.1: mdadm/raid5 hung/d-state
am 04.11.2007 13:48:12 von Michael Tokarev
Justin Piszcz wrote:
> # ps auxww | grep D
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 273 0.0 0.0 0 0 ? D Oct21 14:40 [pdflush]
> root 274 0.0 0.0 0 0 ? D Oct21 13:00 [pdflush]
>
> After several days/weeks, this is the second time this has happened,
> while doing regular file I/O (decompressing a file), everything on the
> device went into D-state.
The next time you come across something like that, do a SysRq-T dump and
post that. It shows a stack trace of all processes - and in particular,
where exactly each task is stuck.
/mjt
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: 2.6.23.1: mdadm/raid5 hung/d-state
am 04.11.2007 13:52:00 von Justin Piszcz
On Sun, 4 Nov 2007, Michael Tokarev wrote:
> Justin Piszcz wrote:
>> # ps auxww | grep D
>> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>> root 273 0.0 0.0 0 0 ? D Oct21 14:40 [pdflush]
>> root 274 0.0 0.0 0 0 ? D Oct21 13:00 [pdflush]
>>
>> After several days/weeks, this is the second time this has happened,
>> while doing regular file I/O (decompressing a file), everything on the
>> device went into D-state.
>
> The next time you come across something like that, do a SysRq-T dump and
> post that. It shows a stack trace of all processes - and in particular,
> where exactly each task is stuck.
>
> /mjt
>
Yes I got it before I rebooted, ran that and then dmesg > file.
Here it is:
[1172609.665902] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172609.668768] ffffffff80747dc0 ffff81015c3aa918 ffff810091c899b4 ffff810091c899a8
[1172609.668871] Call Trace:
[1172609.674472] [] schedule_timeout+0x5f/0xd0
[1172609.677362] [] process_timeout+0x0/0x10
[1172609.680243] [] do_select+0x468/0x560
[1172609.683105] [] __pollwait+0x0/0x130
[1172609.685969] [] default_wake_function+0x0/0x10
[1172609.688851] [] default_wake_function+0x0/0x10
[1172609.691712] [] default_wake_function+0x0/0x10
[1172609.694534] [] default_wake_function+0x0/0x10
[1172609.697324] [] skb_copy_datagram_iovec+0x1a1/0x260
[1172609.700103] [] _spin_lock_bh+0x9/0x20
[1172609.702856] [] release_sock+0x13/0xb0
[1172609.705598] [] tcp_recvmsg+0x370/0x940
[1172609.708303] [] sock_common_recvmsg+0x30/0x50
[1172609.710999] [] sock_aio_read+0x11b/0x130
[1172609.713694] [] core_sys_select+0x209/0x300
[1172609.716397] [] autoremove_wake_function+0x0/0x30
[1172609.719112] [] default_wake_function+0x0/0x10
[1172609.721824] [] current_fs_time+0x1e/0x30
[1172609.724525] [] tty_ldisc_deref+0x52/0x80
[1172609.727215] [] sys_select+0xd1/0x1c0
[1172609.729880] [] system_call+0x7e/0x83
[1172609.732517]
[1172609.735115] bash S 0000000000000000 0 30959 30958
[1172609.737742] ffff810091c8be88 0000000000000086 0000000000000000 ffff8101ea172e20
[1172609.740404] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172609.743087] ffffffff80747dc0 ffff81015c3ab028 ffff810091c8be54 ffff810091c8be48
[1172609.743190] Call Trace:
[1172609.748404] [] do_wait+0x599/0xc90
[1172609.751071] [] __wake_up+0x43/0x70
[1172609.753714] [] default_wake_function+0x0/0x10
[1172609.756345] [] system_call+0x7e/0x83
[1172609.758967]
[1172609.761522] sr S 0000000000000000 0 30966 30959
[1172609.764123] ffff810122d7de88 0000000000000082 0000000000000000 ffff8101eab3ee20
[1172609.766769] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172609.769442] ffffffff80747dc0 ffff8101ea173028 ffff810122d7de54 ffff810122d7de48
[1172609.769545] Call Trace:
[1172609.774734] [] do_wait+0x599/0xc90
[1172609.777369] [] default_wake_function+0x0/0x10
[1172609.779999] [] system_call+0x7e/0x83
[1172609.782616]
[1172609.785168] screen S 0000000000000000 0 30972 30966
[1172609.787768] ffff810144597f68 0000000000000086 ffff810144597f30 00000000ffffffff
[1172609.790416] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172609.793085] ffffffff80747dc0 ffff8101eab3f028 ffff810144597f34 ffff810144597f28
[1172609.793188] Call Trace:
[1172609.798381] [] alarm_setitimer+0x35/0x70
[1172609.801049] [] sys_pause+0x19/0x30
[1172609.803705] [] system_call+0x7e/0x83
[1172609.806361]
[1172609.808980] sshd S 0000000000000000 0 30973 7582
[1172609.811659] ffff810084003bf8 0000000000000082 0000000000000000 ffffffff80508e74
[1172609.814376] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172609.817104] ffffffff80747dc0 ffff8101ea172208 ffff810084003bc4 ffff810084003bb8
[1172609.817207] Call Trace:
[1172609.822530] [] skb_queue_tail+0x24/0x60
[1172609.825292] [] schedule_timeout+0x95/0xd0
[1172609.828060] [] prepare_to_wait+0x23/0x80
[1172609.830820] [] unix_stream_recvmsg+0x386/0x550
[1172609.833587] [] autoremove_wake_function+0x0/0x30
[1172609.836344] [] link_path_walk+0x80/0xf0
[1172609.839074] [] sock_aio_read+0x11b/0x130
[1172609.841794] [] get_unused_fd_flags+0x79/0x120
[1172609.844488] [] do_sync_read+0xd9/0x120
[1172609.847161] [] autoremove_wake_function+0x0/0x30
[1172609.849848] [] __dentry_open+0x11f/0x1b0
[1172609.852541] [] do_filp_open+0x3a/0x50
[1172609.855235] [] vfs_read+0x157/0x160
[1172609.857922] [] sys_read+0x53/0x90
[1172609.860620] [] system_call+0x7e/0x83
[1172609.863343]
[1172609.866063] sshd S 0000000000000000 0 30975 30973
[1172609.868838] ffff810175c219e8 0000000000000086 ffff810175c219b0 0000000000000002
[1172609.871649] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172609.874490] ffffffff80747dc0 ffff81021b27d738 ffff810175c219b4 ffff810175c219a8
[1172609.874594] Call Trace:
[1172609.880153] [] schedule_timeout+0x5f/0xd0
[1172609.883020] [] process_timeout+0x0/0x10
[1172609.885890] [] do_select+0x468/0x560
[1172609.888742] [] __pollwait+0x0/0x130
[1172609.891581] [] default_wake_function+0x0/0x10
[1172609.894430] [] default_wake_function+0x0/0x10
[1172609.897258] [] default_wake_function+0x0/0x10
[1172609.900060] [] default_wake_function+0x0/0x10
[1172609.902841] [] add_partial+0x19/0x60
[1172609.905606] [] __slab_free+0x15d/0x310
[1172609.908363] [] _spin_lock_bh+0x9/0x20
[1172609.911093] [] release_sock+0x13/0xb0
[1172609.913795] [] tcp_recvmsg+0x370/0x940
[1172609.916486] [] sock_common_recvmsg+0x30/0x50
[1172609.919151] [] sock_aio_read+0x11b/0x130
[1172609.921799] [] core_sys_select+0x209/0x300
[1172609.924455] [] autoremove_wake_function+0x0/0x30
[1172609.927122] [] default_wake_function+0x0/0x10
[1172609.929786] [] current_fs_time+0x1e/0x30
[1172609.932438] [] tty_ldisc_deref+0x52/0x80
[1172609.935083] [] sys_select+0xd1/0x1c0
[1172609.937702] [] system_call+0x7e/0x83
[1172609.940292]
[1172609.942843] bash S 0000000000000000 0 30976 30975
[1172609.945423] ffff8101bf371e88 0000000000000082 0000000000000000 ffff81021e322710
[1172609.948037] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172609.950671] ffffffff80747dc0 ffff8101882bf738 ffff8101bf371e54 ffff8101bf371e48
[1172609.950774] Call Trace:
[1172609.955888] [] do_wait+0x599/0xc90
[1172609.958505] [] __wake_up+0x43/0x70
[1172609.961098] [] vfs_ioctl+0x220/0x2c0
[1172609.963662] [] default_wake_function+0x0/0x10
[1172609.966234] [] sys_ioctl+0x49/0x80
[1172609.968766] [] system_call+0x7e/0x83
[1172609.971279]
[1172609.973759] screen S 0000000000000000 0 30991 30976
[1172609.976308] ffff8101a8329f68 0000000000000086 0000000000000000 00000000ffffffff
[1172609.978892] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172609.981501] ffffffff80747dc0 ffff81021e322918 ffff8101a8329f34 ffff8101a8329f28
[1172609.981605] Call Trace:
[1172609.986634] [] alarm_setitimer+0x35/0x70
[1172609.989220] [] sys_pause+0x19/0x30
[1172609.991766] [] system_call+0x7e/0x83
[1172609.994292]
[1172609.996787] screen D ffff8100a18ff800 0 30992 30991
[1172609.999344] ffff8101a854dd28 0000000000000086 ffff81022854ddb7 ffff8101a854dcd8
[1172610.001953] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.004574] ffffffff80747dc0 ffff810170233028 ffffffff80656bcb ffffffff8021f8bc
[1172610.004677] Call Trace:
[1172610.009752] [] task_rq_lock+0x4c/0x90
[1172610.012366] [] try_to_wake_up+0x68/0x3b0
[1172610.014981] [] wait_for_completion+0x7d/0xc0
[1172610.017594] [] default_wake_function+0x0/0x10
[1172610.020208] [] flush_cpu_workqueue+0x6a/0x90
[1172610.022828] [] wq_barrier_func+0x0/0x10
[1172610.025447] [] flush_workqueue+0x33/0x50
[1172610.028076] [] release_dev+0x44f/0x750
[1172610.030710] [] mntput_no_expire+0x27/0xb0
[1172610.033339] [] tty_release+0x11/0x20
[1172610.035958] [] __fput+0xb1/0x1a0
[1172610.038547] [] filp_close+0x54/0x90
[1172610.041106] [] sys_close+0x96/0x100
[1172610.043652] [] system_call+0x7e/0x83
[1172610.046160]
[1172610.048618] bash ? 0000000000000000 0 30993 30992
[1172610.051135] ffff8101aa2a3ee8 0000000000000046 ffff8101aa2a3eb0 0000000000000011
[1172610.053708] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.056312] ffffffff80747dc0 ffff810170233738 ffff8101aa2a3eb4 ffff8101aa2a3ea8
[1172610.056415] Call Trace:
[1172610.061510] [] do_exit+0x5be/0x8a0
[1172610.064172] [] do_group_exit+0x2c/0x80
[1172610.066859] [] system_call+0x7e/0x83
[1172610.069537]
[1172610.072190] sshd S 0000000000000000 0 7001 7582
[1172610.074908] ffff8100792b1bf8 0000000000000082 0000000000000000 ffff8101e9c51b80
[1172610.077679] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.080477] ffffffff80747dc0 ffff8102234ff738 ffff8100792b1bc4 ffff8100792b1bb8
[1172610.080580] Call Trace:
[1172610.086042] [] schedule_timeout+0x95/0xd0
[1172610.088861] [] prepare_to_wait+0x23/0x80
[1172610.091673] [] unix_stream_recvmsg+0x386/0x550
[1172610.094492] [] autoremove_wake_function+0x0/0x30
[1172610.097318] [] link_path_walk+0x80/0xf0
[1172610.100148] [] sock_aio_read+0x11b/0x130
[1172610.102976] [] get_unused_fd_flags+0x79/0x120
[1172610.105822] [] do_sync_read+0xd9/0x120
[1172610.108651] [] autoremove_wake_function+0x0/0x30
[1172610.111495] [] __dentry_open+0x11f/0x1b0
[1172610.114319] [] do_filp_open+0x3a/0x50
[1172610.117118] [] vfs_read+0x157/0x160
[1172610.119902] [] sys_read+0x53/0x90
[1172610.122638] [] system_call+0x7e/0x83
[1172610.125360]
[1172610.128056] sshd S 0000000000000000 0 7003 7001
[1172610.130818] ffff8100675a39e8 0000000000000082 ffff8100675a39b0 0000000000000002
[1172610.133623] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.136446] ffffffff80747dc0 ffff810225459028 ffff8100675a39b4 ffff8100675a39a8
[1172610.136549] Call Trace:
[1172610.142064] [] schedule_timeout+0x5f/0xd0
[1172610.144899] [] process_timeout+0x0/0x10
[1172610.147716] [] do_select+0x468/0x560
[1172610.150495] [] __pollwait+0x0/0x130
[1172610.153260] [] default_wake_function+0x0/0x10
[1172610.156005] [] default_wake_function+0x0/0x10
[1172610.158707] [] default_wake_function+0x0/0x10
[1172610.161378] [] default_wake_function+0x0/0x10
[1172610.164026] [] skb_copy_datagram_iovec+0x1a1/0x260
[1172610.166675] [] _spin_lock_bh+0x9/0x20
[1172610.169315] [] release_sock+0x13/0xb0
[1172610.171917] [] tcp_recvmsg+0x370/0x940
[1172610.174494] [] sock_common_recvmsg+0x30/0x50
[1172610.177085] [] sock_aio_read+0x11b/0x130
[1172610.179638] [] core_sys_select+0x209/0x300
[1172610.182178] [] autoremove_wake_function+0x0/0x30
[1172610.184734] [] default_wake_function+0x0/0x10
[1172610.187290] [] current_fs_time+0x1e/0x30
[1172610.189837] [] tty_ldisc_deref+0x52/0x80
[1172610.192370] [] sys_select+0xd1/0x1c0
[1172610.194900] [] system_call+0x7e/0x83
[1172610.197426]
[1172610.199919] bash S 000000000000000e 0 7004 7003
[1172610.202470] ffff8100cc263e88 0000000000000082 80000000804ca065 ffff81022367f530
[1172610.205071] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.207699] ffffffff80747dc0 ffff8102234fe918 ffff8100cc263e38 ffff810035a16348
[1172610.207802] Call Trace:
[1172610.212949] [] do_page_fault+0x202/0x890
[1172610.215618] [] do_wait+0x599/0xc90
[1172610.218263] [] __wake_up+0x43/0x70
[1172610.220900] [] vfs_ioctl+0x220/0x2c0
[1172610.223509] [] default_wake_function+0x0/0x10
[1172610.226109] [] sys_ioctl+0x49/0x80
[1172610.228693] [] system_call+0x7e/0x83
[1172610.231240]
[1172610.233746] aur S 0000000000000000 0 7014 7004
[1172610.236319] ffff810098071e88 0000000000000086 ffff810098071e50 ffffffff80232c93
[1172610.238941] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.241566] ffffffff80747dc0 ffff81022367f738 ffff810098071e54 ffff810098071e48
[1172610.241669] Call Trace:
[1172610.246766] [] get_signal_to_deliver+0x73/0x470
[1172610.249380] [] do_wait+0x599/0xc90
[1172610.251983] [] default_wake_function+0x0/0x10
[1172610.254563] [] system_call+0x7e/0x83
[1172610.257122]
[1172610.259648] aur S 0000000000000004 0 7066 7014
[1172610.262226] ffff810085231e88 0000000000000086 ffff8101ea314ce8 ffffffff80232c93
[1172610.264844] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.267471] ffffffff80747dc0 ffff8101ea314918 ffffffff802302ce ffffffff8020b3d6
[1172610.267574] Call Trace:
[1172610.272674] [] get_signal_to_deliver+0x73/0x470
[1172610.275315] [] recalc_sigpending+0xe/0x30
[1172610.277948] [] do_notify_resume+0x536/0x7a0
[1172610.280577] [] do_wait+0x599/0xc90
[1172610.283199] [] default_wake_function+0x0/0x10
[1172610.285840] [] system_call+0x7e/0x83
[1172610.288491]
[1172610.291116] unrar D ffff8100aa785c80 0 7135 7066
[1172610.293792] ffff8101ecf4ddb8 0000000000000086 ffff8101ecf4dd80 0000000000000000
[1172610.296525] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.299256] ffffffff80747dc0 ffff81021e53f028 ffff8101ecf4dd84 ffff8101ecf4dd78
[1172610.299359] Call Trace:
[1172610.304629] [] vn_iowait+0x75/0xa0
[1172610.307301] [] autoremove_wake_function+0x0/0x30
[1172610.309979] [] xfs_trans_alloc+0x9c/0xb0
[1172610.312653] [] xfs_itruncate_start+0x35/0xe0
[1172610.315340] [] xfs_free_eofblocks+0x17a/0x280
[1172610.318032] [] xfs_release+0x134/0x1e0
[1172610.320711] [] xfs_file_release+0x1a/0x30
[1172610.323417] [] __fput+0xb1/0x1a0
[1172610.326144] [] filp_close+0x54/0x90
[1172610.328895] [] sys_close+0x96/0x100
[1172610.331631] [] system_call+0x7e/0x83
[1172610.334353]
[1172610.337050] sshd D 0000000000000000 0 7187 7582
[1172610.339811] ffff81002b62fd28 0000000000000086 ffff81002b62fcf0 ffff81002b62fcd8
[1172610.342618] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.345448] ffffffff80747dc0 ffff8101ccda3028 ffff81002b62fcf4 ffff81002b62fce8
[1172610.345551] Call Trace:
[1172610.351072] [] wait_for_completion+0x7d/0xc0
[1172610.353915] [] default_wake_function+0x0/0x10
[1172610.356765] [] flush_cpu_workqueue+0x6a/0x90
[1172610.359622] [] wq_barrier_func+0x0/0x10
[1172610.362477] [] flush_workqueue+0x33/0x50
[1172610.365337] [] release_dev+0x44f/0x750
[1172610.368184] [] sys_fchmodat+0x6a/0x120
[1172610.371026] [] tty_release+0x11/0x20
[1172610.373843] [] __fput+0xb1/0x1a0
[1172610.376628] [] filp_close+0x54/0x90
[1172610.379402] [] sys_close+0x96/0x100
[1172610.382135] [] system_call+0x7e/0x83
[1172610.384846]
[1172610.387529] sshd ? 0000000000000000 0 7218 7187
[1172610.390280] ffff81013bd7bee8 0000000000000046 ffff81013bd7beb0 0000000000000011
[1172610.393084] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.395907] ffffffff80747dc0 ffff8101bf6ae918 ffff81013bd7beb4 ffff81013bd7bea8
[1172610.396010] Call Trace:
[1172610.401528] [] __cond_resched+0x1c/0x50
[1172610.404362] [] do_exit+0x5be/0x8a0
[1172610.407192] [] do_group_exit+0x2c/0x80
[1172610.409993] [] system_call+0x7e/0x83
[1172610.412776]
[1172610.415520] sshd S 0000000000000000 0 7236 7582
[1172610.418293] ffff8101e4a89bf8 0000000000000082 ffff8101e4a89bc0 ffff81013bf542c0
[1172610.421090] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.423898] ffffffff80747dc0 ffff810168684208 ffff8101e4a89bc4 ffff8101e4a89bb8
[1172610.424001] Call Trace:
[1172610.429474] [] schedule_timeout+0x95/0xd0
[1172610.432284] [] prepare_to_wait+0x23/0x80
[1172610.435096] [] unix_stream_recvmsg+0x386/0x550
[1172610.437896] [] autoremove_wake_function+0x0/0x30
[1172610.440690] [] link_path_walk+0x80/0xf0
[1172610.443487] [] sock_aio_read+0x11b/0x130
[1172610.446249] [] get_unused_fd_flags+0x79/0x120
[1172610.448997] [] do_sync_read+0xd9/0x120
[1172610.451737] [] autoremove_wake_function+0x0/0x30
[1172610.454491] [] __dentry_open+0x11f/0x1b0
[1172610.457244] [] do_filp_open+0x3a/0x50
[1172610.459989] [] vfs_read+0x157/0x160
[1172610.462724] [] sys_read+0x53/0x90
[1172610.465430] [] system_call+0x7e/0x83
[1172610.468131]
[1172610.470765] sshd S 0000000000000000 0 7238 7236
[1172610.473440] ffff810046e1f9e8 0000000000000082 ffff810046e1f9b0 0000000000000002
[1172610.476161] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.478873] ffffffff80747dc0 ffff810168685028 ffff810046e1f9b4 ffff810046e1f9a8
[1172610.478975] Call Trace:
[1172610.484236] [] schedule_timeout+0x5f/0xd0
[1172610.486940] [] process_timeout+0x0/0x10
[1172610.489645] [] do_select+0x468/0x560
[1172610.492340] [] __pollwait+0x0/0x130
[1172610.495030] [] default_wake_function+0x0/0x10
[1172610.497738] [] default_wake_function+0x0/0x10
[1172610.500417] [] default_wake_function+0x0/0x10
[1172610.503076] [] default_wake_function+0x0/0x10
[1172610.505711] [] skb_copy_datagram_iovec+0x1a1/0x260
[1172610.508366] [] _spin_lock_bh+0x9/0x20
[1172610.511004] [] release_sock+0x13/0xb0
[1172610.513638] [] tcp_recvmsg+0x370/0x940
[1172610.516245] [] sock_common_recvmsg+0x30/0x50
[1172610.518841] [] sock_aio_read+0x11b/0x130
[1172610.521423] [] core_sys_select+0x209/0x300
[1172610.523974] [] autoremove_wake_function+0x0/0x30
[1172610.526518] [] default_wake_function+0x0/0x10
[1172610.529058] [] current_fs_time+0x1e/0x30
[1172610.531592] [] tty_ldisc_deref+0x52/0x80
[1172610.534118] [] sys_select+0xd1/0x1c0
[1172610.536645] [] system_call+0x7e/0x83
[1172610.539162]
[1172610.541651] bash S 000000000000000e 0 7239 7238
[1172610.544203] ffff8100aae5fe88 0000000000000082 80000001bab2c065 ffff810145b6ae20
[1172610.546809] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.549446] ffffffff80747dc0 ffff810168685738 ffff8100aae5fe38 ffff810065785f18
[1172610.549550] Call Trace:
[1172610.554709] [] do_page_fault+0x202/0x890
[1172610.557368] [] update_curr+0x109/0x120
[1172610.560022] [] do_wait+0x599/0xc90
[1172610.562647] [] __sched_text_start+0x166/0x23d
[1172610.565267] [] __wake_up+0x43/0x70
[1172610.567871] [] vfs_ioctl+0x220/0x2c0
[1172610.570435] [] default_wake_function+0x0/0x10
[1172610.572998] [] sys_ioctl+0x49/0x80
[1172610.575555] [] system_call+0x7e/0x83
[1172610.578118]
[1172610.580652] sshd S 0000000000000000 0 7248 7582
[1172610.583235] ffff8101120d5bf8 0000000000000082 ffff8101120d5bc0 ffff81001e998dc0
[1172610.585865] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.588489] ffffffff80747dc0 ffff810130906208 ffff8101120d5bc4 ffff8101120d5bb8
[1172610.588592] Call Trace:
[1172610.593666] [] schedule_timeout+0x95/0xd0
[1172610.596253] [] prepare_to_wait+0x23/0x80
[1172610.598824] [] unix_stream_recvmsg+0x386/0x550
[1172610.601405] [] autoremove_wake_function+0x0/0x30
[1172610.603992] [] link_path_walk+0x80/0xf0
[1172610.606571] [] sock_aio_read+0x11b/0x130
[1172610.609138] [] get_unused_fd_flags+0x79/0x120
[1172610.611720] [] do_sync_read+0xd9/0x120
[1172610.614293] [] autoremove_wake_function+0x0/0x30
[1172610.616883] [] __dentry_open+0x11f/0x1b0
[1172610.619463] [] do_filp_open+0x3a/0x50
[1172610.622029] [] vfs_read+0x157/0x160
[1172610.624594] [] sys_read+0x53/0x90
[1172610.627144] [] system_call+0x7e/0x83
[1172610.629703]
[1172610.632237] sshd S 0000000000000000 0 7250 7248
[1172610.634822] ffff810126f3d9e8 0000000000000086 ffff810126f3d9b0 0000000000000002
[1172610.637453] ffffffff80747dc0 ffffffff80747dc0 ffffffff80747dc0 ffffffff80744d80
[1172610.640086] ffffffff80747dc0 ffff810130907028 ffff810126f3d9b4 ffff810126f3d9a8
[1172610.640190] Call Trace:
[1172610.645268] [] schedule_timeout+0x5f/0xd0
[1172610.647857] [] process_timeout+0x0/0x10
[1172610.650429] [] do_select+0x468/0x560
[1172610.652990] [] __pollwait+0x0/0x130
[1172610.655552] [] default_wake_function+0x0/0x10
[1172610.658131] [] default_wake_function+0x0/0x10
[1172610.660680] [] default_wake_function+0x0/0x10
[1172610.663230] [] default_wake_function+0x0/0x10
[1172610.665779] [] skb_copy_datagram_iovec+0x1a1/0x260
[1172610.668368] [] _spin_lock_bh+0x9/0x20
[1172610.670946] [] release_sock+0x13/0xb0
[1172610.673510] [] tcp_recvmsg+0x370/0x940
[1172610.676075] [] sock_common_recvmsg+0x30/0x50
[1172610.678653] [] sock_aio_read+0x11b/0x130
[1172610.681222] [] core_sys_select+0x209/0x300
[1172610.683798] [] autoremove_wake_function+0x0/0x30
[1172610.686386] [] default_wake_function+0x0/0x10
[1172610.688970] [] current_fs_time+0x1e/0x30
[1172610.691546] [] tty_ldisc_deref+0x52/0x80
[1172610.694114] [] sys_select+0xd1/0x1c0