possible bug - bitmap dirty pages status

Hello!
I have raid1 array with bitmap (md3), one disk has died, been replaced
and array resynced:

Aug 25 15:38:03 gamma2 kernel: [ 5.986791] md: md3 stopped.
Aug 25 15:38:03 gamma2 kernel: [ 6.043306] raid1: raid set md3
active with 1 out of 2 mirrors
Aug 25 15:38:03 gamma2 kernel: [ 6.044378] md3: bitmap initialized
from disk: read 2/2 pages, set 357 bits
Aug 25 15:38:03 gamma2 kernel: [ 6.044442] created bitmap (22
pages) for device md3
Aug 25 15:38:03 gamma2 kernel: [ 6.070492] md3: detected capacity
change from 0 to 1478197903360
Aug 25 15:38:03 gamma2 kernel: [ 6.070862] md3: unknown partition table
Aug 26 19:33:33 gamma2 kernel: [100325.814179] md: md3: recovery done.

Now, /proc/mdstats still shows "dirty" bitmap, 16 of 22 pages:

root@gamma2:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sdc3[1] sdb3[0]
1443552640 blocks [2/2] [UU]
bitmap: 16/22 pages [64KB], 32768KB chunk

root@gamma2:~# mdadm -D /dev/md3
/dev/md3:
Version : 00.90
Creation Time : Wed Oct 13 03:13:52 2010
Raid Level : raid1
Array Size : 1443552640 (1376.68 GiB 1478.20 GB)
Used Dev Size : 1443552640 (1376.68 GiB 1478.20 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 3
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Sat Aug 27 13:53:57 2011
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

UUID : f5a9c1da:83dd4c40:d363f6aa:3cbcebe5
Events : 0.1381014

Number Major Minor RaidDevice State
0 8 19 0 active sync /dev/sdb3
1 8 35 1 active sync /dev/sdc3



That is strange as I can understand.

distrib & mdadm info:
Debian Lenny with openvz kernel from lenny-backports.

root@gamma2:~# cat /proc/version
Linux version 2.6.32-bpo.5-openvz-amd64 (Debian 2.6.32-35~bpo50+1)
(norbert@tretkowski.de) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SMP
Wed Jul 20 11:23:01 UTC 2011
root@gamma2:~# mdadm --version
mdadm - v2.6.7.2 - 14th November 2008




--
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: possible bug - bitmap dirty pages status

On Sat, Aug 27, 2011 at 1:58 PM, CoolCold wrote=
:
> Hello!
> I have raid1 array with bitmap (md3), one disk has died, been replace=
d
> and array resynced:
>
> Aug 25 15:38:03 gamma2 kernel: [ =A0 =A05.986791] md: md3 stopped.
> Aug 25 15:38:03 gamma2 kernel: [ =A0 =A06.043306] raid1: raid set md3
> active with 1 out of 2 mirrors
> Aug 25 15:38:03 gamma2 kernel: [ =A0 =A06.044378] md3: bitmap initial=
ized
> from disk: read 2/2 pages, set 357 bits
> Aug 25 15:38:03 gamma2 kernel: [ =A0 =A06.044442] created bitmap (22
> pages) for device md3
> Aug 25 15:38:03 gamma2 kernel: [ =A0 =A06.070492] md3: detected capac=
ity
> change from 0 to 1478197903360
> Aug 25 15:38:03 gamma2 kernel: [ =A0 =A06.070862] =A0md3: unknown par=
tition table
> Aug 26 19:33:33 gamma2 kernel: [100325.814179] md: md3: recovery done=

>
> Now, /proc/mdstats still shows "dirty" bitmap, 16 of 22 pages:
>
> root@gamma2:~# cat /proc/mdstat
> Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> md3 : active raid1 sdc3[1] sdb3[0]
> =A0 =A0 =A01443552640 blocks [2/2] [UU]
> =A0 =A0 =A0bitmap: 16/22 pages [64KB], 32768KB chunk
>
> root@gamma2:~# mdadm -D /dev/md3
> /dev/md3:
> =A0 =A0 =A0 =A0Version : 00.90
> =A0Creation Time : Wed Oct 13 03:13:52 2010
> =A0 =A0 Raid Level : raid1
> =A0 =A0 Array Size : 1443552640 (1376.68 GiB 1478.20 GB)
> =A0Used Dev Size : 1443552640 (1376.68 GiB 1478.20 GB)
> =A0 Raid Devices : 2
> =A0Total Devices : 2
> Preferred Minor : 3
> =A0 =A0Persistence : Superblock is persistent
>
> =A0Intent Bitmap : Internal
>
> =A0 =A0Update Time : Sat Aug 27 13:53:57 2011
> =A0 =A0 =A0 =A0 =A0State : active
> =A0Active Devices : 2
> Working Devices : 2
> =A0Failed Devices : 0
> =A0Spare Devices : 0
>
> =A0 =A0 =A0 =A0 =A0 UUID : f5a9c1da:83dd4c40:d363f6aa:3cbcebe5
> =A0 =A0 =A0 =A0 Events : 0.1381014
>
> =A0 =A0Number =A0 Major =A0 Minor =A0 RaidDevice State
> =A0 =A0 =A0 0 =A0 =A0 =A0 8 =A0 =A0 =A0 19 =A0 =A0 =A0 =A00 =A0 =A0 =A0=
active sync =A0 /dev/sdb3
> =A0 =A0 =A0 1 =A0 =A0 =A0 8 =A0 =A0 =A0 35 =A0 =A0 =A0 =A01 =A0 =A0 =A0=
active sync =A0 /dev/sdc3
>
>
>
> That is strange as I can understand.
>
> distrib & mdadm info:
> Debian Lenny with openvz kernel from lenny-backports.
>
> root@gamma2:~# cat /proc/version
> Linux version 2.6.32-bpo.5-openvz-amd64 (Debian 2.6.32-35~bpo50+1)
> (norbert@tretkowski.de) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SM=
P
> Wed Jul 20 11:23:01 UTC 2011
> root@gamma2:~# mdadm --version
> mdadm - v2.6.7.2 - 14th November 2008
>
More details now:
Array still has 16/22 dirty pages:

root@gamma2:~# date;cat /proc/mdstat |head -n 5
Wed Aug 31 13:04:26 MSD 2011
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md3 : active raid1 sdc3[1] sdb3[0]
1443552640 blocks [2/2] [UU]
bitmap: 16/22 pages [64KB], 32768KB chunk


bitmap info:

root@gamma2:~# mdadm --examine-bitmap /dev/sdb3
Filename : /dev/sdb3
Magic : 6d746962
Version : 4
UUID : f5a9c1da:83dd4c40:d363f6aa:3cbcebe5
Events : 1381014
Events Cleared : 1381014
State : OK
Chunksize : 32 MB
Daemon : 5s flush period
Write Mode : Normal
Sync Size : 1443552640 (1376.68 GiB 1478.20 GB)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
root@gamma2:~# mdadm --examine-bitmap /dev/sdc3
Filename : /dev/sdc3
Magic : 6d746962
Version : 4
UUID : f5a9c1da:83dd4c40:d363f6aa:3cbcebe5
Events : 1381014
Events Cleared : 1381014
State : OK
Chunksize : 32 MB
Daemon : 5s flush period
Write Mode : Normal
Sync Size : 1443552640 (1376.68 GiB 1478.20 GB)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)


>
> --
> Best regards,
> [COOLCOLD-RIPN]
>



--=20
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: possible bug - bitmap dirty pages status

On Wed, Aug 31, 2011 at 5:05 AM, CoolCold wrote=
:

>> root@gamma2:~# cat /proc/mdstat
>> Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
>> md3 : active raid1 sdc3[1] sdb3[0]
>> =A0 =A0 =A01443552640 blocks [2/2] [UU]
>> =A0 =A0 =A0bitmap: 16/22 pages [64KB], 32768KB chunk

> More details now:
> Array still has 16/22 dirty pages:

> root@gamma2:~# mdadm --examine-bitmap /dev/sdc3

> =A0 =A0 =A0 Sync Size : 1443552640 (1376.68 GiB 1478.20 GB)
> =A0 =A0 =A0 =A0 =A0Bitmap : 44054 bits (chunks), 189 dirty (0.4%)

But only 189 bits dirty. This means the bits are just distributed
across the disk (which is why you have 16/22 pages dirty).

Any activity on the disk? 189 bits could easily be explained by a
small amount of background disk activity on a disk that big.

--
Paul
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: possible bug - bitmap dirty pages status

On 31/08/2011 13:30, Paul Clements wrote:
> On Wed, Aug 31, 2011 at 5:05 AM, CoolCold wrote:
>
>>> root@gamma2:~# cat /proc/mdstat
>>> Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
>>> md3 : active raid1 sdc3[1] sdb3[0]
>>> 1443552640 blocks [2/2] [UU]
>>> bitmap: 16/22 pages [64KB], 32768KB chunk
>
>> More details now:
>> Array still has 16/22 dirty pages:
>
>> root@gamma2:~# mdadm --examine-bitmap /dev/sdc3
>
>> Sync Size : 1443552640 (1376.68 GiB 1478.20 GB)
>> Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
>
> But only 189 bits dirty. This means the bits are just distributed
> across the disk (which is why you have 16/22 pages dirty).
>
> Any activity on the disk? 189 bits could easily be explained by a
> small amount of background disk activity on a disk that big.

That makes sense to me. I have:

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active raid6 sde2[4] sdd2[3] sdc2[2] sdb2[1] sda2[0]
2929966080 blocks level 6, 512k chunk, algorithm 2 [5/5] [UUUUU]
bitmap: 2/4 pages [8KB], 131072KB chunk

Oh no! Half my array is dirty! But then:

# mdadm --examine-bitmap /dev/sdb2
Filename : /dev/sdb2
Magic : 6d746962
Version : 4
UUID : d8c57a89:166ee722:23adec48:1574b5fc
Events : 1338800
Events Cleared : 1338800
State : OK
Chunksize : 128 MB
Daemon : 5s flush period
Write Mode : Normal
Sync Size : 976655360 (931.41 GiB 1000.10 GB)
Bitmap : 7452 bits (chunks), 7 dirty (0.1%)

Not so bad after all.

On the other hand, repeatedly checking /proc/mdstat shows different
numbers of pages being dirty, and --examine-bitmap shows different
numbers of bits being dirty each time, whereas CoolCold managed 16 pages
repeatedly and 189 bits being dirty twice in a row. CoolCold, please can
you test --examine-bitmap again several times at least 5 seconds apart?

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: possible bug - bitmap dirty pages status

On Wed, Aug 31, 2011 at 4:56 PM, John Robinson
wrote:
> On 31/08/2011 13:30, Paul Clements wrote:
>>
>> On Wed, Aug 31, 2011 at 5:05 AM, CoolCold =A0=
wrote:
>>
>>>> root@gamma2:~# cat /proc/mdstat
>>>> Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
>>>> md3 : active raid1 sdc3[1] sdb3[0]
>>>> =A0 =A0 =A01443552640 blocks [2/2] [UU]
>>>> =A0 =A0 =A0bitmap: 16/22 pages [64KB], 32768KB chunk
>>
>>> More details now:
>>> Array still has 16/22 dirty pages:
>>
>>> root@gamma2:~# mdadm --examine-bitmap /dev/sdc3
>>
>>> =A0 =A0 =A0 Sync Size : 1443552640 (1376.68 GiB 1478.20 GB)
>>> =A0 =A0 =A0 =A0 =A0Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
>>
>> But only 189 bits dirty. This means the bits are just distributed
>> across the disk (which is why you have 16/22 pages dirty).
>>
>> Any activity on the disk? 189 bits could easily be explained by a
>> small amount of background disk activity on a disk that big.
>
> That makes sense to me. I have:
>
> $ cat /proc/mdstat
> Personalities : [raid6] [raid5] [raid4] [raid1]
> md1 : active raid6 sde2[4] sdd2[3] sdc2[2] sdb2[1] sda2[0]
> =A0 =A0 =A02929966080 blocks level 6, 512k chunk, algorithm 2 [5/5] [=
UUUUU]
> =A0 =A0 =A0bitmap: 2/4 pages [8KB], 131072KB chunk
>
> Oh no! Half my array is dirty! But then:
>
> # mdadm --examine-bitmap /dev/sdb2
> =A0 =A0 =A0 =A0Filename : /dev/sdb2
> =A0 =A0 =A0 =A0 =A0 Magic : 6d746962
> =A0 =A0 =A0 =A0 Version : 4
> =A0 =A0 =A0 =A0 =A0 =A0UUID : d8c57a89:166ee722:23adec48:1574b5fc
> =A0 =A0 =A0 =A0 =A0Events : 1338800
> =A0Events Cleared : 1338800
> =A0 =A0 =A0 =A0 =A0 State : OK
> =A0 =A0 =A0 Chunksize : 128 MB
> =A0 =A0 =A0 =A0 =A0Daemon : 5s flush period
> =A0 =A0 =A0Write Mode : Normal
> =A0 =A0 =A0 Sync Size : 976655360 (931.41 GiB 1000.10 GB)
> =A0 =A0 =A0 =A0 =A0Bitmap : 7452 bits (chunks), 7 dirty (0.1%)
>
> Not so bad after all.
Makes sense, but...

>
> On the other hand, repeatedly checking /proc/mdstat shows different n=
umbers
> of pages being dirty, and --examine-bitmap shows different numbers of=
bits
> being dirty each time, whereas CoolCold managed 16 pages repeatedly a=
nd 189
> bits being dirty twice in a row. CoolCold, please can you test
> --examine-bitmap again several times at least 5 seconds apart?

fast test with "sleep 5" reveals:
root@gamma2:~# for i in {1..20};do mdadm --examine-bitmap
/dev/sdc3|grep "Bitmap :";sleep 5;done
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
Bitmap : 44054 bits (chunks), 189 dirty (0.4%)

And 16/22 lasts for 4 days.

>
> Cheers,
>
> John.
>
>



--=20
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: possible bug - bitmap dirty pages status

On Wed, Aug 31, 2011 at 9:16 AM, CoolCold wrote=
:

> =A0 =A0 =A0 =A0 =A0Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
>
> And 16/22 lasts for 4 days.

So if you force another resync, does it change/clear up?

If you unmount/stop all activity does it change?

--
Paul
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: possible bug - bitmap dirty pages status

On Wed, Aug 31, 2011 at 6:08 PM, Paul Clements
wrote:
> On Wed, Aug 31, 2011 at 9:16 AM, CoolCold wro=
te:
>
>> =A0 =A0 =A0 =A0 =A0Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
>>
>> And 16/22 lasts for 4 days.
>
> So if you force another resync, does it change/clear up?
>
> If you unmount/stop all activity does it change?
Well, this server is in production now, may be i'll be able to do
array stop/start later..right now i've set "cat /proc/mdstat" every
minute, and bitmap examine every minute, will see later is it changing
or not.

>
> --
> Paul
>



--=20
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: possible bug - bitmap dirty pages status

On Thu, 1 Sep 2011 00:16:36 +0400 CoolCold wrot=
e:

> On Wed, Aug 31, 2011 at 6:08 PM, Paul Clements
> wrote:
> > On Wed, Aug 31, 2011 at 9:16 AM, CoolCold w=
rote:
> >
> >> =A0 =A0 =A0 =A0 =A0Bitmap : 44054 bits (chunks), 189 dirty (0.4%)
> >>
> >> And 16/22 lasts for 4 days.
> >
> > So if you force another resync, does it change/clear up?
> >
> > If you unmount/stop all activity does it change?
> Well, this server is in production now, may be i'll be able to do
> array stop/start later..right now i've set "cat /proc/mdstat" every
> minute, and bitmap examine every minute, will see later is it changin=
g
> or not.
>=20

I spent altogether too long staring at the code and I can see various t=
hings
that could be usefully tidied but but nothing that really explains what=
you
have.

If there was no write activity to the array at all I can just see how t=
hat
last bits to be set might not get cleared, but as soon as another write
happened all those old bits would get cleared pretty quickly. And it s=
eems
unlikely that there have been no writes for over 4 days (???).

I don't think having these bits here is harmful and it would be easy to=
get
rid of them by using "mdadm --grow" to remove and then re-add the bitma=
p,
but I wish I knew what caused it...

I clean up the little issues I found in mainline and hope there isn't a
larger problem luking behind all this..

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html