MD devnode still present after "remove" udev event, and mdadm reports"does not appear to be active"

Greetings everybody,

I issue
mdadm --stop /dev/md0
and I want to reliably determine that the MD devnode (/dev/md0) is gone.
So I look for the udev 'remove' event for that devnode.
However, in some cases even after I see the udev event, I issue
mdadm --detail /dev/md0
and I get:
mdadm: md device /dev/md0 does not appear to be active

According to Detail.c, this means that mdadm can successfully do
open("/dev/md0") and receive a valid fd.
But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives ENODEV
from the kernel.

Can somebody suggest an explanation for this behavior? Is there a
reliable way to know when a MD devnode is gone?

Thanks,
Alex.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas
wrote:

> Greetings everybody,
>
> I issue
> mdadm --stop /dev/md0
> and I want to reliably determine that the MD devnode (/dev/md0) is gone.
> So I look for the udev 'remove' event for that devnode.
> However, in some cases even after I see the udev event, I issue
> mdadm --detail /dev/md0
> and I get:
> mdadm: md device /dev/md0 does not appear to be active
>
> According to Detail.c, this means that mdadm can successfully do
> open("/dev/md0") and receive a valid fd.
> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives ENODEV
> from the kernel.
>
> Can somebody suggest an explanation for this behavior? Is there a
> reliable way to know when a MD devnode is gone?

run "udevadm settle" after stopping /dev/md0 is most likely to work.

I suspect that udev removes the node *after* you see the 'remove' event.
Sometimes so soon after that you don't see the lag - sometimes a bit later.

NeilBrown

>
> Thanks,
> Alex.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

Thanks, Neil.

Although according to udev documentation: "the udev events are sent
out after udev has finished its event processing, all rules have been
processed, and needed device nodes are created."

Also looking at udev-worker code of udevd, the
udev_monitor_send_device() call is done after all the rules have been
processed.

Nevertheless, I looked at udevadm_settle.c and did some equivalent of
that in my code, and it looks like the issue is resolved. Perhaps
there is something md-specific here?

Another thing, since you are reading this thread, I wanted to ask
whether you have any advice on the "RAID5: failing an active component
during spare rebuild - arrays hangs" thread I opened some time ago.
Since you were not answering, I assume there is nothing additional you
can advise about, correct? I apologize if this off-topic was
inappropriate.

Thanks for the help,
Alex.



On Tue, Aug 30, 2011 at 12:25 AM, NeilBrown wrote:
> On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas il.com>
> wrote:
>
>> Greetings everybody,
>>
>> I issue
>> mdadm --stop /dev/md0
>> and I want to reliably determine that the MD devnode (/dev/md0) is g=
one.
>> So I look for the udev 'remove' event for that devnode.
>> However, in some cases even after I see the udev event, I issue
>> mdadm --detail /dev/md0
>> and I get:
>> mdadm: md device /dev/md0 does not appear to be active
>>
>> According to Detail.c, this means that mdadm can successfully do
>> open("/dev/md0") and receive a valid fd.
>> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives ENODEV
>> from the kernel.
>>
>> Can somebody suggest an explanation for this behavior? Is there a
>> reliable way to know when a MD devnode is gone?
>
> run "udevadm settle" after stopping /dev/md0 =A0is most likely to wor=
k.
>
> I suspect that udev removes the node *after* you see the 'remove' eve=
nt.
> Sometimes so soon after that you don't see the lag - sometimes a bit =
later.
>
> NeilBrown
>
>>
>> Thanks,
>> =A0 Alex.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid=
" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

On Tue, 30 Aug 2011 18:18:11 +0300 Alexander Lyakas com>
wrote:

> Thanks, Neil.
>=20
> Although according to udev documentation: "the udev events are sent
> out after udev has finished its event processing, all rules have been
> processed, and needed device nodes are created."
>=20
> Also looking at udev-worker code of udevd, the
> udev_monitor_send_device() call is done after all the rules have been
> processed.
>=20
> Nevertheless, I looked at udevadm_settle.c and did some equivalent of
> that in my code, and it looks like the issue is resolved. Perhaps
> there is something md-specific here?

I cannot see how it would be md-specific. mdadm doesn't create or remo=
ve
devices when udev is active - it leaves all that to udev.
If you are curious I suggest you ask the udev developers.

>=20
> Another thing, since you are reading this thread, I wanted to ask
> whether you have any advice on the "RAID5: failing an active componen=
t
> during spare rebuild - arrays hangs" thread I opened some time ago.
> Since you were not answering, I assume there is nothing additional yo=
u
> can advise about, correct? I apologize if this off-topic was
> inappropriate.

It could mean that I had nothing extra to say, but it could also mean t=
hat I
got distracted, forgot, and never got back to it. I live in a world of
distractions :-(

But a reminder never hurts - it shows that it is important to you, so t=
hat
makes it at least a little bit important to me. I'll go back and have =
a look
and see if I have anything useful to add.

NeilBrown


>=20
> Thanks for the help,
> Alex.
>=20
>=20
>=20
> On Tue, Aug 30, 2011 at 12:25 AM, NeilBrown wrote:
> > On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas mail.com>
> > wrote:
> >
> >> Greetings everybody,
> >>
> >> I issue
> >> mdadm --stop /dev/md0
> >> and I want to reliably determine that the MD devnode (/dev/md0) is=
gone.
> >> So I look for the udev 'remove' event for that devnode.
> >> However, in some cases even after I see the udev event, I issue
> >> mdadm --detail /dev/md0
> >> and I get:
> >> mdadm: md device /dev/md0 does not appear to be active
> >>
> >> According to Detail.c, this means that mdadm can successfully do
> >> open("/dev/md0") and receive a valid fd.
> >> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives ENOD=
EV
> >> from the kernel.
> >>
> >> Can somebody suggest an explanation for this behavior? Is there a
> >> reliable way to know when a MD devnode is gone?
> >
> > run "udevadm settle" after stopping /dev/md0 =A0is most likely to w=
ork.
> >
> > I suspect that udev removes the node *after* you see the 'remove' e=
vent.
> > Sometimes so soon after that you don't see the lag - sometimes a bi=
t later.
> >
> > NeilBrown
> >
> >>
> >> Thanks,
> >> =A0 Alex.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-ra=
id" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at =A0http://vger.kernel.org/majordomo-info.ht=
ml
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

Thank you for looking at both issues, Neil.

Alex.


On Wed, Aug 31, 2011 at 3:54 AM, NeilBrown wrote:
> On Tue, 30 Aug 2011 18:18:11 +0300 Alexander Lyakas il.com>
> wrote:
>
>> Thanks, Neil.
>>
>> Although according to udev documentation: "the udev events are sent
>> out after udev has finished its event processing, all rules have bee=
n
>> processed, and needed device nodes are created."
>>
>> Also looking at udev-worker code of udevd, the
>> udev_monitor_send_device() call is done after all the rules have bee=
n
>> processed.
>>
>> Nevertheless, I looked at udevadm_settle.c and did some equivalent o=
f
>> that in my code, and it looks like the issue is resolved. Perhaps
>> there is something md-specific here?
>
> I cannot see how it would be md-specific. =A0mdadm doesn't create or =
remove
> devices when udev is active - it leaves all that to udev.
> If you are curious I suggest you ask the udev developers.
>
>>
>> Another thing, since you are reading this thread, I wanted to ask
>> whether you have any advice on the "RAID5: failing an active compone=
nt
>> during spare rebuild - arrays hangs" thread I opened some time ago.
>> Since you were not answering, I assume there is nothing additional y=
ou
>> can advise about, correct? I apologize if this off-topic was
>> inappropriate.
>
> It could mean that I had nothing extra to say, but it could also mean=
that I
> got distracted, forgot, and never got back to it. =A0I live in a worl=
d of
> distractions :-(
>
> But a reminder never hurts - it shows that it is important to you, so=
that
> makes it at least a little bit important to me. =A0I'll go back and h=
ave a look
> and see if I have anything useful to add.
>
> NeilBrown
>
>
>>
>> Thanks for the help,
>> =A0 Alex.
>>
>>
>>
>> On Tue, Aug 30, 2011 at 12:25 AM, NeilBrown wrote:
>> > On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas gmail.com>
>> > wrote:
>> >
>> >> Greetings everybody,
>> >>
>> >> I issue
>> >> mdadm --stop /dev/md0
>> >> and I want to reliably determine that the MD devnode (/dev/md0) i=
s gone.
>> >> So I look for the udev 'remove' event for that devnode.
>> >> However, in some cases even after I see the udev event, I issue
>> >> mdadm --detail /dev/md0
>> >> and I get:
>> >> mdadm: md device /dev/md0 does not appear to be active
>> >>
>> >> According to Detail.c, this means that mdadm can successfully do
>> >> open("/dev/md0") and receive a valid fd.
>> >> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives ENO=
DEV
>> >> from the kernel.
>> >>
>> >> Can somebody suggest an explanation for this behavior? Is there a
>> >> reliable way to know when a MD devnode is gone?
>> >
>> > run "udevadm settle" after stopping /dev/md0 =A0is most likely to =
work.
>> >
>> > I suspect that udev removes the node *after* you see the 'remove' =
event.
>> > Sometimes so soon after that you don't see the lag - sometimes a b=
it later.
>> >
>> > NeilBrown
>> >
>> >>
>> >> Thanks,
>> >> =A0 Alex.
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe linux-r=
aid" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at =A0http://vger.kernel.org/majordomo-info.h=
tml
>> >
>> >
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid=
" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

Hello Neil,
I am sorry for opening this again, but I am convinced now that I don't
understand what's going on:)

Basically, I see that GET_ARRAY_INFO can also return ENODEV in case
the device in the kernel exists, but "we are not initialized yet":
/* if we are not initialised yet, only ADD_NEW_DISK, STOP_ARRAY,
* RUN_ARRAY, and GET_ and SET_BITMAP_FILE are allowed */
if ((!mddev->raid_disks && !mddev->external)
&& cmd !=3D ADD_NEW_DISK && cmd !=3D STOP_ARRAY
&& cmd !=3D RUN_ARRAY && cmd !=3D SET_BITMAP_FILE
&& cmd !=3D GET_BITMAP_FILE) {
err =3D -ENODEV;
goto abort_unlock;

I thought that ENODEV means that the device in the kernel does not
exist, although I am not this familiar with the kernel sources (yet)
to verify that.

Basically, I just wanted to know whether there is a reliable way to
determine whether the kernel MD device exists or no. (Obviously,
success to open a devnode from user space is not enough).

Thanks,
Alex.







On Tue, Aug 30, 2011 at 12:25 AM, NeilBrown wrote:
> On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas il.com>
> wrote:
>
>> Greetings everybody,
>>
>> I issue
>> mdadm --stop /dev/md0
>> and I want to reliably determine that the MD devnode (/dev/md0) is g=
one.
>> So I look for the udev 'remove' event for that devnode.
>> However, in some cases even after I see the udev event, I issue
>> mdadm --detail /dev/md0
>> and I get:
>> mdadm: md device /dev/md0 does not appear to be active
>>
>> According to Detail.c, this means that mdadm can successfully do
>> open("/dev/md0") and receive a valid fd.
>> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives ENODEV
>> from the kernel.
>>
>> Can somebody suggest an explanation for this behavior? Is there a
>> reliable way to know when a MD devnode is gone?
>
> run "udevadm settle" after stopping /dev/md0 =A0is most likely to wor=
k.
>
> I suspect that udev removes the node *after* you see the 'remove' eve=
nt.
> Sometimes so soon after that you don't see the lag - sometimes a bit =
later.
>
> NeilBrown
>
>>
>> Thanks,
>> =A0 Alex.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid=
" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

--Sig_/pBTLdAJz+fbWKqxW_DW6v5R
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, 13 Sep 2011 11:49:12 +0300 Alexander Lyakas
wrote:

> Hello Neil,
> I am sorry for opening this again, but I am convinced now that I don't
> understand what's going on:)
>=20
> Basically, I see that GET_ARRAY_INFO can also return ENODEV in case
> the device in the kernel exists, but "we are not initialized yet":
> /* if we are not initialised yet, only ADD_NEW_DISK, STOP_ARRAY,
> * RUN_ARRAY, and GET_ and SET_BITMAP_FILE are allowed */
> if ((!mddev->raid_disks && !mddev->external)
> && cmd !=3D ADD_NEW_DISK && cmd !=3D STOP_ARRAY
> && cmd !=3D RUN_ARRAY && cmd !=3D SET_BITMAP_FILE
> && cmd !=3D GET_BITMAP_FILE) {
> err =3D -ENODEV;
> goto abort_unlock;
>=20
> I thought that ENODEV means that the device in the kernel does not
> exist, although I am not this familiar with the kernel sources (yet)
> to verify that.
>=20
> Basically, I just wanted to know whether there is a reliable way to
> determine whether the kernel MD device exists or no. (Obviously,
> success to open a devnode from user space is not enough).
>=20
> Thanks,
> Alex.

What exactly do you mean by "the kernel MD device exists" ??

When you open a device-special-file for an md device (major == 9) it
automatically creates an inactive array. You can then fill in the details
and activate it, or explicitly deactivate it. If you do that it will
disappear.

Opening the devnode is enough to check that the device exists, because it
creates the device and then you know that it exists.
If you want to know if it already exists - whether inactive or not - look
in /proc/mdstat or /sys/block/md*.
If you want to know if it already exists and is active, look in /proc/mdsta=
t,
or open the device and use GET_ARRAY_INFO, or look in /sys/block/md*
and look at the device size. or maybe /sys/block/mdXX/md/raid_disks.

It depends on why you are asking.

NeilBrown



>=20
>=20
>=20
>=20
>=20
>=20
>=20
> On Tue, Aug 30, 2011 at 12:25 AM, NeilBrown wrote:
> > On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas ..com>
> > wrote:
> >
> >> Greetings everybody,
> >>
> >> I issue
> >> mdadm --stop /dev/md0
> >> and I want to reliably determine that the MD devnode (/dev/md0) is gon=
e.
> >> So I look for the udev 'remove' event for that devnode.
> >> However, in some cases even after I see the udev event, I issue
> >> mdadm --detail /dev/md0
> >> and I get:
> >> mdadm: md device /dev/md0 does not appear to be active
> >>
> >> According to Detail.c, this means that mdadm can successfully do
> >> open("/dev/md0") and receive a valid fd.
> >> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives ENODEV
> >> from the kernel.
> >>
> >> Can somebody suggest an explanation for this behavior? Is there a
> >> reliable way to know when a MD devnode is gone?
> >
> > run "udevadm settle" after stopping /dev/md0 =A0is most likely to work.
> >
> > I suspect that udev removes the node *after* you see the 'remove' event.
> > Sometimes so soon after that you don't see the lag - sometimes a bit la=
ter.
> >
> > NeilBrown
> >
> >>
> >> Thanks,
> >> =A0 Alex.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" =
in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
> >
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html


--Sig_/pBTLdAJz+fbWKqxW_DW6v5R
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iD8DBQFOeXAbG5fc6gV+Wb0RAlk5AJ9MIzhfZeYoACvopdjXUzI96InD5gCg tbOi
sLCU7BkZTDlx54b5sbWM8a8=
=uiWF
-----END PGP SIGNATURE-----

--Sig_/pBTLdAJz+fbWKqxW_DW6v5R--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

Thank you, Neil, for answering.
I'm not sure that understand all of this, because my knowledge of
Linux user-kernel interaction is, unfortunately, not sufficient. In
the future, I hope to know more.
=46or example, I don't understand, how opening a "/dev/mdXX" can create
a device in the kernel, if the devnode "/dev/mdXX" does not exist. In
that case, I actually fail to open it with ENOENT.

But what I did is actually similar to what you advised:
- if I fail to open the devnode with ENOENT, I know (?) that the
device does not exist
- otherwise, I do GET_ARRAY_INFO
- if it returns ok, then I go ahead and do GET_DISK_INFOs to get the
disks information
- otherwise if it returns ENODEV, I close the fd and then I read /proc/=
mdstat
- if the md is there, then I know it's inactive array (and I have to
--stop it and reassemble or do incremental assembly)
- if the md is not there, then I know that it really does not exist
(this is the case when md deletion happened but the devnode did not
disappear yet)

Does it sound right? It passes stress testing pretty well.

By the way, I understand that /proc/mdstat can be only of 4K size...so
if I have many arrays, I should probably switch to look at
/sys/block....

Thanks,
Alex.






On Wed, Sep 21, 2011 at 8:03 AM, NeilBrown wrote:
>
> On Tue, 13 Sep 2011 11:49:12 +0300 Alexander Lyakas il.com>
> wrote:
>
> > Hello Neil,
> > I am sorry for opening this again, but I am convinced now that I do=
n't
> > understand what's going on:)
> >
> > Basically, I see that GET_ARRAY_INFO can also return ENODEV in case
> > the device in the kernel exists, but "we are not initialized yet":
> > /* if we are not initialised yet, only ADD_NEW_DISK, STOP_ARRAY,
> > =A0* RUN_ARRAY, and GET_ and SET_BITMAP_FILE are allowed */
> > if ((!mddev->raid_disks && !mddev->external)
> > =A0 =A0 && cmd !=3D ADD_NEW_DISK && cmd !=3D STOP_ARRAY
> > =A0 =A0 && cmd !=3D RUN_ARRAY && cmd !=3D SET_BITMAP_FILE
> > =A0 =A0 && cmd !=3D GET_BITMAP_FILE) {
> > =A0 =A0 =A0 err =3D -ENODEV;
> > =A0 =A0 =A0 goto abort_unlock;
> >
> > I thought that ENODEV means that the device in the kernel does not
> > exist, although I am not this familiar with the kernel sources (yet=
)
> > to verify that.
> >
> > Basically, I just wanted to know whether there is a reliable way to
> > determine whether the kernel MD device exists or no. (Obviously,
> > success to open a devnode from user space is not enough).
> >
> > Thanks,
> > =A0 Alex.
>
> What exactly do you mean by "the kernel MD device exists" ??
>
> When you open a device-special-file for an md device (major == 9)=
it
> automatically creates an inactive array. =A0You can then fill in the =
details
> and activate it, or explicitly deactivate it. =A0If you do that it wi=
ll
> disappear.
>
> Opening the devnode is enough to check that the device exists, becaus=
e it
> creates the device and then you know that it exists.
> If you want to know if it already exists - whether inactive or not - =
look
> in /proc/mdstat or /sys/block/md*.
> If you want to know if it already exists and is active, look in /proc=
/mdstat,
> or open the device and use GET_ARRAY_INFO, or look in /sys/block/md*
> and look at the device size. or maybe /sys/block/mdXX/md/raid_disks.
>
> It depends on why you are asking.
>
> NeilBrown
>
>
>
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Aug 30, 2011 at 12:25 AM, NeilBrown wrote:
> > > On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas @gmail.com>
> > > wrote:
> > >
> > >> Greetings everybody,
> > >>
> > >> I issue
> > >> mdadm --stop /dev/md0
> > >> and I want to reliably determine that the MD devnode (/dev/md0) =
is gone.
> > >> So I look for the udev 'remove' event for that devnode.
> > >> However, in some cases even after I see the udev event, I issue
> > >> mdadm --detail /dev/md0
> > >> and I get:
> > >> mdadm: md device /dev/md0 does not appear to be active
> > >>
> > >> According to Detail.c, this means that mdadm can successfully do
> > >> open("/dev/md0") and receive a valid fd.
> > >> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives EN=
ODEV
> > >> from the kernel.
> > >>
> > >> Can somebody suggest an explanation for this behavior? Is there =
a
> > >> reliable way to know when a MD devnode is gone?
> > >
> > > run "udevadm settle" after stopping /dev/md0 =A0is most likely to=
work.
> > >
> > > I suspect that udev removes the node *after* you see the 'remove'=
event.
> > > Sometimes so soon after that you don't see the lag - sometimes a =
bit later.
> > >
> > > NeilBrown
> > >
> > >>
> > >> Thanks,
> > >> =A0 Alex.
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe linux-=
raid" in
> > >> the body of a message to majordomo@vger.kernel.org
> > >> More majordomo info at =A0http://vger.kernel.org/majordomo-info.=
html
> > >
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rai=
d" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm=
l
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

--Sig_/yhhRG7Xiw4qEMEtuGv5vhy4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, 23 Sep 2011 22:24:08 +0300 Alexander Lyakas
wrote:

> Thank you, Neil, for answering.
> I'm not sure that understand all of this, because my knowledge of
> Linux user-kernel interaction is, unfortunately, not sufficient. In
> the future, I hope to know more.
> For example, I don't understand, how opening a "/dev/mdXX" can create
> a device in the kernel, if the devnode "/dev/mdXX" does not exist. In
> that case, I actually fail to open it with ENOENT.

/dev/mdXX is a "device special file". It is not the device itself.
You can think of it like a symbolic link.
The "real" name for the device is something like "block device with major 9
and minor X" That thing can exist quite independently of whether
the /dev/mdXX thing exists. Just like a file may or may not exist
independently of whether some sym-link to it exists.

When the device (block,9,XX) appears, udev is told and it should create
things in /dev. when the device disappears, udev is told and it should
remove the /dev entry. But there can be races, and other things might
sometimes add or remove /dev entries (though they shouldn't). So the
existence of something in /dev isn't a guarantee that it really exists.


>=20
> But what I did is actually similar to what you advised:
> - if I fail to open the devnode with ENOENT, I know (?) that the
> device does not exist
> - otherwise, I do GET_ARRAY_INFO
> - if it returns ok, then I go ahead and do GET_DISK_INFOs to get the
> disks information
> - otherwise if it returns ENODEV, I close the fd and then I read /proc/md=
stat
> - if the md is there, then I know it's inactive array (and I have to
> --stop it and reassemble or do incremental assembly)
> - if the md is not there, then I know that it really does not exist
> (this is the case when md deletion happened but the devnode did not
> disappear yet)
>=20
> Does it sound right? It passes stress testing pretty well.

Yes, that sounds right.

>=20
> By the way, I understand that /proc/mdstat can be only of 4K size...so
> if I have many arrays, I should probably switch to look at
> /sys/block....

Correct.

NeilBrown


>=20
> Thanks,
> Alex.
>=20
>=20
>=20
>=20
>=20
>=20
> On Wed, Sep 21, 2011 at 8:03 AM, NeilBrown wrote:
> >
> > On Tue, 13 Sep 2011 11:49:12 +0300 Alexander Lyakas ..com>
> > wrote:
> >
> > > Hello Neil,
> > > I am sorry for opening this again, but I am convinced now that I don't
> > > understand what's going on:)
> > >
> > > Basically, I see that GET_ARRAY_INFO can also return ENODEV in case
> > > the device in the kernel exists, but "we are not initialized yet":
> > > /* if we are not initialised yet, only ADD_NEW_DISK, STOP_ARRAY,
> > > =A0* RUN_ARRAY, and GET_ and SET_BITMAP_FILE are allowed */
> > > if ((!mddev->raid_disks && !mddev->external)
> > > =A0 =A0 && cmd !=3D ADD_NEW_DISK && cmd !=3D STOP_ARRAY
> > > =A0 =A0 && cmd !=3D RUN_ARRAY && cmd !=3D SET_BITMAP_FILE
> > > =A0 =A0 && cmd !=3D GET_BITMAP_FILE) {
> > > =A0 =A0 =A0 err =3D -ENODEV;
> > > =A0 =A0 =A0 goto abort_unlock;
> > >
> > > I thought that ENODEV means that the device in the kernel does not
> > > exist, although I am not this familiar with the kernel sources (yet)
> > > to verify that.
> > >
> > > Basically, I just wanted to know whether there is a reliable way to
> > > determine whether the kernel MD device exists or no. (Obviously,
> > > success to open a devnode from user space is not enough).
> > >
> > > Thanks,
> > > =A0 Alex.
> >
> > What exactly do you mean by "the kernel MD device exists" ??
> >
> > When you open a device-special-file for an md device (major == 9) it
> > automatically creates an inactive array. =A0You can then fill in the de=
tails
> > and activate it, or explicitly deactivate it. =A0If you do that it will
> > disappear.
> >
> > Opening the devnode is enough to check that the device exists, because =
it
> > creates the device and then you know that it exists.
> > If you want to know if it already exists - whether inactive or not - lo=
ok
> > in /proc/mdstat or /sys/block/md*.
> > If you want to know if it already exists and is active, look in /proc/m=
dstat,
> > or open the device and use GET_ARRAY_INFO, or look in /sys/block/md*
> > and look at the device size. or maybe /sys/block/mdXX/md/raid_disks.
> >
> > It depends on why you are asking.
> >
> > NeilBrown
> >
> >
> >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Aug 30, 2011 at 12:25 AM, NeilBrown wrote:
> > > > On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas mail.com>
> > > > wrote:
> > > >
> > > >> Greetings everybody,
> > > >>
> > > >> I issue
> > > >> mdadm --stop /dev/md0
> > > >> and I want to reliably determine that the MD devnode (/dev/md0) is=
gone.
> > > >> So I look for the udev 'remove' event for that devnode.
> > > >> However, in some cases even after I see the udev event, I issue
> > > >> mdadm --detail /dev/md0
> > > >> and I get:
> > > >> mdadm: md device /dev/md0 does not appear to be active
> > > >>
> > > >> According to Detail.c, this means that mdadm can successfully do
> > > >> open("/dev/md0") and receive a valid fd.
> > > >> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives ENOD=
EV
> > > >> from the kernel.
> > > >>
> > > >> Can somebody suggest an explanation for this behavior? Is there a
> > > >> reliable way to know when a MD devnode is gone?
> > > >
> > > > run "udevadm settle" after stopping /dev/md0 =A0is most likely to w=
ork.
> > > >
> > > > I suspect that udev removes the node *after* you see the 'remove' e=
vent.
> > > > Sometimes so soon after that you don't see the lag - sometimes a bi=
t later.
> > > >
> > > > NeilBrown
> > > >
> > > >>
> > > >> Thanks,
> > > >> =A0 Alex.
> > > >> --
> > > >> To unsubscribe from this list: send the line "unsubscribe linux-ra=
id" in
> > > >> the body of a message to majordomo@vger.kernel.org
> > > >> More majordomo info at =A0http://vger.kernel.org/majordomo-info.ht=
ml
> > > >
> > > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-raid"=
in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
> >


--Sig_/yhhRG7Xiw4qEMEtuGv5vhy4
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iD8DBQFOfv8uG5fc6gV+Wb0RAlJdAJ9mjR7leOsodxWYV71vwrxHQCgmRQCg gBlm
km8GGvboD6ywOcgqgPcMr14=
=ddqj
-----END PGP SIGNATURE-----

--Sig_/yhhRG7Xiw4qEMEtuGv5vhy4--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

Hello Neil,
can you please confirm for me something?
In case the array is FAILED (when your enough() function returns 0) -
for example, after simultaneous failure of all drives - then the only
option to try to recover such array is to do:
mdadm --stop
and then attempt
mdadm --assemble

correct?

I did not see any other option to recover such array Incremental
assemble doesn't work in that case, it simply adds back the drives as
spares.

Thanks,
Alex.

On Sun, Sep 25, 2011 at 12:15 PM, NeilBrown wrote:
> On Fri, 23 Sep 2011 22:24:08 +0300 Alexander Lyakas il.com>
> wrote:
>
>> Thank you, Neil, for answering.
>> I'm not sure that understand all of this, because my knowledge of
>> Linux user-kernel interaction is, unfortunately, not sufficient. In
>> the future, I hope to know more.
>> For example, I don't understand, how opening a "/dev/mdXX" can creat=
e
>> a device in the kernel, if the devnode "/dev/mdXX" does not exist. I=
n
>> that case, I actually fail to open it with ENOENT.
>
> /dev/mdXX is a "device special file". =A0It is not the device itself.
> You can think of it like a symbolic link.
> The "real" name for the device is something like "block device with m=
ajor 9
> and minor X" =A0That thing can exist quite independently of whether
> the /dev/mdXX thing exists. =A0Just like a file may or may not exist
> independently of whether some sym-link to it exists.
>
> When the device (block,9,XX) appears, udev is told and it should crea=
te
> things in /dev. =A0when the device disappears, udev is told and it sh=
ould
> remove the /dev entry. =A0But there can be races, and other things mi=
ght
> sometimes add or remove /dev entries (though they shouldn't). =A0So t=
he
> existence of something in /dev isn't a guarantee that it really exist=
s.
>
>
>>
>> But what I did is actually similar to what you advised:
>> - if I fail to open the devnode with ENOENT, I know (?) that the
>> device does not exist
>> - otherwise, I do GET_ARRAY_INFO
>> - if it returns ok, then I go ahead and do GET_DISK_INFOs to get the
>> disks information
>> - otherwise if it returns ENODEV, I close the fd and then I read /pr=
oc/mdstat
>> - if the md is there, then I know it's inactive array (and I have to
>> --stop it and reassemble or do incremental assembly)
>> - if the md is not there, then I know that it really does not exist
>> (this is the case when md deletion happened but the devnode did not
>> disappear yet)
>>
>> Does it sound right? It passes stress testing pretty well.
>
> Yes, that sounds right.
>
>>
>> By the way, I understand that /proc/mdstat can be only of 4K size...=
so
>> if I have many arrays, I should probably switch to look at
>> /sys/block....
>
> Correct.
>
> NeilBrown
>
>
>>
>> Thanks,
>> =A0 Alex.
>>
>>
>>
>>
>>
>>
>> On Wed, Sep 21, 2011 at 8:03 AM, NeilBrown wrote:
>> >
>> > On Tue, 13 Sep 2011 11:49:12 +0300 Alexander Lyakas gmail.com>
>> > wrote:
>> >
>> > > Hello Neil,
>> > > I am sorry for opening this again, but I am convinced now that I=
don't
>> > > understand what's going on:)
>> > >
>> > > Basically, I see that GET_ARRAY_INFO can also return ENODEV in c=
ase
>> > > the device in the kernel exists, but "we are not initialized yet=
":
>> > > /* if we are not initialised yet, only ADD_NEW_DISK, STOP_ARRAY,
>> > > =A0* RUN_ARRAY, and GET_ and SET_BITMAP_FILE are allowed */
>> > > if ((!mddev->raid_disks && !mddev->external)
>> > > =A0 =A0 && cmd !=3D ADD_NEW_DISK && cmd !=3D STOP_ARRAY
>> > > =A0 =A0 && cmd !=3D RUN_ARRAY && cmd !=3D SET_BITMAP_FILE
>> > > =A0 =A0 && cmd !=3D GET_BITMAP_FILE) {
>> > > =A0 =A0 =A0 err =3D -ENODEV;
>> > > =A0 =A0 =A0 goto abort_unlock;
>> > >
>> > > I thought that ENODEV means that the device in the kernel does n=
ot
>> > > exist, although I am not this familiar with the kernel sources (=
yet)
>> > > to verify that.
>> > >
>> > > Basically, I just wanted to know whether there is a reliable way=
to
>> > > determine whether the kernel MD device exists or no. (Obviously,
>> > > success to open a devnode from user space is not enough).
>> > >
>> > > Thanks,
>> > > =A0 Alex.
>> >
>> > What exactly do you mean by "the kernel MD device exists" ??
>> >
>> > When you open a device-special-file for an md device (major ===
9) it
>> > automatically creates an inactive array. =A0You can then fill in t=
he details
>> > and activate it, or explicitly deactivate it. =A0If you do that it=
will
>> > disappear.
>> >
>> > Opening the devnode is enough to check that the device exists, bec=
ause it
>> > creates the device and then you know that it exists.
>> > If you want to know if it already exists - whether inactive or not=
- look
>> > in /proc/mdstat or /sys/block/md*.
>> > If you want to know if it already exists and is active, look in /p=
roc/mdstat,
>> > or open the device and use GET_ARRAY_INFO, or look in /sys/block/m=
d*
>> > and look at the device size. or maybe /sys/block/mdXX/md/raid_disk=
s.
>> >
>> > It depends on why you are asking.
>> >
>> > NeilBrown
>> >
>> >
>> >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Tue, Aug 30, 2011 at 12:25 AM, NeilBrown wrot=
e:
>> > > > On Mon, 29 Aug 2011 20:17:34 +0300 Alexander Lyakas hoy@gmail.com>
>> > > > wrote:
>> > > >
>> > > >> Greetings everybody,
>> > > >>
>> > > >> I issue
>> > > >> mdadm --stop /dev/md0
>> > > >> and I want to reliably determine that the MD devnode (/dev/md=
0) is gone.
>> > > >> So I look for the udev 'remove' event for that devnode.
>> > > >> However, in some cases even after I see the udev event, I iss=
ue
>> > > >> mdadm --detail /dev/md0
>> > > >> and I get:
>> > > >> mdadm: md device /dev/md0 does not appear to be active
>> > > >>
>> > > >> According to Detail.c, this means that mdadm can successfully=
do
>> > > >> open("/dev/md0") and receive a valid fd.
>> > > >> But later, when issuing ioctl(fd, GET_ARRAY_INFO) it receives=
ENODEV
>> > > >> from the kernel.
>> > > >>
>> > > >> Can somebody suggest an explanation for this behavior? Is the=
re a
>> > > >> reliable way to know when a MD devnode is gone?
>> > > >
>> > > > run "udevadm settle" after stopping /dev/md0 =A0is most likely=
to work.
>> > > >
>> > > > I suspect that udev removes the node *after* you see the 'remo=
ve' event.
>> > > > Sometimes so soon after that you don't see the lag - sometimes=
a bit later.
>> > > >
>> > > > NeilBrown
>> > > >
>> > > >>
>> > > >> Thanks,
>> > > >> =A0 Alex.
>> > > >> --
>> > > >> To unsubscribe from this list: send the line "unsubscribe lin=
ux-raid" in
>> > > >> the body of a message to majordomo@vger.kernel.org
>> > > >> More majordomo info at =A0http://vger.kernel.org/majordomo-in=
fo.html
>> > > >
>> > > >
>> > > --
>> > > To unsubscribe from this list: send the line "unsubscribe linux-=
raid" in
>> > > the body of a message to majordomo@vger.kernel.org
>> > > More majordomo info at =A0http://vger.kernel.org/majordomo-info.=
html
>> >
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: MD devnode still present after "remove" udev event, and mdadmreports "does not appear to be acti

--Sig_/n7+YAKePBPA6ejKerGcj6=h
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

On Tue, 11 Oct 2011 15:11:47 +0200 Alexander Lyakas
wrote:

> Hello Neil,
> can you please confirm for me something?
> In case the array is FAILED (when your enough() function returns 0) -
> for example, after simultaneous failure of all drives - then the only
> option to try to recover such array is to do:
> mdadm --stop
> and then attempt
> mdadm --assemble
>=20
> correct?

Yes, though you will probably want a --force as well.

>=20
> I did not see any other option to recover such array Incremental
> assemble doesn't work in that case, it simply adds back the drives as
> spares.

In recent version of mdadm it shouldn't add them as spare. It should say
that it cannot add it and give up.

NeilBrown



--Sig_/n7+YAKePBPA6ejKerGcj6=h
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iQIVAwUBTpUNWznsnt1WYoG5AQJAaw/+Mcyi4BXkhBUeLQwl7ddTplLOImk5 xJyS
RxR1KYUVH4UMHqQbO7HohkAb3ZOE+3IPf+w13OYGpHsK7VZMl3YDQr0We3p7 3hyw
ntDy2Sj038DYmdABvFeAmxUlyWxI8k7wxJjKRh6RU45F5Y1SlETfNCkS60KY w8/F
L6r3TGyNstuVGo/hC9YIVJmCNGo/yRmpPAHmGG/8n4vYPoeP8yP9RUTFo8MB 7eyY
XlLma/+k5T/dD1gvimpNslNcpyhGWDawbpImwT1TVKNfBkzcmXMU+jrVh13R Ty5H
bjx7qclCiG1hp0on94S3XLoBBzdNG+JmFSLmNFJlsc6HeKyFVqxc2iqgO5p1 4EW6
VLL2IdQhoxsdKwZ8+kSOOVef7zeKv+p2eqKzSoeT535DSsRj9gF0Z0/sLGNc UyX2
WL8iJfdOK6OUkCSrZrTAKlzngnOdSsBV8319MAPnTFRWF4DQndHiJoRShYjk vw3I
g0p9v37t8Sez9vTJR/0z6OFaH8yrlvlGLzL9Ns4Jl4QMeEieQ042TqgANNE8 f3vR
7g2/6lZecQ8goBoLpRa34EInGcwRKH5GQPpmJ/0UQ9awzLrY7GN+TUEXVvgS saYS
mT7hN2MuW7NhX4nU+mgbDlpy1cScHMBm6l5hnKGs2XdtsJK6ikE9I+SSG0Yu QUYX
EAvLGVcu3x4=
=jFJC
-----END PGP SIGNATURE-----

--Sig_/n7+YAKePBPA6ejKerGcj6=h--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html