Re: Bug#624343: linux-image-2.6.38-2-amd64: frequent message "biotoo big device md0 (248 > 2

On Mon, 02 May 2011 11:08:11 +0200 David Brown wrote:

> On 02/05/2011 02:22, NeilBrown wrote:
> > On Mon, 02 May 2011 01:00:57 +0100 Ben Hutchings wrote:
> >
> >> On Sun, 2011-05-01 at 15:06 -0700, Jameson Graef Rollins wrote:
> >>> On Fri, 29 Apr 2011 05:39:40 +0100, Ben Hutchings wrote:
> >>>> On Wed, 2011-04-27 at 09:19 -0700, Jameson Graef Rollins wrote:
> >>>>> I run what I imagine is a fairly unusual disk setup on my laptop,
> >>>>> consisting of:
> >>>>>
> >>>>> ssd -> raid1 -> dm-crypt -> lvm -> ext4
> >>>>>
> >>>>> I use the raid1 as a backup. The raid1 operates normally in degraded
> >>>>> mode. For backups I then hot-add a usb hdd, let the raid1 sync, and
> >>>>> then fail/remove the external hdd.
> >>>>
> >>>> Well, this is not expected to work. Possibly the hot-addition of a disk
> >>>> with different bio restrictions should be rejected. But I'm not sure,
> >>>> because it is safe to do that if there is no mounted filesystem or
> >>>> stacking device on top of the RAID.
> >>>
> >>> Hi, Ben. Can you explain why this is not expected to work? Which part
> >>> exactly is not expected to work and why?
> >>
> >> Adding another type of disk controller (USB storage versus whatever the
> >> SSD interface is) to a RAID that is already in use.
> >
> > Normally this practice is perfectly OK.
> > If a filesysytem is mounted directly from an md array, then adding devices
> > to the array at any time is fine, even if the new devices have quite
> > different characteristics than the old.
> >
> > However if there is another layer in between md and the filesystem - such as
> > dm - then there can be problem.
> > There is no mechanism in the kernl for md to tell dm that things have
> > changed, so dm never changes its configuration to match any change in the
> > config of the md device.
> >
>
> While I can see that there might be limitations in informing the dm
> layer about changes to the md layer, I fail to see what changes we are
> talking about. If the OP were changing the size of the raid1, for
> example, then that would be a metadata change that needed to propagate
> up so that lvm could grow its physical volume. But the dm layer should
> not care if a disk is added or removed from the md raid1 set - as long
> as the /dev/mdX device stays online and valid, it should work correctly.
>

The changes we are talking about are "maximum supported request size" aka
max_sectors.

md sets max_sectors from the minimum of the max_sectors values of all
component devices. Of course if a device changes its max_sectors value, md
won't notice.

dm sets max_sectors from the minimum of the max_sectors values of all
component devices. Of course if a device changes its max_sectors value
after it has been included in the map, dm doesn't notice.

Every time a filesystem creates a request, it check the max_sectors of
the device and limits the request size accordingly.

So if I add a device to an md/raid array which has a smaller max_sectors
value, then the max_sectors of the md array will change, but no-one will
notice.

There might be a way to tell dm to re-evaluate max_sectors etc, I don't
know. But even if there was having to do that would be a clumsy solution.

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Bug#624343: linux-image-2.6.38-2-amd64: frequent message "biotoo big device md0 (248 > 2

On 02/05/2011 12:00, NeilBrown wrote:
> On Mon, 02 May 2011 11:08:11 +0200 David Brown wrote:
>
>> On 02/05/2011 02:22, NeilBrown wrote:
>>> On Mon, 02 May 2011 01:00:57 +0100 Ben Hutchings wrote:
>>>
>>>> On Sun, 2011-05-01 at 15:06 -0700, Jameson Graef Rollins wrote:
>>>>> On Fri, 29 Apr 2011 05:39:40 +0100, Ben Hutchings wrote:
>>>>>> On Wed, 2011-04-27 at 09:19 -0700, Jameson Graef Rollins wrote:
>>>>>>> I run what I imagine is a fairly unusual disk setup on my laptop,
>>>>>>> consisting of:
>>>>>>>
>>>>>>> ssd -> raid1 -> dm-crypt -> lvm -> ext4
>>>>>>>
>>>>>>> I use the raid1 as a backup. The raid1 operates normally in degraded
>>>>>>> mode. For backups I then hot-add a usb hdd, let the raid1 sync, and
>>>>>>> then fail/remove the external hdd.
>>>>>>
>>>>>> Well, this is not expected to work. Possibly the hot-addition of a disk
>>>>>> with different bio restrictions should be rejected. But I'm not sure,
>>>>>> because it is safe to do that if there is no mounted filesystem or
>>>>>> stacking device on top of the RAID.
>>>>>
>>>>> Hi, Ben. Can you explain why this is not expected to work? Which part
>>>>> exactly is not expected to work and why?
>>>>
>>>> Adding another type of disk controller (USB storage versus whatever the
>>>> SSD interface is) to a RAID that is already in use.
>>>
>>> Normally this practice is perfectly OK.
>>> If a filesysytem is mounted directly from an md array, then adding devices
>>> to the array at any time is fine, even if the new devices have quite
>>> different characteristics than the old.
>>>
>>> However if there is another layer in between md and the filesystem - such as
>>> dm - then there can be problem.
>>> There is no mechanism in the kernl for md to tell dm that things have
>>> changed, so dm never changes its configuration to match any change in the
>>> config of the md device.
>>>
>>
>> While I can see that there might be limitations in informing the dm
>> layer about changes to the md layer, I fail to see what changes we are
>> talking about. If the OP were changing the size of the raid1, for
>> example, then that would be a metadata change that needed to propagate
>> up so that lvm could grow its physical volume. But the dm layer should
>> not care if a disk is added or removed from the md raid1 set - as long
>> as the /dev/mdX device stays online and valid, it should work correctly.
>>
>
> The changes we are talking about are "maximum supported request size" aka
> max_sectors.
>
> md sets max_sectors from the minimum of the max_sectors values of all
> component devices. Of course if a device changes its max_sectors value, md
> won't notice.
>
> dm sets max_sectors from the minimum of the max_sectors values of all
> component devices. Of course if a device changes its max_sectors value
> after it has been included in the map, dm doesn't notice.
>
> Every time a filesystem creates a request, it check the max_sectors of
> the device and limits the request size accordingly.
>
> So if I add a device to an md/raid array which has a smaller max_sectors
> value, then the max_sectors of the md array will change, but no-one will
> notice.
>
> There might be a way to tell dm to re-evaluate max_sectors etc, I don't
> know. But even if there was having to do that would be a clumsy solution.
>

I would think the one possible solution would be that if one layer (or
the filesystem) requested too large a block from the next layer, there
would be an error code returned. This would avoid the need to do any
re-evaluation under normal circumstances - there would only be extra
effort if the devices changed and caused a problem.

Another idea would be for a block layer to transparently split the
request into multiple smaller requests if needed - but that may cause
other complications such as when the order and atomicity of requests is
important.

Is there such a big variation of the maximum supported request size? Or
could it simply be fixed at some value - and any devices that can't
support that directly would emulate it with multiple requests?

I know next to nothing about the internals of the block layers (in case
that's not obvious from my posts...). But this strikes me as a strange
limitation to have on the flexibility of block layers.

I can see that in the great majority of cases, the maximum request size
would be static - as the layers of raid, dm, etc., are built up, with
the filesystem mounted on top, then the request size limitations are
passed on. What is unusual about the circumstances we are discussing is
that the maximum request size may change dynamically.



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Bug#624343: linux-image-2.6.38-2-amd64: frequent message "biotoo big device md0 (248 > 2

On 02/05/2011 12:00, NeilBrown wrote:
> On Mon, 02 May 2011 11:08:11 +0200 David Brown wrote:
>
>> On 02/05/2011 02:22, NeilBrown wrote:
>>> On Mon, 02 May 2011 01:00:57 +0100 Ben Hutchings wrote:
>>>
>>>> On Sun, 2011-05-01 at 15:06 -0700, Jameson Graef Rollins wrote:
>>>>> On Fri, 29 Apr 2011 05:39:40 +0100, Ben Hutchings wrote:
>>>>>> On Wed, 2011-04-27 at 09:19 -0700, Jameson Graef Rollins wrote:
>>>>>>> I run what I imagine is a fairly unusual disk setup on my laptop,
>>>>>>> consisting of:
>>>>>>>
>>>>>>> ssd -> raid1 -> dm-crypt -> lvm -> ext4
>>>>>>>
>>>>>>> I use the raid1 as a backup. The raid1 operates normally in degraded
>>>>>>> mode. For backups I then hot-add a usb hdd, let the raid1 sync, and
>>>>>>> then fail/remove the external hdd.
>>>>>>
>>>>>> Well, this is not expected to work. Possibly the hot-addition of a disk
>>>>>> with different bio restrictions should be rejected. But I'm not sure,
>>>>>> because it is safe to do that if there is no mounted filesystem or
>>>>>> stacking device on top of the RAID.
>>>>>
>>>>> Hi, Ben. Can you explain why this is not expected to work? Which part
>>>>> exactly is not expected to work and why?
>>>>
>>>> Adding another type of disk controller (USB storage versus whatever the
>>>> SSD interface is) to a RAID that is already in use.
>>>
>>> Normally this practice is perfectly OK.
>>> If a filesysytem is mounted directly from an md array, then adding devices
>>> to the array at any time is fine, even if the new devices have quite
>>> different characteristics than the old.
>>>
>>> However if there is another layer in between md and the filesystem - such as
>>> dm - then there can be problem.
>>> There is no mechanism in the kernl for md to tell dm that things have
>>> changed, so dm never changes its configuration to match any change in the
>>> config of the md device.
>>>
>>
>> While I can see that there might be limitations in informing the dm
>> layer about changes to the md layer, I fail to see what changes we are
>> talking about. If the OP were changing the size of the raid1, for
>> example, then that would be a metadata change that needed to propagate
>> up so that lvm could grow its physical volume. But the dm layer should
>> not care if a disk is added or removed from the md raid1 set - as long
>> as the /dev/mdX device stays online and valid, it should work correctly.
>>
>
> The changes we are talking about are "maximum supported request size" aka
> max_sectors.
>
> md sets max_sectors from the minimum of the max_sectors values of all
> component devices. Of course if a device changes its max_sectors value, md
> won't notice.
>
> dm sets max_sectors from the minimum of the max_sectors values of all
> component devices. Of course if a device changes its max_sectors value
> after it has been included in the map, dm doesn't notice.
>
> Every time a filesystem creates a request, it check the max_sectors of
> the device and limits the request size accordingly.
>
> So if I add a device to an md/raid array which has a smaller max_sectors
> value, then the max_sectors of the md array will change, but no-one will
> notice.
>
> There might be a way to tell dm to re-evaluate max_sectors etc, I don't
> know. But even if there was having to do that would be a clumsy solution.
>

I've done a little more reading about max_sectors, and it seems to be
specific to USB. It is also dynamically configurable through the sysfs
interface. This means that the user can freely play around with the
value while testing throughput on a USB device. So if the dm layer has
a problem with underlying devices changing their max_sectors value, then
anyone fiddling with max_sectors on a USB device with a dm layer and a
mounted filesystem is going to get in big trouble. Since dm-crypt and
USB are a common combination (encrypted flash stick), surely this is a
serious bug in the dm code?



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html