Cancel reshape?

I've just started reshaping a RAID-5 to a RAID-6. I had meant to
increase the number of drives too, so avoiding using the backup file all
the way through. Unfortunately I forgot to specify that, so it's going
rather slowly. I said:

mdadm /dev/md1 --grow --level=6 --backup-file=/some/where/else

where md1 had 3 active and 2 spare devices. It's now reshaping itself to
RAID-6 with 4 active and 1 spare.

What I meant was

mdadm /dev/md1 --grow --level=6 --raid-disks=5 --backup-file=...

Can I cancel reshape that's currently running, and using the backup file
that's actually on another (RAID-1) array on three of the same drives,
and running at ~7MB/s, and try again?

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Cancel reshape?

On Wed, 12 Jan 2011 04:19:30 +0000 John Robinson
wrote:

> I've just started reshaping a RAID-5 to a RAID-6. I had meant to
> increase the number of drives too, so avoiding using the backup file all
> the way through. Unfortunately I forgot to specify that, so it's going
> rather slowly. I said:
>
> mdadm /dev/md1 --grow --level=6 --backup-file=/some/where/else
>
> where md1 had 3 active and 2 spare devices. It's now reshaping itself to
> RAID-6 with 4 active and 1 spare.
>
> What I meant was
>
> mdadm /dev/md1 --grow --level=6 --raid-disks=5 --backup-file=...
>
> Can I cancel reshape that's currently running, and using the backup file
> that's actually on another (RAID-1) array on three of the same drives,
> and running at ~7MB/s, and try again?
>
Sorry, but no. This isn't supported yet.
It would require modification to the kernel to be able to run a reshape in
reverse.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Cancel reshape?

On 12/01/2011 04:19, John Robinson wrote:
[...]
> Can I cancel reshape that's currently running, and using the backup file
> that's actually on another (RAID-1) array on three of the same drives,
> and running at ~7MB/s, and try again?

Blast. I found this patch to add a revert-reshape option from archives
of this list - http://www.spinics.net/lists/raid/msg30961.html - but
it's not included in mdadm 3.1.4 and I'm running from a rescue CD so I
can't rebuild mdadm with it. I think I might just have to wait a while.

Perhaps I can stop the reshape and restart it with the backup file
elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to
give it a wee workout :-)

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Cancel reshape?

On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson
wrote:

> On 12/01/2011 04:19, John Robinson wrote:
> [...]
> > Can I cancel reshape that's currently running, and using the backup file
> > that's actually on another (RAID-1) array on three of the same drives,
> > and running at ~7MB/s, and try again?
>
> Blast. I found this patch to add a revert-reshape option from archives
> of this list - http://www.spinics.net/lists/raid/msg30961.html - but
> it's not included in mdadm 3.1.4 and I'm running from a rescue CD so I
> can't rebuild mdadm with it. I think I might just have to wait a while.
>

That patch only works to revert a reshape that is increasing the number of
data-devices in the array.

> Perhaps I can stop the reshape and restart it with the backup file
> elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to
> give it a wee workout :-)

That should work.

mdadm --stop ...
copy file
mdadm --assemble ...... --backup-file=/new/place

NeilBrown



>
> Cheers,
>
> John.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Speed up reshape? (was Re: Cancel reshape?)

On 12/01/2011 05:32, NeilBrown wrote:
> On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson
> wrote:
[...]
>> Perhaps I can stop the reshape and restart it with the backup file
>> elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to
>> give it a wee workout :-)
>
> That should work.
>
> mdadm --stop ...
> copy file
> mdadm --assemble ...... --backup-file=/new/place

Yup. Not much of an improvement, though; the reshape is now running at
~10MB/s instead of ~6.5MB/s, so ~27 hours to go instead of ~40 hours.

iostat tells me the original 3 discs are doing ~120 transactions per
second, reading ~20MB/s and writing ~10MB/s, while the one new disc is
doing ~60 tps, reading nothing (obviously) and writing ~10MB/s and the
backup (SSD) is doing ~45 tps and writing ~20MB/s. ~120tps sounds about
right for 120 seeks, 60 reads and 60 writes, and would explain the
speed, but otherwise this looks odd to me - why are we reading ~60MB/s
in total and writing ~40MB/s? I'd have expected to be reading less than
we're writing. And why are we apparently reading and writing in blocks
of under 512KB per device, when the backup file is 32MB? Is there
something I can tweak to run the reshape in bigger chunks?
sync_speed_min is already 200000...

Currently running SystemRescueCD 2.0.0, 32-bit kernel 2.6.35 and mdadm
3.1.4. CPU usage is 2% system, 4% iowait.

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Speed up reshape? (was Re: Cancel reshape?)

On Wed, 12 Jan 2011 05:53:40 +0000 John Robinson
wrote:

> On 12/01/2011 05:32, NeilBrown wrote:
> > On Wed, 12 Jan 2011 04:49:35 +0000 John Robinson
> > wrote:
> [...]
> >> Perhaps I can stop the reshape and restart it with the backup file
> >> elsewhere. Hmm, I've an SSD I haven't installed yet... Maybe time to
> >> give it a wee workout :-)
> >
> > That should work.
> >
> > mdadm --stop ...
> > copy file
> > mdadm --assemble ...... --backup-file=/new/place
>
> Yup. Not much of an improvement, though; the reshape is now running at
> ~10MB/s instead of ~6.5MB/s, so ~27 hours to go instead of ~40 hours.
>
> iostat tells me the original 3 discs are doing ~120 transactions per
> second, reading ~20MB/s and writing ~10MB/s, while the one new disc is
> doing ~60 tps, reading nothing (obviously) and writing ~10MB/s and the
> backup (SSD) is doing ~45 tps and writing ~20MB/s. ~120tps sounds about
> right for 120 seeks, 60 reads and 60 writes, and would explain the
> speed, but otherwise this looks odd to me - why are we reading ~60MB/s
> in total and writing ~40MB/s? I'd have expected to be reading less than
> we're writing. And why are we apparently reading and writing in blocks
> of under 512KB per device, when the backup file is 32MB? Is there
> something I can tweak to run the reshape in bigger chunks?
> sync_speed_min is already 200000...

1/ everything is read twice, once to back it up, once to relocate it.
That is unfortunate, but awkward to avoid.

2/ try increasing the stripe_cache_size - it might help.

NeilBrown


>
> Currently running SystemRescueCD 2.0.0, 32-bit kernel 2.6.35 and mdadm
> 3.1.4. CPU usage is 2% system, 4% iowait.
>
> Cheers,
>
> John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Speed up reshape? (was Re: Cancel reshape?)

On 12/01/2011 06:02, NeilBrown wrote:
> On Wed, 12 Jan 2011 05:53:40 +0000 John Robinson
[...]
> 1/ everything is read twice, once to back it up, once to relocate it.
> That is unfortunate, but awkward to avoid.

OK... but why only one stripe at a time? I eventually worked out that
the disc transactions (shown by iostat) were of 256KB at a time which is
the chunk size of the array, and the ~120 transactions per second would
relate to the spin speed of the discs, 7200rpm being 120 revolutions per
second. Even at the slow end of the discs, there's at least 512KB of
data per revolution, so slurping up at least a couple of revolutions'
worth before rewriting it might save waiting for quite so many disc
revolutions to finish.

I may test this by doing another in-place reshape, changing the chunk
size to 512KB, which I expect will run at twice the speed of the reshape
on 256KB chunks.

Also, would there be any way to allow the second read to come from cache
rather than from the media surface again? Is this a side-effect of using
O_DIRECT or something? Could O_DIRECT be used only for writing?

> 2/ try increasing the stripe_cache_size - it might help.

mdadm had already set it to 1071, I upped it to 8192, then 32768, but it
didn't make any difference at all, and stripe_cache_active showed 8 or
even 0 when I checked. I imagine this is because the in-place reshape
needs to wait for writes to be sync'ed all the time (or written with
O_DIRECT or whatever) so there's little or no scope for cacheing stripes
outside the reshape process itself.

I'm now reshaping again, from 4 to 5 discs. That's whizzing along at a
fairly healthy 35MB/s, which according to iostat breaks down as about
50MB/s read from 4 drives and 35MB/s written to 5 drives and is as much
as I could reasonably expect from these discs. stripe_cache_active shows
~25000, CPU usage is 12% system, 14% iowait. I expect the speed to slow
down a bit towards the end of the reshape, where the slowest of the
drives can only manage 60MB/s streaming and the seeks between reading
and writing are longer (further across the disc surface).

Anyway, quite apart from all my whingeing about the speed of the
in-place reshape, it's all working perfectly which is after all the most
important thing, so many thanks for all your hard work!

Cheers,

John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: Speed up reshape? (was Re: Cancel reshape?)

On 13/01/2011 13:44, John Robinson wrote:
[...]
> I may test this by doing another in-place reshape, changing the chunk
> size to 512KB, which I expect will run at twice the speed of the reshape
> on 256KB chunks.

Well, I got that wrong. Currently doing ~9MB/s, though the backup file
is writing faster (~27MB/s). Another 30 hours reshaping. Oh well it
should be worth it in the end.

Cheers,

John.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html