RAID/dmaengine violates the dma-streaming API

RAID/dmaengine violates the dma-streaming API

am 17.04.2011 18:00:52 von saeed bishara

Hi,
when md uses the dma for offloading xor and memcpy operations, it
violates the dma-mapping API. here is the scenario I'm taking about
(under write to degraded raid5):
1. ops_run_prexor sends xor operation from buffers A and B, and the
destination is A.
2. ops_run_biodrain: sends mempcy operation from C to B.
3. ops_run_reconstruct5: sends xor operation from A and B, and the
destination is A again.

in step 1, the async tx maps A using dma_map_page, and in step 3, it
maps again the same buffer. but, if the request from step 1 still
being handled the dma engine, then we end with a case where the buffer
mapped while it still belongs to the dma hw.
when the arch is ARMv6/SMP mode (without io coherency), the cache
maintenance involves read/write access to the buffers, that means, the
second mapping above may access the buffer(with read/write) while the
dma is writing to it!!.

also, from performance sake, it would be great if this multiple
mappings get avoided.

saeed
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: RAID/dmaengine violates the dma-streaming API

am 27.04.2011 13:42:57 von saeed bishara

On Sun, Apr 17, 2011 at 7:00 PM, saeed bishara > wrote:
> Hi,
>     when md uses the dma for offloading xor and memcpy oper=
ations, it
> violates the dma-mapping API.  here is the scenario I'm taking a=
bout
> (under write to degraded raid5):
> 1. ops_run_prexor sends xor operation from buffers A and B, and the
> destination is A.
> 2. ops_run_biodrain: sends mempcy operation from C to B.
> 3. ops_run_reconstruct5: sends xor operation from A and B, and the
> destination is A again.
>
> in step 1, the async tx maps A using dma_map_page, and in step 3, it
> maps again the same buffer. but, if the request from step 1 still
> being handled the dma engine, then we end with a case where the buffe=
r
> mapped while it still belongs to the dma hw.
> when the arch is ARMv6/SMP mode (without io coherency), the cache
> maintenance involves read/write access to the buffers, that means, th=
e
> second mapping above may access the buffer(with read/write) while the
> dma is writing to it!!.
>
Russell/Dan,
can you have a look into this issue? what I see here is that the
raid stack issues dma_map_page to a buffer that still owned by DMA.

saeed

> also, from performance sake, it would be great if this multiple
> mappings get avoided.
>
> saeed
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: RAID/dmaengine violates the dma-streaming API

am 27.04.2011 22:20:16 von Russell King - ARM Linux

On Wed, Apr 27, 2011 at 02:42:57PM +0300, saeed bishara wrote:
> On Sun, Apr 17, 2011 at 7:00 PM, saeed bishara om> wrote:
> > Hi,
> > =A0 =A0 when md uses the dma for offloading xor and memcpy operatio=
ns, it
> > violates the dma-mapping API. =A0here is the scenario I'm taking ab=
out
> > (under write to degraded raid5):
> > 1. ops_run_prexor sends xor operation from buffers A and B, and the
> > destination is A.
> > 2. ops_run_biodrain: sends mempcy operation from C to B.
> > 3. ops_run_reconstruct5: sends xor operation from A and B, and the
> > destination is A again.
> >
> > in step 1, the async tx maps A using dma_map_page, and in step 3, i=
t
> > maps again the same buffer. but, if the request from step 1 still
> > being handled the dma engine, then we end with a case where the buf=
fer
> > mapped while it still belongs to the dma hw.
> > when the arch is ARMv6/SMP mode (without io coherency), the cache
> > maintenance involves read/write access to the buffers, that means, =
the
> > second mapping above may access the buffer(with read/write) while t=
he
> > dma is writing to it!!.
> >
> Russell/Dan,
> can you have a look into this issue? what I see here is that the
> raid stack issues dma_map_page to a buffer that still owned by DMA.

I already mentioned this issue to Dan, and pointed out that it's
a violation of the buffer ownership rules. I don't remember clearly
what the outcome of it is, but there's not a lot which can be done at
architecture level about it.

I think the buffer mapping was going to be moved upwards, to prevent
the multiple buffer mapping issue. I don't know if patches were
produced though (and I don't have hardware to be able to produce and
test such patches against.)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: RAID/dmaengine violates the dma-streaming API

am 28.04.2011 01:12:49 von Dan Williams

On Wed, Apr 27, 2011 at 1:20 PM, Russell King - ARM Linux
wrote:
> On Wed, Apr 27, 2011 at 02:42:57PM +0300, saeed bishara wrote:
>> On Sun, Apr 17, 2011 at 7:00 PM, saeed bishara com> wrote:
>> > Hi,
>> > =A0 =A0 when md uses the dma for offloading xor and memcpy operati=
ons, it
>> > violates the dma-mapping API. =A0here is the scenario I'm taking a=
bout
>> > (under write to degraded raid5):
>> > 1. ops_run_prexor sends xor operation from buffers A and B, and th=
e
>> > destination is A.
>> > 2. ops_run_biodrain: sends mempcy operation from C to B.
>> > 3. ops_run_reconstruct5: sends xor operation from A and B, and the
>> > destination is A again.
>> >
>> > in step 1, the async tx maps A using dma_map_page, and in step 3, =
it
>> > maps again the same buffer. but, if the request from step 1 still
>> > being handled the dma engine, then we end with a case where the bu=
ffer
>> > mapped while it still belongs to the dma hw.
>> > when the arch is ARMv6/SMP mode (without io coherency), the cache
>> > maintenance involves read/write access to the buffers, that means,=
the
>> > second mapping above may access the buffer(with read/write) while =
the
>> > dma is writing to it!!.
>> >
>> Russell/Dan,
>> =A0 =A0 =A0can you have a look into this issue? what I see here is t=
hat the
>> raid stack issues dma_map_page to a buffer that still owned by DMA.
>
> I already mentioned this issue to Dan, and pointed out that it's
> a violation of the buffer ownership rules. =A0I don't remember clearl=
y
> what the outcome of it is, but there's not a lot which can be done at
> architecture level about it.
>
> I think the buffer mapping was going to be moved upwards, to prevent
> the multiple buffer mapping issue. =A0I don't know if patches were
> produced though (and I don't have hardware to be able to produce and
> test such patches against.)
>

This is still on my plate and is waiting for me to get out from
underneath the isci driver effort. I was thinking to push it all to
md and kill the api, but then had an idea of an async_session data
structure that could automatically marshal a chain of transfers
between mapping domains. The fast path would be a chain that stays
within one mapping domain, but the infrastructure would be able to
devolve to support pathological cases like
dma_domain1->cpu->dma_domain2 chains. So md would need to manipulate
async_sessions, but it could rely on the async_tx api to handle dma
mapping details.

That's my 10,000 foot view at least.

--
Dan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" i=
n
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html