Query regarding Copy-on-write

Hi,

When a process forks, every resource of the parent, including the
virtual memory is copied to the child process. The copying of VM uses
copy-on-write(COW). I know that COW comes when a write request comes,
and then the copy is made. Now my query follows:

How will the copy be distributed. Whether giving the child process a new
copy of VM be permanent or whether they will be merged anywhere? And
shouldn't the operations/updations by one process be visible to the
other which inherited the copy of the same VM?

How can this work? Can someone please help me on this regard?


--
With regards,

Jagadeesh Bhaskar P
R&D Engineer
HCL Infosystems Ltd
Pondicherry
INDIA

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

Jagadeesh Bhaskar P a =E9crit :
> Hi,
>=20
> When a process forks, every resource of the parent, including the
> virtual memory is copied to the child process. The copying of VM uses
> copy-on-write(COW). I know that COW comes when a write request comes,
> and then the copy is made. Now my query follows:
>=20
> How will the copy be distributed.

If a child modify a page that is shared with his parent, this page=20
will be copied to a new page. Then his page table will be modified to=20
point to this new page and reloaded in the MMU. Only then the running=20
process will normally modify his own copy of that page.

It works the same way if it is the parent that modify the page.=20
Otherwise we would have a problem if the parent have many childs. It=20
also simplify things a lot for SMP systems.


Be carefull: "page" and "page table" are 2 very different things.

"Page table" will point to a page struct
"page" is a struct that point to a physical page, or to a swap area=20
(if I remember well).
Virtual memory is the address seen by the program. Two identical=20
virtual adresses will normally point to different physical locations.


Note that each process have his own copy of his memory page table=20
since the fork. And the page table will typically point to many memory=
=20
pages.

The shared pages pointed by the 2 page tables are only thoses where=20
the data are still unmodified since the fork.


> Whether giving the child process a new
> copy of VM be permanent or whether they will be merged anywhere?

Its permanent at the page level. Anyway, once modified, it is very=20
unlikely that the page will ever be identical again.

Also, they never really shared the same virtual memory space since=20
they each have their own page tables. Their pointers use the same=20
virtual addresses, but they are resolved to the same physical pages onl=
y=20
for the pages that are still identicals.


I'm not sure if you understand properly how the virtual memory is=20
working at the processor level, but it is very important if you want to=
=20
understand how COW is working.

> And
> shouldn't the operations/updations by one process be visible to the
> other which inherited the copy of the same VM?
>=20

No, it must be completelly transparent for the process and they=20
should not be able to see the modifications (otherwise we have a=20
security bug).

You must know that the processor use the MMU to resolve the memory=20
page physical address. Since each process have his own memory table, a=
=20
pointer to an identical virtual adress usually point to a different=20
physical address. So it means that once a shared page is copied and=20
unlinked by modifying the page table of one of the process, the same=20
pointer with the same pointing adress will now point to 2 different=20
physical memory location. All of that magic is done thanks to the MMU.


I don't know if the kernel is doing COW at all in the case of a=20
MMU-less processor (usually a microcontroler).

> How can this work? Can someone please help me on this regard?
>=20

I think I gave a good overview of the question. I tried to keep=20
things as simple as I could, at the cost of not giving an exact=20
description of how Linux handle COW. For example, there is in fact 3=20
levels of page table indirections in Linux, but it was not necessary to=
=20
understand COW.


Simon Valiquette

---
Often, the source contains important documentation
- Slackware FAQ, 1993


-
To unsubscribe from this list: send the line "unsubscribe linux-newbie"=
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

Jagadeesh Bhaskar P wrote:
> Hi,
>
> When a process forks, every resource of the parent, including the
> virtual memory is copied to the child process. The copying of VM uses
> copy-on-write(COW). I know that COW comes when a write request comes,
> and then the copy is made. Now my query follows:
>
> How will the copy be distributed. Whether giving the child process a new
> copy of VM be permanent or whether they will be merged anywhere? And
> shouldn't the operations/updations by one process be visible to the
> other which inherited the copy of the same VM?
>
> How can this work? Can someone please help me on this regard?
>
>

Standard *nix forking does not allow that kind of interaction between the data of
parent and child processes. Pipes are designed for that kind of interprocess
communication.

POSIX threads do run in the same address space - it requires a bit more
programming effort, but it seems to fit what you're asking for a little better.

http://www.cs.nmsu.edu/~jcook/Tools/pthreads/pthreads.html

Is a decent overview on pthreads. I haven't used them - nothing I've written
needed that kind of interactivity.

Not every computer system can run pthreads on Linux - i686 or higher, SPARC, and
SPARC64 are the only ones I *know* have pthreads working (i. e. I own the machine
and pthreaded Linux apps work on them). i586 and earlier PC's lack some
fundamental capabilities necessary to make threading work.

Anyone else care to take it from here?

Jim
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

Simon Valiquette wrote:
> Jagadeesh Bhaskar P a =E9crit :

>> How can this work? Can someone please help me on this regard?
>>
>=20
> I think I gave a good overview of the question. I tried to keep=20
> things as simple as I could, at the cost of not giving an exact=20
> description of how Linux handle COW. For example, there is in fact 3=
=20
> levels of page table indirections in Linux, but it was not necessary =
to=20
> understand COW.
>=20

4-level page tables just were merged into the kernel mainline as of 2.6=
10-rc2 -=20
mostly to support the big iron that has almost-terabyte-level system me=
mory (the=20
SGI Altix, some of the big stuff from IBM, some other NUMA machines).

Jim
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie"=
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

Hi again,
I got the concept. I had no problem understanding the explanation...I
already knew the concepts of virtual memory and paging so that was not =
a
problem.

According to ur reply, after the sharing, will always the range of
addresses generated by the process, ie the virtual addresses be the sam=
e
of the parent, even if the physical memory pointed to by that is
different.

So basically, is it that sharing VM imply that the range of addresses
is shared, and physical pages pointed to by them may or maynot be the
same?

Please do help, and thanks for the previous elaborate explanation!

On Sat, 2004-11-20 at 19:29, Simon Valiquette wrote:
> Jagadeesh Bhaskar P a =E9crit :
> > Hi,
> >=20
> > When a process forks, every resource of the parent, including the
> > virtual memory is copied to the child process. The copying of VM us=
es
> > copy-on-write(COW). I know that COW comes when a write request come=
s,
> > and then the copy is made. Now my query follows:
> >=20
> > How will the copy be distributed.
>=20
> If a child modify a page that is shared with his parent, this page=
=20
> will be copied to a new page. Then his page table will be modified t=
o=20
> point to this new page and reloaded in the MMU. Only then the runnin=
g=20
> process will normally modify his own copy of that page.
>=20
> It works the same way if it is the parent that modify the page.=20
> Otherwise we would have a problem if the parent have many childs. It=20
> also simplify things a lot for SMP systems.
>=20
>=20
> Be carefull: "page" and "page table" are 2 very different things.
>=20
> "Page table" will point to a page struct
> "page" is a struct that point to a physical page, or to a swap are=
a=20
> (if I remember well).
> Virtual memory is the address seen by the program. Two identical=20
> virtual adresses will normally point to different physical locations.
>=20
>=20
> Note that each process have his own copy of his memory page table=20
> since the fork. And the page table will typically point to many memo=
ry=20
> pages.
>=20
> The shared pages pointed by the 2 page tables are only thoses wher=
e=20
> the data are still unmodified since the fork.
>=20
>=20
> > Whether giving the child process a new
> > copy of VM be permanent or whether they will be merged anywhere?
>=20
> Its permanent at the page level. Anyway, once modified, it is ver=
y=20
> unlikely that the page will ever be identical again.
>=20
> Also, they never really shared the same virtual memory space since=
=20
> they each have their own page tables. Their pointers use the same=20
> virtual addresses, but they are resolved to the same physical pages o=
nly=20
> for the pages that are still identicals.
>=20
>=20
> I'm not sure if you understand properly how the virtual memory is=20
> working at the processor level, but it is very important if you want =
to=20
> understand how COW is working.
>=20
> > And
> > shouldn't the operations/updations by one process be visible to the
> > other which inherited the copy of the same VM?
> >=20
>=20
> No, it must be completelly transparent for the process and they=20
> should not be able to see the modifications (otherwise we have a=20
> security bug).
>=20
> You must know that the processor use the MMU to resolve the memory=
=20
> page physical address. Since each process have his own memory table,=
a=20
> pointer to an identical virtual adress usually point to a different=20
> physical address. So it means that once a shared page is copied and=20
> unlinked by modifying the page table of one of the process, the same=20
> pointer with the same pointing adress will now point to 2 different=20
> physical memory location. All of that magic is done thanks to the MM=
U.
>=20
>=20
> I don't know if the kernel is doing COW at all in the case of a=20
> MMU-less processor (usually a microcontroler).
>=20
> > How can this work? Can someone please help me on this regard?
> >=20
>=20
> I think I gave a good overview of the question. I tried to keep=20
> things as simple as I could, at the cost of not giving an exact=20
> description of how Linux handle COW. For example, there is in fact 3=
=20
> levels of page table indirections in Linux, but it was not necessary =
to=20
> understand COW.
>=20
>=20
> Simon Valiquette
>=20
> ---
> Often, the source contains important documentation
> - Slackware FAQ, 1993
>=20
>=20
> -
> To unsubscribe from this list: send the line "unsubscribe linux-newbi=
e" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.linux-learn.org/faqs
--=20
With regards,

Jagadeesh Bhaskar P
R&D Engineer
HCL Infosystems Ltd
Pondicherry
INDIA

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie"=
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

Jagadeesh Bhaskar P a =E9crit :
> Hi again,
> I got the concept. I had no problem understanding the explanation..=
I
> already knew the concepts of virtual memory and paging so that was no=
t a
> problem.
>=20
> According to ur reply, after the sharing, will always the range of
> addresses generated by the process, ie the virtual addresses be the s=
ame
> of the parent, even if the physical memory pointed to by that is
> different.
>=20

Yes, otherwise the pointers would point to the wrong adresses and th=
e=20
program will probably quickly segfault.

> So basically, is it that sharing VM imply that the range of address=
es
> is shared, and physical pages pointed to by them may or maynot be the
> same?
>=20

The range of virtual addresses is reused, not shared, but I guess it=
=20
is just a question of using the good words (you are probably also a=20
non-native english speakers). But yes that's the basic idea.

> Please do help, and thanks for the previous elaborate explanation!
>=20

When I first looked inside the Linux kernel, I started by trying to=20
understand how it handled memory. I finally realized how crazy I was,=20
and switched to an easier part. But by the time, I had learned quite a=
=20
lot about VM, and also made a minimalist OS for a MC68000 microcontrole=
r=20
(only basic process support without even support for a filesystem).=20
That's also probably why I don't think the actual kernel support=20
copy-on-write on MMU-less processor, as I can't think of any good and=20
efficient implementation for that.


Simon Valiquette

---
The memory management on the PowerPC can be used
to frighten small children.
- Linus Torvalds

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie"=
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

On Mon, 2004-11-22 at 12:01, Simon Valiquette wrote:

> The range of virtual addresses is reused, not shared, but I guess it
> is just a question of using the good words (you are probably also a
> non-native english speakers). But yes that's the basic idea.
Yes, Im a non-native English speaker, and I got ur idea well and clear.
Thanks!!


> and also made a minimalist OS for a MC68000 microcontroler
> (only basic process support without even support for a filesystem).


I too want to learn something like a minimalist OS. Where to start
digging?

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

Jagadeesh Bhaskar P a =E9crit :
>=20
>>and also made a minimalist OS for a MC68000 microcontroler=20
>>(only basic process support without even support for a filesystem).=20
>=20
>=20
>=20
> I too want to learn something like a minimalist OS. Where to start
> digging?
>=20

I'm not too sure. In my case, I started mixing assembly language an=
d=20
Pascal about 15 years ago while quite young. It really helps :o)



I know there is things likes Nachos, OSP or Minix (yes, that's the=20
one used by Linus) for learning how to write operating systems. I=20
personnally prefer a real OS like Hurd, but if you have a teacher or=20
someone to assist you, I guess they can be usefull.


To write a real OS, you need to understant how your hardware is=20
working and to have the documentation book to know how to program it an=
d=20
manage it's interruptions. Beeing efficient in assembly language is a=20
must, even if you do most of your coding in C language, as it helps you=
=20
to understand your computer.

A book that might help you saving time, and that I found both very=20
informative and interesting is this one:

"Operating systems" from William Stallings, Prentice Hall.

It won't tell you how to code everythings, but it will explain the=20
algorithmes and ideas of differents Operating systems for doing similar=
=20
tasks. For example, it compares Posix threads with Linux, Windows and=20
Solaris threads. I found it quite complete; it even talk about RAID-6,=
=20
and my book is many years old.


Understanding as many OS as possible will certainly helps you=20
understanding differents concepts and improve your skills. OS likes=20
Inferno, QNX and Plan 9 are probably good for that.


For going back to your question, once you feel ready, you should=20
probably start by trying to write a small OS. At first, all you will=20
need to write is a sheduler, a way to start programs and a way to kill=20
them. Writing something similar to /sbin/init with a couple of built-i=
n=20
commands (for starting, showing and stoping processes) will also=20
simplify things.


Eventually, you could add (or emulate) a file system. By the way,=20
the sheduler should be activated by interruptions, otherwise you will=20
just have an improved version of DOS 1.0 ;o)


Simon Valiquette

---
win-nt from the people who invented edlin
(By mwikholm@at8.abo.fi, MaDsen Wikholm)

-
To unsubscribe from this list: send the line "unsubscribe linux-newbie"=
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Query regarding Copy-on-write

On Mon, 22 Nov 2004 06:48:00 -0500, Simon Valiquette =
wrote:
> Jagadeesh Bhaskar P a =E9crit :
> >
> >>and also made a minimalist OS for a MC68000 microcontroler
> >>(only basic process support without even support for a filesystem).
> >
> >
> >
> > I too want to learn something like a minimalist OS. Where to start
> > digging?
> >
>=20
> I'm not too sure. In my case, I started mixing assembly language a=
nd
> Pascal about 15 years ago while quite young. It really helps :o)
>=20
> I know there is things likes Nachos, OSP or Minix (yes, that's the
> one used by Linus) for learning how to write operating systems. I
> personnally prefer a real OS like Hurd, but if you have a teacher or
> someone to assist you, I guess they can be usefull.
>=20
> To write a real OS, you need to understant how your hardware is
> working and to have the documentation book to know how to program it =
and
> manage it's interruptions. Beeing efficient in assembly language is =
a
> must, even if you do most of your coding in C language, as it helps y=
ou
> to understand your computer.
>=20
> A book that might help you saving time, and that I found both very
> informative and interesting is this one:
>=20
> "Operating systems" from William Stallings, Prentice Hall.
>=20
> It won't tell you how to code everythings, but it will explain the
> algorithmes and ideas of differents Operating systems for doing simil=
ar
> tasks. For example, it compares Posix threads with Linux, Windows an=
d
> Solaris threads. I found it quite complete; it even talk about RAID-=
6,
> and my book is many years old.
>=20
> Understanding as many OS as possible will certainly helps you
> understanding differents concepts and improve your skills. OS likes
> Inferno, QNX and Plan 9 are probably good for that.
>=20
> For going back to your question, once you feel ready, you should
> probably start by trying to write a small OS. At first, all you will
> need to write is a sheduler, a way to start programs and a way to kil=
l
> them. Writing something similar to /sbin/init with a couple of built=
-in
> commands (for starting, showing and stoping processes) will also
> simplify things.
>=20
> Eventually, you could add (or emulate) a file system. By the way,
> the sheduler should be activated by interruptions, otherwise you will
> just have an improved version of DOS 1.0 ;o)
>=20
> Simon Valiquette
>=20
> ---
> win-nt from the people who invented edlin
> (By mwikholm@at8.abo.fi, MaDsen Wikholm)



=46urthermore,
There are a lot of resources in the internet. I too started writing
an OS few years back. Those resources were very much helpful.

www.osdev.org
www.nondot.org/~sabre/os
www.mega-tokyo.org/forum
http://my.execpc.com/~geezer/os/
http://mega-tokyo.com/osfaq2/
sources of Linux, Mach, e.t.c.
(google)
http://www.onesmartclick.com/rtos/rtos.html (if you are concerned with =
rtos)

regards manish


--=20
Manish Regmi
-
To unsubscribe from this list: send the line "unsubscribe linux-newbie"=
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs