[ANN] parallel 1.3: simple parallel processing.

Version 1.3 of ssh_parallel and parallel have been released.

This release features much better error handling in ssh_parallel, so
network errors are less likely to cause jobs to go missing.
Additionally, it allows you to specify a nice level for processes.

Same as the last time, I'm interested in any comments/critiques anyone
may have.


Original release notes (with updated URLs):


If you ever write a shell script along the lines of:

some | pipieline | bash

then you may be interested in my program parallel. The above line
would be written as:

some | pipeline | parallel 4

to perform the same task, but with 4 commands running in parallel.
There is a companion program, ssh_parallel which will distribute the
jobs over a number of machines (assuming you have passwordless SSH set
up).

Webpage: http://mi.eng.cam.ac.uk/~er258/code/index.html
Download: http://mi.eng.cam.ac.uk/~er258/code/dist/parallel-1.3.tar.gz

-Ed

--
(You can't go wrong with psycho-rats.)(http://mi.eng.cam.ac.uk/~er258)

/d{def}def/f{/Times s selectfont}d/s{11}d/r{roll}d f 2/m{moveto}d -1
r 230 350 m 0 1 179{ 1 index show 88 rotate 4 mul 0 rmoveto}for/s 12
d f pop 235 420 translate 0 0 moveto 1 2 scale show showpage

Re: parallel 1.3: simple parallel processing.

On 4 Apr., 06:40, Edward Rosten wrote:
> Version 1.3 of ssh_parallel and parallel have been released.
>
> This release features much better error handling in ssh_parallel, so
> network errors are less likely to cause jobs to go missing.
> Additionally, it allows you to specify a nice level for processes.
>
> Same as the last time, I'm interested in any comments/critiques anyone
> may have.
>
> Original release notes (with updated URLs):
>
> If you ever write a shell script along the lines of:
>
> some | pipieline | bash
>
> then you may be interested in my program parallel. The above line
> would be written as:
>
> some | pipeline | parallel 4
>
> to perform the same task, but with 4 commands running in parallel.
> There is a companion program, ssh_parallel which will distribute the
> jobs over a number of machines (assuming you have passwordless SSH set
> up).
>
> Webpage:http://mi.eng.cam.ac.uk/~er258/code/index.html
> Download:http://mi.eng.cam.ac.uk/~er258/code/dist/parallel-1 .3.tar.gz
>
> -Ed
>

I had a peek into the sources of parallel.c and wonder about the
feature to be able to start 2 or more processes but the restriction
to not start a single one. Though, the latter may be useful if the
argument is computed instead of hard coded. It seems an unnecessary
restriction. Just curious.

A useful program, BTW.

Janis

Re: parallel 1.3: simple parallel processing.

On Apr 4, 7:22 am, Janis wrote:
> On 4 Apr., 06:40, Edward Rosten wrote:
>
>
>
> > Version 1.3 of ssh_parallel and parallel have been released.
>
> > This release features much better error handling in ssh_parallel, so
> > network errors are less likely to cause jobs to go missing.
> > Additionally, it allows you to specify a nice level for processes.
>
> > Same as the last time, I'm interested in any comments/critiques anyone
> > may have.
>
> > Original release notes (with updated URLs):
>
> > If you ever write a shell script along the lines of:
>
> > some | pipieline | bash
>
> > then you may be interested in my program parallel. The above line
> > would be written as:
>
> > some | pipeline | parallel 4
>
> > to perform the same task, but with 4 commands running in parallel.
> > There is a companion program, ssh_parallel which will distribute the
> > jobs over a number of machines (assuming you have passwordless SSH set
> > up).
>
> > Webpage:http://mi.eng.cam.ac.uk/~er258/code/index.html
> > Download:http://mi.eng.cam.ac.uk/~er258/code/dist/parallel-1 .3.tar.gz
>
> > -Ed
>
> I had a peek into the sources of parallel.c and wonder about the
> feature to be able to start 2 or more processes but the restriction
> to not start a single one. Though, the latter may be useful if the
> argument is computed instead of hard coded. It seems an unnecessary
> restriction. Just curious.

Good point. I'll change that. I think it would be useful to allow one
process to be started. That would remove special cases if you (for
example) compute the number based on something like load average. I'll
put the new version up in a few hours.

> A useful program, BTW.

Thanks. It can make a big difference since so many machines have >= 2
CPUs.


-Ed