$nntp ->list()

[repost from elsewhere]
I've been looking at Mats Peterson's client for the last week. He has a
perl script and a resource file for things like your server name.

It occurs to me that a person would want to have another resource file of
some type that will hold the list of newsgroups. With my newsfeed, there's
100,000 of them, and I use a 56K dial-up, so it takes five or ten minutes.
When I fire up OE or binary vortex for the first time, it says "hold on, I'm
downloading the list of newsgroups. This could take a while, but you
prettymuch only have to do it once." Consequently it has to be serialized
somehow.

When I've had my minimal clients before, I would hard-code the group I was
looking at, like so:
#!/usr/bin/perl -w

use strict;
use Net::NNTP ();

use constant NUMBER_OF_ARTICLES => 100;
use constant GROUP_NAME => 'comp.lang.c';
use constant SERVER_NAME => 'news.newsgroups.com';
use constant NNTP_DEBUG => 0;

my $nntp = Net::NNTP->new(SERVER_NAME, 'Debug' => NNTP_DEBUG) or die;
my $USER = 'wade196884';
my $PASS = '';

$nntp->authinfo($USER,$PASS) or die $!;


my($article_count, $first_article, $last_article) = $nntp->group(GROUP_NAME)
or die;


# Which XOVER fields contain Subject: and From:?
my $count = 0;
my %xover_fmt = map( ($_, $count++), @{ $nntp->overview_fmt or die} );
die unless exists $xover_fmt{'Subject:'};
my $subject_offset = $xover_fmt{'Subject:'};
my $from_offset = $xover_fmt{'From:'};

my(@xover, $start_article);
RETRIEVE: while ($#xover+1 < NUMBER_OF_ARTICLES and $last_article >=
$first_article) {

# How many articles do we need? Stop retrieving if we have enough
my $articles_required = NUMBER_OF_ARTICLES - ($#xover+1) or last
RETRIEVE;


# Fetch overview information for the articles
$start_article = $last_article - ($articles_required-1);
$start_article = $start_article > $first_article ? $start_article :
$first_article;

my $xover_query = $start_article == $last_article ?
$start_article :
[$start_article, $last_article];
my $xover_ref = $nntp->xover($xover_query) or die;

# Store headers for the articles we've retrieved
foreach (sort {$b <=> $a} keys %$xover_ref) {
push @xover, $xover_ref->{$_};
}
} continue {
# Move the pointer forward to fetch previous articles
$last_article = $start_article - 1;
}

# Disconnect from the NNTP server
$nntp->quit;

print join("\n", map ($_->[$subject_offset].' from '.$_->[$from_offset],
@xover)),"\n";

# perl mats1.pl 2>text50.txt >text51.txt
__END__

In Mats' script I think he gets the group list so:
sub get_all_groups {
my $groups = $nntp->list() or do {prompt $nntp; return};
$_ = prompt('File name: '); length or return;
open(FILE, ">$_") or die "open: $!\n";
print FILE "$_!\n" foreach (sort keys %$groups);
close(FILE);
}

The nntp object is already instantiated, and it looks like the list()
method is the workhorse here. How would I get all the groups using my
bare-bones script and write them to a FILE? One answer might be that it's
right in front of my nose, but I have trouble following execution in perl.

--
"A belief in a supernatural source of evil is not necessary; men alone
are quite capable of every wickedness."

~~ Joseph Conrad (1857-1924), novelist

Re: $nntp ->list()

"Gerry Ford" wrote in
news:1208639075_1352@news.newsgroups.com:

>
> When I've had my minimal clients before, I would hard-code the
> group I was looking at, like so:

Sorry, don't have time wade through your script. I don't know much
about NNTP and the nitty gritty. Besides, it is not relevant to your
question (unless you wanted us to re-write your script for you).

> In Mats' script I think he gets the group list so:
> sub get_all_groups {
> my $groups = $nntp->list() or do {prompt $nntp; return};
> $_ = prompt('File name: '); length or return;
> open(FILE, ">$_") or die "open: $!\n";
> print FILE "$_!\n" foreach (sort keys %$groups);
> close(FILE);
> }
>
> The nntp object is already instantiated, and it looks like the
> list() method is the workhorse here. How would I get all the
> groups using my bare-bones script and write them to a FILE? One
> answer might be that it's right in front of my nose, but I have
> trouble following execution in perl.

The answer is literally in front of you. From
http://search.cpan.org/~gbarr/libnet-1.22/Net/NNTP.pm :

list ()

Obtain information about all the active newsgroups. The results
is a reference to a hash where the key is a group name and each
value is a reference to an array. The elements in this array are:-
the last article number in the group, the first article number in
the group and any information flags about the group.

I rewrote it a little bit to make it more pleasing to me. The script
takes the username and password from command line arguments. I only
did that so I don't accidentally post my credentials.

#!/usr/bin/perl -w

use strict;
use warnings;

use Net::NNTP ();

my %settings = (
SERVER_NAME => '127.0.0.1',
SERVER_PORT => 563,
NEWSRC => 'newsrc.txt',
DEBUG => 1,
);

@ARGV == 2
or die "Provide username and password on the command line\n";

@settings{ qw( USER PASS ) } = @ARGV;

my $nntp = Net::NNTP->new(
"$settings{SERVER_NAME}:$settings{SERVER_PORT}",
'Debug' => $settings{DEBUG},
) or die $@;

$nntp->authinfo(@settings{ qw( USER PASS ) }) or die $@;

my $groups = $nntp->list();

open my $newsrc, '>', $settings{NEWSRC}
or die "Cannot open '$settings{NEWSRC}': $!";

print $newsrc join( "!\n", sort keys %$groups );

close $newsrc
or die "Cannot close '$settings{NEWSRC}': $!";

__END__

Here is the debug log:

Net::NNTP>>> Net::NNTP(2.24)
Net::NNTP>>> Net::Cmd(2.29)
Net::NNTP>>> Exporter(5.62)
Net::NNTP>>> IO::Socket::INET(1.31)
Net::NNTP>>> IO::Socket(1.30_01)
Net::NNTP>>> IO::Handle(1.27)
Net::NNTP=GLOB(0x1a82274)<<< 200 bgtnsc04 Welcome ...
Net::NNTP=GLOB(0x1a82274)>>> MODE READER
Net::NNTP=GLOB(0x1a82274)<<< 200 bgtnsc04 ...
Net::NNTP=GLOB(0x1a82274)>>> AUTHINFO USER userid
Net::NNTP=GLOB(0x1a82274)<<< 381 More Authentication Required
Net::NNTP=GLOB(0x1a82274)>>> AUTHINFO PASS ....
Net::NNTP=GLOB(0x1a82274)<<< 281 Authentication Accepted
Net::NNTP=GLOB(0x1a82274)>>> LIST
Net::NNTP=GLOB(0x1a82274)<<< 215 NewsGroups Follow
Net::NNTP=GLOB(0x1a82274)>>> QUIT
Net::NNTP=GLOB(0x1a82274)<<< 205 GoodBye

C:\Temp\nntp> grep comp.lang.perl newsrc.txt
comp.lang.perl.announce!
comp.lang.perl.misc!
comp.lang.perl.moderated!
comp.lang.perl.modules!
comp.lang.perl.tk!
de.comp.lang.perl.cgi!
de.comp.lang.perl.misc!
fj.comp.lang.perl!
fr.comp.lang.perl!
han.comp.lang.perl!
pl.comp.lang.perl!

--
A. Sinan Unur <1usa@llenroc.ude.invalid>
(remove .invalid and reverse each component for email address)

comp.lang.perl.misc guidelines on the WWW:
http://www.rehabitation.com/clpmisc/

Re: $nntp ->list()

"A. Sinan Unur" <1usa@llenroc.ude.invalid> wrote in message
news:Xns9A85B4AAEA4E8asu1cornelledu@127.0.0.1...
> "Gerry Ford" wrote in
> news:1208639075_1352@news.newsgroups.com:
>
>>
>> When I've had my minimal clients before, I would hard-code the
>> group I was looking at, like so:
>
> Sorry, don't have time wade through your script. I don't know much
> about NNTP and the nitty gritty. Besides, it is not relevant to your
> question (unless you wanted us to re-write your script for you).
>
>> In Mats' script I think he gets the group list so:
>> sub get_all_groups {
>> my $groups = $nntp->list() or do {prompt $nntp; return};
>> $_ = prompt('File name: '); length or return;
>> open(FILE, ">$_") or die "open: $!\n";
>> print FILE "$_!\n" foreach (sort keys %$groups);
>> close(FILE);
>> }
>>
>> The nntp object is already instantiated, and it looks like the
>> list() method is the workhorse here. How would I get all the
>> groups using my bare-bones script and write them to a FILE? One
>> answer might be that it's right in front of my nose, but I have
>> trouble following execution in perl.
>
> The answer is literally in front of you. From
> http://search.cpan.org/~gbarr/libnet-1.22/Net/NNTP.pm :
>
> list ()
>
> Obtain information about all the active newsgroups. The results
> is a reference to a hash where the key is a group name and each
> value is a reference to an array. The elements in this array are:-
> the last article number in the group, the first article number in
> the group and any information flags about the group.
>
> I rewrote it a little bit to make it more pleasing to me. The script
> takes the username and password from command line arguments. I only
> did that so I don't accidentally post my credentials.
>
> #!/usr/bin/perl -w
>
> use strict;
> use warnings;
>
> use Net::NNTP ();
>
> my %settings = (
> SERVER_NAME => '127.0.0.1',
> SERVER_PORT => 563,
> NEWSRC => 'newsrc.txt',
> DEBUG => 1,
> );
>
> @ARGV == 2
> or die "Provide username and password on the command line\n";
>
> @settings{ qw( USER PASS ) } = @ARGV;
>
> my $nntp = Net::NNTP->new(
> "$settings{SERVER_NAME}:$settings{SERVER_PORT}",
> 'Debug' => $settings{DEBUG},
> ) or die $@;
>
> $nntp->authinfo(@settings{ qw( USER PASS ) }) or die $@;
>
> my $groups = $nntp->list();
>
> open my $newsrc, '>', $settings{NEWSRC}
> or die "Cannot open '$settings{NEWSRC}': $!";
>
> print $newsrc join( "!\n", sort keys %$groups );
>
> close $newsrc
> or die "Cannot close '$settings{NEWSRC}': $!";
>
> __END__
>
> Here is the debug log:
>
> Net::NNTP>>> Net::NNTP(2.24)
> Net::NNTP>>> Net::Cmd(2.29)
> Net::NNTP>>> Exporter(5.62)
> Net::NNTP>>> IO::Socket::INET(1.31)
> Net::NNTP>>> IO::Socket(1.30_01)
> Net::NNTP>>> IO::Handle(1.27)
> Net::NNTP=GLOB(0x1a82274)<<< 200 bgtnsc04 Welcome ...
> Net::NNTP=GLOB(0x1a82274)>>> MODE READER
> Net::NNTP=GLOB(0x1a82274)<<< 200 bgtnsc04 ...
> Net::NNTP=GLOB(0x1a82274)>>> AUTHINFO USER userid
> Net::NNTP=GLOB(0x1a82274)<<< 381 More Authentication Required
> Net::NNTP=GLOB(0x1a82274)>>> AUTHINFO PASS ....
> Net::NNTP=GLOB(0x1a82274)<<< 281 Authentication Accepted
> Net::NNTP=GLOB(0x1a82274)>>> LIST
> Net::NNTP=GLOB(0x1a82274)<<< 215 NewsGroups Follow
> Net::NNTP=GLOB(0x1a82274)>>> QUIT
> Net::NNTP=GLOB(0x1a82274)<<< 205 GoodBye
>
> C:\Temp\nntp> grep comp.lang.perl newsrc.txt
> comp.lang.perl.announce!
> comp.lang.perl.misc!
> comp.lang.perl.moderated!
> comp.lang.perl.modules!
> comp.lang.perl.tk!
> de.comp.lang.perl.cgi!
> de.comp.lang.perl.misc!
> fj.comp.lang.perl!
> fr.comp.lang.perl!
> han.comp.lang.perl!
> pl.comp.lang.perl!
I think I married the two here, yet I get no new file called 'newsrc2.txt'
in my scripts folder.

I get 3 arts and print subject and from from clc. Why doesn't this then
create the file with the whole thing? Would it maybe not create the file
until the whole thing were downloaded?
#!/usr/bin/perl -w

use strict;
use Net::NNTP ();

use constant NUMBER_OF_ARTICLES => 3;
use constant GROUP_NAME => 'comp.lang.c';
use constant SERVER_NAME => 'news.newsgroups.com';
use constant NNTP_DEBUG => 0;

my $nntp = Net::NNTP->new(SERVER_NAME, 'Debug' => NNTP_DEBUG) or die;
my $USER = '';
my $PASS = '';

my %settings = (
SERVER_NAME => '127.0.0.1',
SERVER_PORT => 563,
NEWSRC => 'newsrc2.txt',
DEBUG => 1,
);

$nntp->authinfo($USER,$PASS) or die $!;


my($article_count, $first_article, $last_article) =

$nntp->group(GROUP_NAME) or die;


# Which XOVER fields contain Subject: and From:?
my $count = 0;
my %xover_fmt = map( ($_, $count++), @{ $nntp->overview_fmt or die} );
die unless exists $xover_fmt{'Subject:'};
my $subject_offset = $xover_fmt{'Subject:'};
my $from_offset = $xover_fmt{'From:'};

my(@xover, $start_article);
RETRIEVE: while ($#xover+1 < NUMBER_OF_ARTICLES and $last_article >=

$first_article) {

# How many articles do we need? Stop retrieving if we have enough
my $articles_required = NUMBER_OF_ARTICLES - ($#xover+1) or last

RETRIEVE;


# Fetch overview information for the articles
$start_article = $last_article - ($articles_required-1);
$start_article = $start_article > $first_article ? $start_article :

$first_article;

my $xover_query = $start_article == $last_article ?
$start_article :
[$start_article, $last_article];
my $xover_ref = $nntp->xover($xover_query) or die;

# Store headers for the articles we've retrieved
foreach (sort {$b <=> $a} keys %$xover_ref) {
push @xover, $xover_ref->{$_};
}
} continue {
# Move the pointer forward to fetch previous articles
$last_article = $start_article - 1;
}

print join("\n", map ($_->[$subject_offset].' from '.$_->[$from_offset],

@xover)),"\n";
# end script that prints 3 arts from clc



my $groups = $nntp->list();

open my $newsrc, '>', $settings{NEWSRC}
or die "Cannot open '$settings{NEWSRC}': $!";

print $newsrc join( "!\n", sort keys %$groups );

close $newsrc
or die "Cannot close '$settings{NEWSRC}': $!";
# Disconnect from the NNTP server
$nntp->quit;


# perl mats3.pl 2>text50.txt >text51.txt
__END__

--
"A belief in a supernatural source of evil is not necessary; men alone
are quite capable of every wickedness."

~~ Joseph Conrad (1857-1924), novelist

Re: $nntp ->list()

"Gerry Ford" wrote in message
news:1208667309_2050@news.newsgroups.com...
>
> "A. Sinan Unur" <1usa@llenroc.ude.invalid> wrote in message

> my $groups = $nntp->list();
>
> open my $newsrc, '>', $settings{NEWSRC}
> or die "Cannot open '$settings{NEWSRC}': $!";
>
> print $newsrc join( "!\n", sort keys %$groups );
>
> close $newsrc
> or die "Cannot close '$settings{NEWSRC}': $!";
> # Disconnect from the NNTP server
> $nntp->quit;
>
>
> # perl mats3.pl 2>text50.txt >text51.txt
> __END__
Dr. Unur,

We did get it; I simply didn't wait enough. There were a lot of finer
points in the comparison of the two scripts. One was debug info. I
switched my debug info from 0 to 1 and was on my way:
http://zaxfuuq.net/perl153.jpg .

From newssrc2.txt:
zer.t-netz.modem!
zer.t-netz.modem.usr!
zer.t-netz.modem.zyxel!
zer.t-netz.netzwesen!
zer.t-netz.newsbrothers!
zer.t-netz.partnerwahl!

Mit freundlichem Gruss,
--
"A belief in a supernatural source of evil is not necessary; men alone
are quite capable of every wickedness."

~~ Joseph Conrad (1857-1924), novelist