Checking URLS across a cluster.

------_=_NextPart_001_01C65C93.4ECC4735
Content-Type: text/plain;
charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

Hi

I am trying to check that a certain URL is available across
a cluster. I can not change the requested URL to contain the IP address
of each cluster member as it needs to still be of the form
http://myhostsite.domain/mypage. Is there anyway to set up the request
so that it can request this URL from 3 different hosts. =20

=20

I think I read in the documentation that LWP automatically set up the
"host" before passing the request onto the network.

I expect it just does a DNS lookup on the host portion of the URL
provided, and this will just give back the virtual IP address of the
cluster.

Is there anyway to override this and send the GET request to individual
members of the cluster ?

=20

Regards Jamie=20


------_=_NextPart_001_01C65C93.4ECC4735--

Re: Checking URLS across a cluster.

--Apple-Mail-1-1011423321
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed


On Apr 10, 2006, at 6:38 AM, J.Slee wrote:

> I expect it just does a DNS lookup on the host portion of the URL
> provided, and this will just give back the virtual IP address of the
> cluster.

What I'm hearing you say is that the DNS is round-robin DNS, so that
there are three possible IP addresses that come back for foo.com.

There's no way that a DNS client can force a given DNS request to
come back with a given IP, at least that I'm aware of. The issue is
one of DNS, not of Mechanize.

--
Andy Lester => andy@petdance.com => www.petdance.com => AIM:petdance




--Apple-Mail-1-1011423321--

Re: Checking URLS across a cluster.

--NzB8fVQJ5HfG6fxh
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On 2006-04-10, Andy Lester wrote:

> On Apr 10, 2006, at 6:38 AM, J.Slee wrote:
>
> > I expect it just does a DNS lookup on the host portion of the URL
> > provided, and this will just give back the virtual IP address of the
> > cluster.
>
> What I'm hearing you say is that the DNS is round-robin DNS, so that
> there are three possible IP addresses that come back for foo.com.

In that case, Net::DNS could get all three IPs. It's only when forced
to return a single IP that the local resolver takes the first one in the
list (the DNS server sorts the list round-robin).

Alternatively, there may only be one IP in the DNS, that is, some form
of host-level clustering or load balancer aka layer X switch. In this
case, knowledge of the separate IPs is local and specific.

What you was is most definitely possible, for example, if the page
http://www.example.org/test/ contained the string "NODE-IS-WORKING",
then the following ought to work:

for ip in 192.0.2.10 192.0.2.11 192.0.2.12 ; do
echo -e "GET /test/ HTTP/1.0\nHost: www.example.org\n\n" \
| netcat $ip http \
| grep -q "NODE-IS-WORKING" \
|| echo Node $ip is down
done

This is an ugly shell script hack. Someone who knows WWW::Mechanize
and/or LWP:: better than I might be able to give hints on how to use
these without this ugly hacking.

This is what Nagios's "check_http" plugin does to check webservers - the
IP and hostname are separate parameters. Look at http://www.nagios.org/
As full blown monitoring, it might be what you're after.

--
Mark Suter http://zwitterion.org/ | I have often regretted my
| speech, never my silence.
mobile 0411 262 316 - gpg 2C71D63D | Xenocrates (396-314 B.C.)

--NzB8fVQJ5HfG6fxh
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Visit keyservers or http://zwitterion.org/keys/

iD8DBQFEOmHgRYso2ixx1j0RAm+qAJ9SVMtuWBag2FXaJyBB4fWS+/ETMQCf fktm
JYj7lrkxHVsWxGC/RKD+vtc=
=yE1B
-----END PGP SIGNATURE-----

--NzB8fVQJ5HfG6fxh--

RE: Checking URLS across a cluster.

------_=_NextPart_001_01C65CA5.D4283035
Content-Type: text/plain;
charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

> What I'm hearing you say is that the DNS is round-robin DNS, so that
there are three possible IP addresses that come back for foo.com. =20

=20

No, I obviously am not being clear. The DNS lookup comes back with one
address from the lookup, which is the virtual IP address of the cluster.

I want to be able to avoid using DNS lookup and tell the LWP module when
it does a GET request to use the specific IP address of each host in the
cluster, so I know that I have checked each one as being available and
placed an entry in the ISS log file. =20

=20

Unfortunately I can't replace the host part of the URL with the
specific IP address for each host as I need to have the URL looking like
http://hostname.mydomain/ as it is
redirected to a specific virtual web site using host headers.

=20

Regards Jamie

=20

________________________________

From: Andy Lester [mailto:andy@petdance.com]=20
Sent: 10 April 2006 14:14
To: J.Slee
Cc: libwww@perl.org
Subject: Re: Checking URLS across a cluster.

=20

=20

On Apr 10, 2006, at 6:38 AM, J.Slee wrote:





I expect it just does a DNS lookup on the host portion of the URL

provided, and this will just give back the virtual IP address of the

cluster.

=20

What I'm hearing you say is that the DNS is round-robin DNS, so that
there are three possible IP addresses that come back for foo.com. =20

=20

There's no way that a DNS client can force a given DNS request to come
back with a given IP, at least that I'm aware of. The issue is one of
DNS, not of Mechanize.

=20

--

Andy Lester =3D> andy@petdance.com =3D> www.petdance.com =3D> =
AIM:petdance

=20





=20


------_=_NextPart_001_01C65CA5.D4283035--

Re: Checking URLS across a cluster.

Could you create a script that added/modified entries to your hosts
file that forced the domain you're trying to reach to have a specific
IP address?

On 4/10/06, J.Slee wrote:
> > What I'm hearing you say is that the DNS is round-robin DNS, so that
> there are three possible IP addresses that come back for foo.com.
>
>
>
> No, I obviously am not being clear. The DNS lookup comes back with one
> address from the lookup, which is the virtual IP address of the cluster.
>
> I want to be able to avoid using DNS lookup and tell the LWP module when
> it does a GET request to use the specific IP address of each host in the
> cluster, so I know that I have checked each one as being available and
> placed an entry in the ISS log file.
>
>
>
> Unfortunately I can't replace the host part of the URL with the
> specific IP address for each host as I need to have the URL looking like
> http://hostname.mydomain/ as it is
> redirected to a specific virtual web site using host headers.
>
>
>
> Regards Jamie
>
>
>
> ________________________________
>
> From: Andy Lester [mailto:andy@petdance.com]
> Sent: 10 April 2006 14:14
> To: J.Slee
> Cc: libwww@perl.org
> Subject: Re: Checking URLS across a cluster.
>
>
>
>
>
> On Apr 10, 2006, at 6:38 AM, J.Slee wrote:
>
>
>
>
>
> I expect it just does a DNS lookup on the host portion of the URL
>
> provided, and this will just give back the virtual IP address of the
>
> cluster.
>
>
>
> What I'm hearing you say is that the DNS is round-robin DNS, so that
> there are three possible IP addresses that come back for foo.com.
>
>
>
> There's no way that a DNS client can force a given DNS request to come
> back with a given IP, at least that I'm aware of. The issue is one of
> DNS, not of Mechanize.
>
>
>
> --
>
> Andy Lester =3D> andy@petdance.com =3D> www.petdance.com =3D> AIM:petdanc=
e
>
>
>
>
>
>
>
>
>
>
>

RE: Checking URLS across a cluster.

>Could you create a script that added/modified entries to your hosts
>file that forced the domain you're trying to reach to have a specific
>IP address?


I have just been thinking about that, but I am looking at doing this for
about 20 - 30 sites and modifying the hosts file and probably doing a
flush of the DNS cache between each check is a little heavy handed. I
was rather hoping there was some method in the LWP module to direct the
GET request to a specific IP address, rather than letting it work it out
for itself from the supplied URL.

Jamie

P.S. the netcat suggestion looked interesting but this is running from a
Windows host - well it is presently.




==================== =====3D=
==================== =====3D=
==================== ==
====
On 4/10/06, J.Slee wrote:
>> > What I'm hearing you say is that the DNS is round-robin DNS, so
that
>>> there are three possible IP addresses that come back for foo.com.
>>>
>>
>> No, I obviously am not being clear. The DNS lookup comes back with
one
>> address from the lookup, which is the virtual IP address of the
cluster.
>>
>> I want to be able to avoid using DNS lookup and tell the LWP module
when
>> it does a GET request to use the specific IP address of each host in
the
>> cluster, so I know that I have checked each one as being available
and
>> placed an entry in the ISS log file.
>
>
>
>> Unfortunately I can't replace the host part of the URL with the
>> specific IP address for each host as I need to have the URL looking
like
>> http://hostname.mydomain/ as it is
>> redirected to a specific virtual web site using host headers.
>
>
>
>> Regards Jamie
>

Re: Checking URLS across a cluster.

On 2006-04-10 16:08, J.Slee wrote:
>
> I have just been thinking about that, but I am looking at doing this for
> about 20 - 30 sites and modifying the hosts file and probably doing a
> flush of the DNS cache between each check is a little heavy handed. I
> was rather hoping there was some method in the LWP module to direct the
> GET request to a specific IP address, rather than letting it work it out
> for itself from the supplied URL.

And more low level using Net::HTTP:

use Net::HTTP;
foreach $ip ("192.168.0.100", "192.168.0.101") {
my $s = Net::HTTP->new(Host => $ip);
$s->write_request(
GET => "http://myhostsite.domain/mypage",
# Other options if needed
);
my ($code, $mess, %h) = $s->read_response_headers;
print "Host: ", $s->host, " answered with code $code\n";
}


--
Paul Bijnens, xplanation Technology Services Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: Paul.Bijnens@xplanation.com
************************************************************ ***********
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
************************************************************ ***********

RE: Checking URLS across a cluster.

Paul
Thanks , that works a treat...

The answer it seems is use the appropriate module and life is a
lot simpler.

Jamie

-----Original Message-----
From: Paul Bijnens [mailto:paul.bijnens@xplanation.com]=20
Sent: 10 April 2006 16:22
To: J.Slee
Cc: libwww@perl.org
Subject: Re: Checking URLS across a cluster.

On 2006-04-10 16:08, J.Slee wrote:
>=20
> I have just been thinking about that, but I am looking at doing this
for
> about 20 - 30 sites and modifying the hosts file and probably doing a
> flush of the DNS cache between each check is a little heavy handed. I
> was rather hoping there was some method in the LWP module to direct
the
> GET request to a specific IP address, rather than letting it work it
out
> for itself from the supplied URL.

And more low level using Net::HTTP:

use Net::HTTP;
foreach $ip ("192.168.0.100", "192.168.0.101") {
my $s =3D Net::HTTP->new(Host =3D> $ip);
$s->write_request(
GET =3D> "http://myhostsite.domain/mypage",
# Other options if needed
);
my ($code, $mess, %h) =3D $s->read_response_headers;
print "Host: ", $s->host, " answered with code $code\n";
}


--=20
Paul Bijnens, xplanation Technology Services Tel +32 16 397.511
Technologielaan 21 bus 2, B-3001 Leuven, BELGIUM Fax +32 16 397.512
http://www.xplanation.com/ email: Paul.Bijnens@xplanation.com
************************************************************ ***********
* I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, ^^, *
* F6, quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, *
* stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, *
* PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, *
* init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... *
* ... "Are you sure?" ... YES ... Phew ... I'm out *
************************************************************ ***********

Re: Checking URLS across a cluster.

Paul Bijnens writes:

> On 2006-04-10 16:08, J.Slee wrote:
> > I have just been thinking about that, but I am looking at doing this
> > for
> > about 20 - 30 sites and modifying the hosts file and probably doing a
> > flush of the DNS cache between each check is a little heavy handed. I
> > was rather hoping there was some method in the LWP module to direct the
> > GET request to a specific IP address, rather than letting it work it out
> > for itself from the supplied URL.
>
> And more low level using Net::HTTP:
>
> use Net::HTTP;
> foreach $ip ("192.168.0.100", "192.168.0.101") {
> my $s = Net::HTTP->new(Host => $ip);
> $s->write_request(
> GET => "http://myhostsite.domain/mypage",
> # Other options if needed
> );
> my ($code, $mess, %h) = $s->read_response_headers;
> print "Host: ", $s->host, " answered with code $code\n";
> }

This out to work as well (untested):

#!/usr/bin/perl -w
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
foreach $ip ("192.168.0.100", "192.168.0.101") {
my $res = $ua->get("http://$ip/mypage", Host => "myhostsite.domain");
print "Host: $ip answered with code ", $res->code, "\n";
}

Regards,
Gisle