Spidering Hacks

Working on a project for a spider able to gather info for my books
list,
I read a bit of "Spidering Hacks" (OReilly) - which fortunately is all
in Perl.

I was deciding to use the Google API, so I looked also to "Google
Hacks".

Up to now I never registered for a Google API, so when going to the
place suggested by the book (http://www.google.com/apis/download.html)
I was redirected to http://code.google.com, where a huge number of
possible APIs are present.

Do you know where I have to look, to simply obtain the (authorization?
key?) to use the Google search api?

thanks!

Alessandro

Re: Spidering Hacks

On Nov 26, 10:28 am, "alexxx.ma...@gmail.com"
wrote:
> Working on a project for a spider able to gather info for my books
> list,
> I read a bit of "Spidering Hacks" (OReilly) - which fortunately is all
> in Perl.
>
> I was deciding to use the Google API, so I looked also to "Google
> Hacks".
>
> Up to now I never registered for a Google API, so when going to the
> place suggested by the book (http://www.google.com/apis/download.html)
> I was redirected tohttp://code.google.com, where a huge number of
> possible APIs are present.
>
> Do you know where I have to look, to simply obtain the (authorization?
> key?) to use the Google search api?
>
> thanks!
>
> Alessandro

Hi,

which API does the book prompt you to use? AFAIK, Google has both a
SOAP and an AJAX version of their search API.
Their SOAP API is no longer open to new registrations and thus cannot
be used (unless of course you already have an SOAP API code).

The AJAX API page can be found at : http://code.google.com/apis/ajaxsearch/
You can register for a key at : http://code.google.com/apis/ajaxsearch/signup.html

Hope this helps,

Spiros

ps. You might also want to check out the GoogleHack proect,
http://google-hack.sourceforge.net/

Re: Spidering Hacks

From what I just read, SOAP was just what I needed - if only I had a
key, which I havent.
And AYAX if Javascript only, right? So I'm out of luck, with Perl -
damn!

Thank you for your help...

Alessandro




Spiros Denaxas ha scritto:

> On Nov 26, 10:28 am, "alexxx.ma...@gmail.com"
> wrote:
> > Working on a project for a spider able to gather info for my books
> > list,
> > I read a bit of "Spidering Hacks" (OReilly) - which fortunately is all
> > in Perl.
> >
> > I was deciding to use the Google API, so I looked also to "Google
> > Hacks".
> >
> > Up to now I never registered for a Google API, so when going to the
> > place suggested by the book (http://www.google.com/apis/download.html)
> > I was redirected tohttp://code.google.com, where a huge number of
> > possible APIs are present.
> >
> > Do you know where I have to look, to simply obtain the (authorization?
> > key?) to use the Google search api?
> >
> > thanks!
> >
> > Alessandro
>
> Hi,
>
> which API does the book prompt you to use? AFAIK, Google has both a
> SOAP and an AJAX version of their search API.
> Their SOAP API is no longer open to new registrations and thus cannot
> be used (unless of course you already have an SOAP API code).
>
> The AJAX API page can be found at : http://code.google.com/apis/ajaxsearch/
> You can register for a key at : http://code.google.com/apis/ajaxsearch/signup.html
>
> Hope this helps,
>
> Spiros
>
> ps. You might also want to check out the GoogleHack proect,
> http://google-hack.sourceforge.net/

Re: Spidering Hacks

"alexxx.magni@gmail.com" writes:

> Do you know where I have to look, to simply obtain the (authorization?
> key?) to use the Google search api?

The same place you'd look to use their API from Ruby, Python, etc.

Did you have a Perl question?

sherm--

--
WV News, Blogging, and Discussion: http://wv-www.com
Cocoa programming in Perl: http://camelbones.sourceforge.net

Re: Spidering Hacks

Yes, I had it, and other people kindly answered it.
Seriously, however, I don't see the reason for such narrowness of
mind...
I documented myself on Spidering Hacks, a Perl-only book on such kind
of programs, and intended to find a way to develop my own using a
Google API.
If you believe this group forbids this kind of questions, it's your
opinion, and yours only

Best regards

Alessandro

Sherman Pendley ha scritto:

> "alexxx.magni@gmail.com" writes:
>
> > Do you know where I have to look, to simply obtain the (authorization?
> > key?) to use the Google search api?
>
> The same place you'd look to use their API from Ruby, Python, etc.
>
> Did you have a Perl question?
>
> sherm--
>
> --
> WV News, Blogging, and Discussion: http://wv-www.com
> Cocoa programming in Perl: http://camelbones.sourceforge.net

Google bad, Yahoo good (was Re: Spidering Hacks)

>>>>> "alexxx" == alexxx magni@gmail com writes:

alexxx> Do you know where I have to look, to simply obtain the (authorization?
alexxx> key?) to use the Google search api?

If I recall, Google withdrew that API about six months ago, replacing with a
page-embedded search result API that ensures that their ads will also show up
in the right place. Yech.

If you want programmatic access to a search result, Yahoo is now the best
player in the space, offering a variety of searches, including JSON results
for direct embedding with no server-side code required. See
http://developer.yahoo.com/search/ and Yahoo::Search in the CPAN, and my usage
of same at .

So much for Google "doing no evil". Yahoo for the win.

print "Just another Perl hacker,"; # the original!

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095

Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Spidering Hacks

[top-posting fixed. don't do that]

Quoth "alexxx.magni@gmail.com" :
> Sherman Pendley ha scritto:
> > "alexxx.magni@gmail.com" writes:
> >
> > > Do you know where I have to look, to simply obtain the (authorization?
> > > key?) to use the Google search api?
> >
> > The same place you'd look to use their API from Ruby, Python, etc.
> >
> > Did you have a Perl question?
>

> If you believe this group forbids this kind of questions, it's your
> opinion, and yours only

It's not Sherm's opinion only; see the section in the Posting Guidelines
called 'Question should be about Perl, not about the application area'.
Those guidelines were drawn up by general consensus among the regulars
here.

Ben

Re: Spidering Hacks

"alexxx.magni@gmail.com" writes:

Upside-down, and quoting signatures. Please don't do that. Have you read
the posting guidelines that appear here frequently?

> Sherman Pendley ha scritto:
>
>> "alexxx.magni@gmail.com" writes:
>>
>> > Do you know where I have to look, to simply obtain the (authorization?
>> > key?) to use the Google search api?
>>
>> The same place you'd look to use their API from Ruby, Python, etc.
>>
>> Did you have a Perl question?
>
> Yes, I had it, and other people kindly answered it.

Yes, but that was another thread - and I was one of those people. :-)

My point was that the question in *this* thread has nothing to do with Perl.
Ruby, Python, C, Java, or any other programmers all obtain their Google keys
the same way - the procedure for doing so has nothing whatsoever to do with
the language with which you intend to use it.

> Seriously, however, I don't see the reason for such narrowness of
> mind...

It's simple. There are many Perl programmers here, and only a small subset
of us may have used Google's API. But, there's nothing Perl-specific about
obtaining a key; a programmer who's done so to use the key in his Ruby app
could help you just as well as anyone here.

By asking your Google questions in a language-neutral forum that's dedicated
to such questions, and your Perl questions in a forum that's dedicated to
those, you increase your chances of getting an answer.

> If you believe this group forbids this kind of questions, it's your
> opinion, and yours only

Actually, it's not. It's the group consensus, as documented in the posting
guidelines that appear here frequently. Have you read them?

sherm--

--
WV News, Blogging, and Discussion: http://wv-www.com
Cocoa programming in Perl: http://camelbones.sourceforge.net

Re: Google bad, Yahoo good

On Mon, 26 Nov 2007 06:53:34 -0800 merlyn@stonehenge.com (Randal L. Schwartz) wrote:

RLS> If I recall, Google withdrew that API about six months ago, replacing with a
RLS> page-embedded search result API that ensures that their ads will also show up
RLS> in the right place. Yech.
....
RLS> So much for Google "doing no evil". Yahoo for the win.

Withdrawing a public API is an inconvenience to its users but hardly
qualifies as "evil" in my opinion. Let's be fair.

Ted

Re: Google bad, Yahoo good

>>>>> "Ted" == Ted Zlatanov writes:

Ted> Withdrawing a public API is an inconvenience to its users but hardly
Ted> qualifies as "evil" in my opinion. Let's be fair.

I disagree. The closest equivalent API (the client-side embedded javascript)
burdens all results with google ads. So, this seems to be a change only to
line the pocketbooks of the stockholders. Still convinced it's not evil?

print "Just another Perl hacker,"; # the original

--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095

Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

Re: Spidering Hacks

On 26 Nov, 17:27, Sherman Pendley wrote:
> "alexxx.ma...@gmail.com" writes:
>
> Upside-down, and quoting signatures. Please don't do that. Have you read
> the posting guidelines that appear here frequently?
>
> > Sherman Pendley ha scritto:
>
> >> "alexxx.ma...@gmail.com" writes:
>
> >> > Do you know where I have to look, to simply obtain the (authorization?
> >> > key?) to use the Google search api?
>
> >> The same place you'd look to use their API from Ruby, Python, etc.
>
> >> Did you have a Perl question?
>
> > Yes, I had it, and other people kindly answered it.
>
> Yes, but that was another thread - and I was one of those people. :-)
>
> My point was that the question in *this* thread has nothing to do with Perl.
> Ruby, Python, C, Java, or any other programmers all obtain their Google keys
> the same way - the procedure for doing so has nothing whatsoever to do with
> the language with which you intend to use it.
>
> > Seriously, however, I don't see the reason for such narrowness of
> > mind...
>
> It's simple. There are many Perl programmers here, and only a small subset
> of us may have used Google's API. But, there's nothing Perl-specific about
> obtaining a key; a programmer who's done so to use the key in his Ruby app
> could help you just as well as anyone here.
>
> By asking your Google questions in a language-neutral forum that's dedicated
> to such questions, and your Perl questions in a forum that's dedicated to
> those, you increase your chances of getting an answer.

Probably then it's my mistake.
I often asked for help/advice in this group, since I use Perl heavily,
and it seldom let me down - I used to feel rather at home here.
Granted, I agree that my question was not related to per se,
but to a , but I failed to see it as a big problem - I wasnt
spamming (I posted on comp.lang.perl.misc only) and I believed the
*misc leaf allowed this.


Alessandro


> > If you believe this group forbids this kind of questions, it's your
> > opinion, and yours only
>
> Actually, it's not. It's the group consensus, as documented in the posting
> guidelines that appear here frequently. Have you read them?
>
> sherm--

Re: Google bad, Yahoo good

merlyn@stonehenge.com (Randal L. Schwartz) wrote:
> >>>>> "Ted" == Ted Zlatanov writes:
>
> Ted> Withdrawing a public API is an inconvenience to its users but hardly
> Ted> qualifies as "evil" in my opinion. Let's be fair.
>
> I disagree. The closest equivalent API (the client-side embedded
> javascript) burdens all results with google ads. So, this seems to be a
> change only to line the pocketbooks of the stockholders. Still convinced
> it's not evil?

Yes. Not being a charity is not the same thing as being evil.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Re: Google bad, Yahoo good

Ted Zlatanov writes:

> Let's be fair.

Fair? This is usenet! :-)

sherm--

--
WV News, Blogging, and Discussion: http://wv-www.com
Cocoa programming in Perl: http://camelbones.sourceforge.net

Re: Google bad, Yahoo good

On Mon, 26 Nov 2007 09:01:07 -0800 merlyn@stonehenge.com (Randal L. Schwartz) wrote:

>>>>>> "Ted" == Ted Zlatanov writes:
Ted> Withdrawing a public API is an inconvenience to its users but hardly
Ted> qualifies as "evil" in my opinion. Let's be fair.

RLS> I disagree. The closest equivalent API (the client-side embedded
RLS> javascript) burdens all results with google ads. So, this seems to
RLS> be a change only to line the pocketbooks of the stockholders.

I've worked for a content delivery company and I can assure you it's not
free to provide bandwidth and server load. Furthermore, every
for-profit company's express mission is to line the pocketbooks of the
shareholders.

RLS> Still convinced it's not evil?

I'd say Yahoo's actions have been a lot more morally and legally
suspect, actually, bringing them before the US Congress a few weeks ago:

http://www.theregister.co.uk/2005/09/07/yahoo_china_dissiden t_case/
http://www.internetnews.com/breakingnews/article.php/3710916

In the context of 10 years in jail for speaking your mind, withdrawing a
public API doesn't seem all that evil.

Ted

Re: Spidering Hacks

"alexxx.magni@gmail.com" writes:

> Probably then it's my mistake.

Don't lose sleep over it - it wasn't a very big one. :-)

> I often asked for help/advice in this group, since I use Perl heavily,
> and it seldom let me down - I used to feel rather at home here.

Please feel free to continue feeling that way! Granted, my first "did you
have a Perl question" was kind of snarky, and I apologize for that. But I
honestly intended to be helpful - it's my opinion that choosing the right
group for a question is as critical a research skill as choosing the right
keywords for a Google search.

sherm--

--
WV News, Blogging, and Discussion: http://wv-www.com
Cocoa programming in Perl: http://camelbones.sourceforge.net

Re: Spidering Hacks

On 26 Nov, 20:07, Sherman Pendley wrote:
> "alexxx.ma...@gmail.com" writes:
> > Probably then it's my mistake.
>
> Don't lose sleep over it - it wasn't a very big one. :-)
>
> > I often asked for help/advice in this group, since I use Perl heavily,
> > and it seldom let me down - I used to feel rather at home here.
>
> Please feel free to continue feeling that way! Granted, my first "did you
> have a Perl question" was kind of snarky, and I apologize for that. But I
> honestly intended to be helpful - it's my opinion that choosing the right
> group for a question is as critical a research skill as choosing the right
> keywords for a Google search.
>

Well, thank you for the kind words, I feel better!

Alessandro

P.S. still unsure, however, which would have been the right
group??? ;-)

Re: Spidering Hacks

"alexxx.magni@gmail.com" writes:

> P.S. still unsure, however, which would have been the right
> group??? ;-)

If you go to the list of Google APIs here:



When you click the name of the API on the list to the left, a list of
resources related to that API appears on the right. It looks to me like
most of them have groups; I don't know if those groups are GG-only, or
if the major usenet providers are also carrying them.

sherm--

--
WV News, Blogging, and Discussion: http://wv-www.com
Cocoa programming in Perl: http://camelbones.sourceforge.net