Retrieving many RSS feeds fast?

Retrieving many RSS feeds fast?

am 30.03.2008 19:29:15 von Duke of Hazard

I have a simple scripts using magpie to retrieve upto 200 RSS feeds.
Unfortunately it takes forever as the script has to sequentially get
each feeds, wait for the response, then move on. Is it possible to
retrieve them all at once or do them in parrallel?

Re: Retrieving many RSS feeds fast?

am 30.03.2008 19:46:56 von George Maicovschi

On Mar 30, 8:29 pm, Duke of Hazard wrote:
> I have a simple scripts using magpie to retrieve upto 200 RSS feeds.
> Unfortunately it takes forever as the script has to sequentially get
> each feeds, wait for the response, then move on. Is it possible to
> retrieve them all at once or do them in parrallel?

Multithreading is not built in PHP, but you can emulate it using two
scripts and requesting one script through the other one using the
Apache ( like fopen('http://localhost/path/to/file/
second_script.php') ) and giving it the RSS feed as a parameter and
then fgets-ing the output of the script. Sorry i don't have the time
to give you more details, i'll get back to this in about an hour.

Re: Retrieving many RSS feeds fast?

am 30.03.2008 20:42:21 von George Maicovschi

OK, me again, sorry for not being more specific earlier.

You should create a script (lets say parse_rss_ext.php) that parses
the RSS and outputs the result.
Then create another script (lets say parse_rss_core) that makes calls
to the parse_rss_ext.php script providing the RSS to be parsed and
receiving the response, then doing whatever needs to be done to the
response.

The code that emulates multi-threading in PHP is the following:

$rss_list is an array containing all the RSS feeds;

[code start]
$threads=array();
foreach ($rss_list as $rss)
$threads[$rss]=fopen('http://localhost/path/to/file/
parse_rss_ext.php?rss={$rss}','r');

foreach ($threads as $thread_rss=>thread)
if ($thread)
{
while (!feof($thread))
$responses[$thread_rss].=fgets($thread);
fclose($thread);
}
[code end]

Hope this helps you. Cheers!