Comments on: Executing multiple curl requests in parallel with PHP and curl_multi_exec

By: Andrew

Andrew — Mon, 05 Dec 2011 21:09:37 +0000

You have a problem in the loop calling curl_multi_exec(). You need to either introduce a call to usleep() or curl_multi_select() to prevent PHP from just endlessly calling curl_multi_exec() and eating up all available CPU time. If you make a request that will take a long time to complete you can see this happening just by watching the output of top. Preferably you should use curl_multi_select() after you stop receiving CURLM_CALL_MULTI_PERFORM from curl_multi_exec(). Basically what this does is prevent PHP from calling curl_multi_exec() before you’ve gotten data back from one of the requests so instead of calling it 1000x a second while you’re waiting on network I/O it’ll just wait for the network I/O to complete before it calls curl_multi_exec() because nothing could have possibly changed before curl_multi_select() returns. I hope I made this clear enough, right now your script just sits around using %100 CPU time while waiting for a response. There’s no need for it when you can just sleep until there’s some change to the status of the multi handle.

By: Stu

Stu — Sun, 27 Nov 2011 23:14:07 +0000

I was wondering what kind of techniques you guys are using for mining the data retreived? I have built an array of urls, then retreive the pages using curl_multi_getcontent, writing the content to an array then mining the array, appending new lines with the extracted data I want BUT have started to run into memory issues.

Am I approaching this all wrong?

By: jon

jon — Thu, 10 Nov 2011 06:00:24 +0000

Stu, I don’t see why not. Looking at the docs I saw this:

// Create a DOM object from a string
$html = str_get_html(‘Hello!‘);

That seems like it’ll do the trick.

By: Stu

Stu — Wed, 09 Nov 2011 20:25:58 +0000

This article was a pleasure to read, thank you all so much for the additional comments, examples and links.

I have been using the Simple HTML DOM Parser requesting pages with file_get_html. I am finding it to be good, but slower than I would like it to be, so when I finally found this article about processing multiple requests in parallel I was excited.

My experience with the extracting the information I want from each page took me to using the Simple HTML DOM Parser as it seemed a great deal more tolerant than other methods.

Can I mix the usage of curl_multi_getcontent and the Simple HTML DOM Parser or is that just insaine? I am new to scraping with PHP (Can you tell?)

By: jayjay

jayjay — Tue, 23 Aug 2011 02:42:08 +0000

Maybe third time lucky ~

//===================================================
#PageLoad Timer: Part A (Top segment)
$starttime = microtime();
$startarray = explode(” “, $starttime);
$starttime = $startarray[1]+$startarray[0];
//===================================================

$nodes = array(
“http://www.iana.org/domains/example/”,
“http://www.php.net/”,
“http://www.search.com”,
“http://ac.com/”,
“http://www.goole.com/”,
“http://ad.com/”,
“http://www.phpcoders.com/”,
“http://www.ah.com/”,
“http://www.javacoders.com/”,
“http://www.rustyrazorblade.com/”,
“http://www.al.com/”,
“http://www.cars.com/”);

$node_count = count($nodes);
$curl_arr = array();
$master = curl_multi_init();
for($i=0; $i0);
echo “RESULTS:”;
for($i=0; $i<$node_count; $i++) {
$results = curl_multi_getcontent($curl_arr[$i]);
$start = strpos($results,"”);
$end = strpos($results,”",$start);
$titles = substr($results, $start, $end-$start);
echo “Titles: |”, trim(strip_tags(str_replace(“\n”, “”, $titles))), “|”;
}

//===================================================
#PageLoad Timer: Part B (Bottom segment – output):

echo “============================================================”;
$endtime = microtime();
$endarray = explode(” “, $endtime);
$endtime = $endarray[1]+$endarray[0];
$totaltime = $endtime-$starttime;
$totaltime = round($totaltime,6);
echo “Pageload time took $totaltime seconds.”.PHP_EOL;
echo”============================================================”;

enjoy~

By: jayjay

jayjay — Tue, 23 Aug 2011 02:28:27 +0000

By: jayjay

jayjay — Tue, 23 Aug 2011 02:24:25 +0000

@DarkCoder [..]Both my above samples are displayed wrong. There seem to be a problem here posting code.[..]
codes posted on wordpress blogs (like this one ) are salted to prevent them executing. When coping code from the examples double and single quotes need to be replaced with their genuine ASCII counterparts. Your code has issues too here: for($i = 0; $i 0); needs to be fixed and the usage of regular expressions for title matching is greedy on time and resources. Here’s a better way to to it (though just as crude – example proposes only )

$nodes = array(
“http://www.rustyrazorblade.com/”,
“http://www.iana.org/domains/example/”,
“http://www.php.net/”,
“http://www.search.com”,
“http://ac.com/”,
“http://www.goole.com/”,
“http://ad.com/”,
“http://www.phpcoders.com/”,
“http://www.ah.com/”,
“http://www.javacoders.com/”,
“http://www.al.com/”,
“http://www.cars.com/”);

$node_count = count($nodes);
$curl_arr = array();
$master = curl_multi_init();
for($i=0; $i0);
echo “RESULTS:”;
for($i=0; $i<$node_count; $i++) {
$results = curl_multi_getcontent($curl_arr[$i]);
$start = strpos($results, "”);
$end = strpos($results, “”, $start);
$titles = substr($results, $start, $end-$start);
echo “Titles: |”, trim(strip_tags(str_replace(“\n”, “”, $titles))), “|”;
}

//===================================================
#PageLoad Timer: Part B (Bottom segment – output):

echo “============================================================”;
$endtime = microtime();
$endarray = explode(” “, $endtime);
$endtime = $endarray[1]+$endarray[0];
$totaltime = $endtime-$starttime;
$totaltime = round($totaltime,6);
echo “Pageload time took $totaltime seconds.”.PHP_EOL;
echo “============================================================”;

enjoy

By: astaza

astaza — Sat, 26 Mar 2011 04:07:55 +0000

hello, when you use this function you must know it’s will take alot of cpu and memory be carfull,
thanks

By: jay johnston

jay johnston — Tue, 22 Mar 2011 03:19:47 +0000

thanks – this totally did the trick!

By: David

David — Thu, 16 Dec 2010 12:10:53 +0000

Thanks. This is exactly what I needed.