Skip to content

Month: September 2008

Developing Webbots, Spiders and Screen Scrapers with PHP cURL

For a long time i wanted to write detailed tutorial on how to use PHP and cURL to create bots and i am still going to do that, but while browsing Amazon i found very interesting book related to this topic A Guide to Developing Internet Agents with PHP/CURL written by Mike Schrenk. I did not read this book yet, but it looks very promising and interesting … atleast to me, because it is not yet-another-php-for-beginners, but a book that focuses and deeply explores one single topic – creating webbots.

In my opinion knowing how to create web bots is one of the most important web programmer skills, there is so much data on the internet that we need bots to take full adventage of the resources on the web and automate day to day boring online tasks. More over you will be surprised how often you will be asked on job interview if you can write a webbot. Continue reading Developing Webbots, Spiders and Screen Scrapers with PHP cURL

Detecting search engine bots with PHP

I have to admit i am writing this post because i will publish soon big article about cURL extension and show you how to use it to create bots which would be even able to login to web pages and do some actions there, actually nothing illegal, but still i fill the need to tell you how to detect “good” (search engine bots) and “bad” (scrappers) bots and how to protect from them. Continue reading Detecting search engine bots with PHP