Recently i was browsing WordTracker website, and i started to wander: how do they get keywords statistics which no one else has? So i decided to do a little research and i found out that “All search terms are collected from the major metacrawlers - Dogpile and Metacrawler”.
I went to Dogpile website but, on the first sight there seemed to be none statistics that i would be interested to, from all the links there only one catched my eye it was Search Spy … i launched it.
It turned out to be a Flash application witch displays what people are currently searching for thru dogpile, but still i cannot get content directly from Flash application it is impossible, however, on the other side Flash cannot get such information from the server so there has to be some kind of third party application which delivers keywords.
Technically speaking, Flash app is application running on my PC because it was downloaded with HTML content and executed in web browser, so if it is getting any data (and i am sure it is) from some hidden third party then it MUST be thru HTTP protocol.
Fortunately i have Live HTTP Headers plugin installed in my Firefox so i opened and after few seconds i got info that there was a request to dogpile innerpage, as you can see it is XML page with list of queries which were made recently in Dogpile, so this is where WordTracker is getting their keywords from? I do not know maybe they have some other agreement with Dogpile and Metacrawler, but all in all … i do not care and i am not going to investigate that any further.
After 15 seconds there was the same request, made to the same Dogpile website to get list of new queries, i am not sure but probably they are making periodical request to this page, i mean there is the same interval between requests (it is logical that they make constantly making requests to get new data which is added every second).
So, now you know where to get keywords like WordTracker has, now you can do the same investigation for MetaCrawler (they are also using Search Spy), find their “secret” XML file with fresh queries, and then maybe a little competition for WordTracker? 
