Twitter rate limiting

Question

I was given a simple task which involves getting the followers count of a number of twitter accounts. There could be 1000 twitter usernames that the application i'm creating is supposed to get each user's followers count and sum them up.

Seems like an easy task, so what I did was to use Twitter API calls (from a C# client) to get the info i need for each user. Since the info i'm after are nothing special just some public data, I used unauthenticated calls.Like this:

string target = "http:twitter.com/users/" + userName + ".xml";
Console.WriteLine("UserName: " + userName);

WebClient client = new WebClient();

Stream stream = client.OpenRead(target); StreamReader reader = new
StreamReader(stream); 
XmlTextReader xml_read = new XmlTextReader(stream);

while (xml_read.Read()) {   
     xml_read.MoveToElement();
     if (xml_read.Name == "name") {         
         Console.WriteLine("Name: " + xml_read.ReadInnerXml().ToString());
     }

    if (xml_read.Name == "followers_count"){
        Console.WriteLine("Followers: " +
                                       xml_read.ReadInnerXml().ToString());         
          Console.WriteLine(); 
      }
}

This works greatly (in terms of speed and sufficing the purpose) however it has the problem with rate limiting. i.e. i can't do more than 100 or so requests per hour .. so for 1000 user i will need 10 hours to do this simple task! ..

I tried to go a different path.. since the info i needed are public I decided to download the html pages of each twitter account to get the followers (and real name). Like so:

WebRequest myWebRequest = WebRequest.Create("http://twitter.com/" + userName);
WebResponse myWebResponse = myWebRequest.GetResponse();
Stream ReceiveStream = myWebResponse.GetResponseStream();
Encoding encode = System.Text.Encoding.GetEncoding("utf-8");
StreamReader readStream = new StreamReader(ReceiveStream, encode);
string strResponse = readStream.ReadToEnd();

Then using simple string parsing to get the followers count and real name.

This solution is valid.. however downloading the whole html file (300kb+ in size) is not really a smart way to do it. PLUS it takes roughly 2hrs to complete the task.

What am I asking for ?

I'm hoping/confident there is a smarter and more efficient way to get public info from Twitter.

Answer 1

Maybe you can use this solution: https://dev.twitter.com/docs/api/1/get/users/lookup

It gives you the ability to request data for up to 100 usernames (comma separated) like this: https://api.twitter.com/1/users/lookup.json?screen_name=bloodyairtimer,geertvdc

You can decide yourself if you would like the result in xml or json. Maybe the size of json will be smaller than xml.

Answer 2

I've the same problem. I asked twitter support via developer forum, but I don't receive a usefull response. Every API method has a rate limit, in general if you are authenticated you can have a greater one (150 for public method, 350 for authenticated) but it's not a great value.

If you can use more than one IP address, you can have different request counts, but it's better if you call an API with authentication because the count in this case is not for IP but it's for user.

The twitter rate limit policy is defined here: https://dev.twitter.com/docs/rate-limiting/faq

Answer 3

you could also use a service like Gnip

they have stored all of twitter!