Announcing Followerlap

Published 15 years, 4 months past

Last week, I got an interesting inquiry from Velda Christensen:

@meyerweb *wondering just how many of your followers follow @zeldman and vice-versa*

I had no idea.  Furthermore, I didn’t know of a tool that could tell me.  So I built one: Followerlap.

As it turned out, the Twitter API made answering the specific question pretty ridiculously easy, thanks to followers/ids.  All it takes is two API requests, one for each username.  The same would be true of friends/ids, on top of which I suspect I’ll fairly shortly build a tool quite similar to Followerlap.

Since I announced Followerlap’s existence on (where else?) Twitter, I’ve had a few repeated (and not unexpected) bits of feedback.

  • Why not list the common followers?  Because followers/ids returns a list of numeric IDs.  Resolving those IDs as usernames would require one API hit per ID.  If there are 15 followers in common, that’s not such a big deal, but if there are 1,500, well, I’ll run out of my hourly allotment of API requests very quickly.  Maybe there’s a better way; if so, I’d love to hear about it, because that would be a great addition.

  • Why can’t I find out how many people follow both Stephen Fry and Shaquille O’Neal?  Past a certain number of followers, somewhere in the 200,000–250,000 range, the API just dies.  You can’t even count on getting a consistent error message back.  There are ways around this, but I didn’t want to stress the API that way, so it just fails.  Sorry.

  • How can I link to a specific comparison?  At the moment, you can’t.  I hope to make that happen soon, but I decided that a tool this simple should have a similarly simple launch.  Ship early, ship often, right?  Anyway, it’s on the list of things to add soon.  Use the new “Livelink to this result” link underneath a result.  (See update below for more.)

So that’s Followerlap.  Any other questions?  I’ll do my best to answer them in the comments, though for a number of reasons I may be slow to respond.

Update 6 Jul 09: as noted in the edited point above, livelinking of comparison results is now, um, live.  So now you can pass around results like the number of people who follow both God and the Devil (thanks to Paul M. Watson for coming up with that one!).  I call this “livelinking” because hitting a result URL will get you the very latest results for that particular comparison.  “Permalinking” to me implied that it would link to a specific result at a specific time, which the tool doesn’t do and very likely never, ever will.

Comments (29)

  1. Hm, you could build a cache of requested IDs, and a histogram of how frequently they come up, then use you spare allotted API requests to build up your local database of ID – username equivalents, starting with the more popular ones, then they can come from your cache while you keep building your dossier. Yep, eventually you could end up duplicating the whole catalog of Twitter users, which might be against their TOS though. Just an idea that may have legs.

  2. That’s really a bummer about not being able to list the common followers. Finding new people to follow would be the primary benefit of such a tool, IMO. I guess the only way to do this would be to use statuses/followers instead of followers/id. You only get 100 results per call though, so if both people are following 1,500 people, that’s 30 calls for what should take 2. Ugh.

    Also <cough>Followinglap</cough>?

  3. Thanks for this; I’ve been wanting a utility that would do this for me for a long while, now.

    Re: the comment above, I would love to see “Followinglap” too. If someone isn’t well known, but says smart stuff and has good taste in who they follow, I’ll give them a follow as well.

  4. Carlos: I really don’t want to recreate Twitter’s user catalog– that’s millions upon millions of people. I’ll give it some thought to see if there’s a lower-impact way to make things happen.

    Jason: Yeah, if the method returned usernames instead of numeric IDs, we’d be golden. I might not do the icons but I could list out, with links, all of the accounts in common. (And the ones that aren’t in common, for that matter.) I did mention the idea of a “Followinglap” in the post (the bit about friends/ids) and hope to create and launch that soon. Shouldn’t be too much effort to adapt the code.

  5. I love the name.

    @Jason, it’s not “Followinglap,” but “Followerlap.” And then think about what happens if you pronounce the w as a v (as in German). Maybe an overlapping logo would help? (If you want one, Eric, I’d be happy to make one, gratis.)

  6. Hey Eric,

    I built something similar to this a few week’s back that does list the followers in common, as well as their friends. It’s not very pretty, but it gets the job done.

    I’m kind of cheating by using App Engine and a whitelisted account to distribute the load/API calls for me, but it does work for most situations. The site was originally built to allow programmatic access to Twitter Avatars in cases like this.

    You can see @meyerweb and @zeldman’s commonality here – be forewarned though that my browser wasn’t really fond of loading 7,000+ images, heh.

  7. As some of your replies mentioned, I built something similar last year at: twitter.infinityplusone.com. The tool in mind would be TwInCommon. It’s done completely on the front-end, which makes it a bit of a hog — have a back-end version I never finished implementing because I got too busy. Consequently, I put a threshold of 1,500 or so in just-in-case (back then, the Twitter API tended to fail over that anyway).

    TwInCommon also distinguishes between follower and following. This is a pretty important distinction, I’d think, because many people are not simply comparing themselves to others; rather, they are looking for new people to follow, or wish to see if a new follower is an unknown member of their community, or just a troll/spam. (It’s a bit buggy now due to API changes, so I should prolly give it a tune-up one of these days…)

    Also have a few other Twitter API hack-tools in there I built for friends. Nothing special, but fun to do at the time.

    ~ @yoni

  8. One way to list common followers would be by using the statuses/followers, statuses/friends method (a quick dirty script I wrote few months ago http://wg.vinayraikar.com/apps/commontweeps/). Best thing about followers/ids is that it is light weight but to list followers, you will need to cache user details and that will lead to data consistency issues (Something like http://twivatar.org for usernames would be wonderful :D ).

  9. And now, I feel stupid for misunderstanding the request for “Followinglap.” That’d be neat to see, though a more descriptive name might be called for.

  10. Here’s a cool site I just heard of. Looks like Zeldman and you are trending at the same amount – http://twittercounter.com/compare/zeldman/meyerweb/all/

  11. Erik, no worries. And if you’ve the time to whip up a spiffy logo as previously mentioned, I’d be only too glad to see it!

  12. I must say I’m not too surprised to see that other people had done similar things in the past. Part of me is sorry I didn’t know they existed, but a bigger part of me is glad—because had I known of them, I wouldn’t have had an excuse to do a little programming and learning!

  13. Hehe – as a guy that’s developed a bunch of Twitter apps, there’s not one that I’ve thought of that somebody else hasn’t done already. It’s really quite amazing.

  14. I had/have similar issues with Twitual. You can only fetch follower details in batches of 100. Which is fine for most accounts. When when you start analyzing users with thousands of followers, it begins to get icky. You have to loop through, building a list, handling sporadic errors along the way, and it’s not very fast making all those HTTP requests.

    Getting an application registered helps with the API request limit. But that’s a minor problem compared to the speed and error handling issues. I’ve been considering caching ID/name mappings server-side, but like you say, attempting to basically mirror all of Twitter’s info is not a very fun prospect.

  15. Dougal: For what I’m doing, I haven’t had to register because the API limit isn’t really an issue: the default hourly limit for an IP address seems to be 100,000 hits, and since each comparison requires two hits that’s 50,000 comparisons an hour, or 13.889 comparisons per second. No problem. Even if I set up a tool to show (or integrate into Followerlap) friend-overlap statistics, that will double my API hits and halve my upper bound to 6.944 comparisons per second. I think I can handle that.

    The key there, of course, is to just present the numbers and not try to build a list of details. What I really wish is that the API had followers/usernames and friends/usernames methods. Then I could at least list and link to them all, even if I didn’t have avatars or bios or whatever. That’d be pretty cool.

  16. Tweepdiff (http://tweepdiff.com) does essentially the same thing. It shows three lists: one for in common and one for unique to each user being compared. It also lets you compare more than 2 users and you can mix and match followers/friends. As mentioned in previous comments, the API tends to fall over when you get above a certain number of friends/followers but I doubt people really want an accurate comparison of 200k followers anyway. I use memcache and a combination of the followers and statuses APIs which works ok with pagination. And yes, I too wrote the tool without checking to see if someone else had already done it, and was happy I got the experience.

    Brian

  17. Eric: Did Twitter bump it up for you or something? On my whitelisted boxes, the hourly rate limit is only 20,000 hits. Without whitelisting it’s 150.

  18. Damon: nope. account/rate_limit_Status returns an hourly-limit value of 100000 when I run the tool in “debug” mode (which, among other things, shows me a var_dump of account/rate_limit_Status). I’m not signing in as me, either, so it must be IP-bound.

  19. There have been many calls for API methods to return all the follower/friend names in one fell swoop. The problem is that the way that the Twitter datastore is structured, it’s a very expensive operation on the Twitter server-side. So it ain’t going to happen any time soon, according to what their API guys have said.

  20. Eric,

    I know that, by the rules, Damon is correct about the API limits: http://apiwiki.twitter.com/Rate-limiting. Have you outputted the rate-limit when *not* in debug mode? In fact, where did you find out about debug mode? And are you sure the TOS allow you to use it like that?

    ~ @yoni

  21. Apparently I was unclear, Jonathan. What I meant is that Followerlap has a “debug mode”, which is invoked with a URL parameter. It’s the same tool doing the same stuff, just outputting values for me to see (or triggering failure states based on values given to said parameter). Here: compare God and the Devil, with the details.

  22. Hey Eric,

    I’m really like your idea on followerlap.
    I’ve some design ideas for it. If you would be interested in it, please let me know!

    Congrats and regards.

  23. Eric,

    That’s actually what I suspected you meant, except that I was trying to find an explanation for why the rate-limit is so high. This is very weird. I just went to: http://twitter.com/account/rate_limit_status.xml, and it gave me 20000 (I’m whitelisted). My wife tried at her office, and it gave her 150. Could your app, perhaps, be hosted on a domain that has “uber”-whitelisting? (Not something mentioned in the API dox. :p)

    Dunno. Lucky you.

    ~ @yoni

  24. What does the @god & @devil comparison tell us? and then again what does it tell us when we compare these?

  25. Periods are legal characters in Twitter names, but your site chokes on them.

  26. According to the Twitter signup form, periods aren’t allowed, Aklap; only letters, numbers, and underscores. Can you point me to a username containing a period?

  27. Eric, perhaps you could leave the option of cross referencing the two lists via JavaScript.

    I mean: you run the core of it on the server, then print a hidden list of the overlapping user IDs. Then, if the browser supports JS, add a button which runs the queries based on the IDs on the client side, so they don’t use up your API limit.

    Maybe there’s something to it I’m not seeing, but would probably be a pretty simple solution to actually SEEING who’s in the overlap.

  28. I had to do this in the past too for a twitter app I created, except I was doing the opposite. If I remember correctly, I think the twitter API returns an array of IDs for all the users… then you can just put those into your own two arrays, and do some array math for whatever you’re trying to accomplish. Should only end up being a couple lines of code after the API requests are made.

    Cool little app, I love simple but useful sites like that :)

  29. I did something similar earlier this year – Worth Retweeting. Enter your name and the name of the person you’re retweeting and you’ll see how many of your followers you’ll annoy.

    I wrote it up in May this year with code examples if anyone wants them – http://shkspr.mobi/blog/index.php/2009/05/worth-retweeting/