So, these billions of events were the result of a flash game developer using Playtomic analytics for their game. The game, PiTSi [1], has over 40 million installs using Conduits toolbar app. Conduit is ranked #32 on Alexa and offers thousands of browser games. The author of this article and I are both surprised that we've never heard of Conduit. Is it unheard of to most of you?
Beyond not knowing of Conduit before... who even imagined putting a game in a toolbar? The entire game is contained within the colorful little rectangle here:
Having thought about nothing other than iPhone or web/SaSS apps in the recent months, seeing this makes me fascinated at just how much of the web (and creative outlets) I don't know.
Yeah I had no idea about this either - it is so encouraging to find new stuff online. New markets, new experiences, new ideas. I think for a lot of us, being out on the forefront of technology and innovation (especially regarding the internet), we forget what it is like to discover a new thing, and how much fun it can be just to feed our curiosity in a new way. I think if you can deliver that same feeling to your users you've got a winning app.
Whoa. I've been on the Conduit site for about 2 hours now reading up and analyzing all of the apps. It is such an interesting platform. I've never heard of it in my life. I also come to the realization that I was in a generation where a toolbar equated to harassment.
Wait, what? Oh, right, your toolbar installer gives you a follow backlink to your domain. A quick yahoo site explore of backlinks to conduit consists of 90% spam/auto generated blogs/monetization in mind sites. I saw a few reliable new sources/sites offering the toolbar. It is an addictive platform once they suck you in to installing it. Easily adding features/gadgets to your toolbar is somewhat nice. They're currently sitting at the #11 spot in google, #9 on yahoo, and #4 on bing for "toolbar".
The Conduit site itself works on a freemium model with payment for toolbar installs/usage (scary). They make money by offering paid services/support/etc.
I'm going to attempt to work on a gadget tomorrow.
so, without knowing your actual infrastructure, here are some questions/comments.
1. do you have a cache layer? the ability to batch-update multiple of the same event without actually hitting a db (or, shudder, disk I/O) can help you in general and provides an edge during huge spikes.
2. reliable monitoring is hard. you usually need an extra host to check the other hosts, typically with a very low resource service/protocol like snmp, ganglia, collectd. send alerts based on threshold or ramping. also try to benchmark your service and get numbers on the amount of memory, cpu, hps etc. that will make it tip over and set hard limits in your applications/servers.
3. i don't know your hoster or how they scale, but i'm a big fan of ultra low cost VPS's that dynamically scale. i wouldn't rely on any single hoster for my entire site, but for excess capacity there are dozens of good cheap providers you can scale out with. i don't usually look at amazon because of (iirc) the huge cost for short return. but definiely look to see if you can expand your hosters for DR.
1) The tracking stuff is static, nothing actually happens except "OK" being returned. Hit logs using a standardized format are generated from the requests which are then bundled up and sent off for processing. The processing stuff does 2 rounds of merges to reduce the volume of db work. The real problem (aside from the volume of data to be processed) was as simple as a ton of people connecting at once which is pretty much just a hardware problem/solution.
2) Yeah, I'm not sure long term what the best option on that is going to be, it's definitely something I'll be researching soon. Benchmarking pretty much just got done for me, I can comfortably do 80k concurrents on a vps, more on the dedicateds although the exact point is unknown.
3) Absolutely agree on the vps boxes. I also don't look at Amazon because of request based billing - that stuff adds up crazy-expensive compared to 'normal' hosting.
[1]http://apps.conduit.com/Conduit-and-baconOppenheim-PiTSi-app...