so, without knowing your actual infrastructure, here are some questions/comments.
1. do you have a cache layer? the ability to batch-update multiple of the same event without actually hitting a db (or, shudder, disk I/O) can help you in general and provides an edge during huge spikes.
2. reliable monitoring is hard. you usually need an extra host to check the other hosts, typically with a very low resource service/protocol like snmp, ganglia, collectd. send alerts based on threshold or ramping. also try to benchmark your service and get numbers on the amount of memory, cpu, hps etc. that will make it tip over and set hard limits in your applications/servers.
3. i don't know your hoster or how they scale, but i'm a big fan of ultra low cost VPS's that dynamically scale. i wouldn't rely on any single hoster for my entire site, but for excess capacity there are dozens of good cheap providers you can scale out with. i don't usually look at amazon because of (iirc) the huge cost for short return. but definiely look to see if you can expand your hosters for DR.
1) The tracking stuff is static, nothing actually happens except "OK" being returned. Hit logs using a standardized format are generated from the requests which are then bundled up and sent off for processing. The processing stuff does 2 rounds of merges to reduce the volume of db work. The real problem (aside from the volume of data to be processed) was as simple as a ton of people connecting at once which is pretty much just a hardware problem/solution.
2) Yeah, I'm not sure long term what the best option on that is going to be, it's definitely something I'll be researching soon. Benchmarking pretty much just got done for me, I can comfortably do 80k concurrents on a vps, more on the dedicateds although the exact point is unknown.
3) Absolutely agree on the vps boxes. I also don't look at Amazon because of request based billing - that stuff adds up crazy-expensive compared to 'normal' hosting.
1. do you have a cache layer? the ability to batch-update multiple of the same event without actually hitting a db (or, shudder, disk I/O) can help you in general and provides an edge during huge spikes.
2. reliable monitoring is hard. you usually need an extra host to check the other hosts, typically with a very low resource service/protocol like snmp, ganglia, collectd. send alerts based on threshold or ramping. also try to benchmark your service and get numbers on the amount of memory, cpu, hps etc. that will make it tip over and set hard limits in your applications/servers.
3. i don't know your hoster or how they scale, but i'm a big fan of ultra low cost VPS's that dynamically scale. i wouldn't rely on any single hoster for my entire site, but for excess capacity there are dozens of good cheap providers you can scale out with. i don't usually look at amazon because of (iirc) the huge cost for short return. but definiely look to see if you can expand your hosters for DR.