The problem is that the LSE can't hire top tier talent. There are plenty of people who are experts in building hugely concurrent and fail-safe exchange software, and they pretty much all work for investment banks.
The LSE can't afford to compete with $1000/day salaries so they end up hiring second-tier developers. And unfortunately second-tier developers are a lot lot worse than first-tier developers.
I don't think it's about talent, but more of a problem of proper planning and execution. It seems things had been rushed to meet the tight demands, causing the chaos.
When LSE acquired MilleniumIT, to redo the platform, they claimed to have developed world fastest trading platform[1] and had enough talent under their belt developing trading systems around the globe for last decade.
None of the exchanges have the required talent. Being the best exchange technology company is like being the best team in a minor league. If you're good at extreme low latency stuff you're going to be working for a bank.
At a bank being great at low-latency can mean tens of millions of dollars of profit for your trading desk and a huge bonus for your personally.
At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.
> At an exchange being great a low-latency means ... well not much, your exchange gets a bit more flow and you get a pat on the back.
Exchanges pay out bigger bonus' than you'd imagine. 7 figures for a programmer is high but no where out of the ordinary. Says a guy who has worked for an exchange.
Exchanges often use (relatively) esoteric hardware + real-time OSes, such as the Tandem NonStop series. Experience with such a platform + domain knowledge about the finance industry is a rare combination, so I find it plausible that a couple of core expert programmers on the stock exchange's system get compensated quite well. High six figures + 80% bonus == seven figure, I guess?
> I seriously doubt any exchange is regularly paying developers multi-million dollar bonuses.
I'd agree with that. Having said that, I doubt many investment banks are regularly paying developers multi-million dollar bonuses.
The people who get those are called quants or traders, even if they have programming skills, just my experience.
> Which exchange did you work for?
I'd rather not say, if you want to know, contact me via email. Not sure how you got my linked in info as it's not attached to my profile.
Ahem, based on what evidence exactly did you reach that conclusion ?
TBH, from all the exchanges/brokers I've seen and dealt with in the past, there are a few that are pretty solid - Deutsche Borse is one of them. There's EuroMTS as well.
Essentially, most exchanges that deal with many market makers have got an extremely challenging technical problem to solve - scaling and latency. One that most investment banks can't even begin to solve.
I'm not saying there aren't any good talent in banks - there are plenty. I'm just saying that there's some really smart, talented people in exchanges.
That's very likely. Any programmer with even an entry level position at the LSE would get head hunted by multiple agents in the first month. And the investment banks do pay £700/day for talent.
I know at least two guys who are getting paid closer to £1000 a day to work in London for a bank. The money is pretty much insane imho, and the guys tells me the 'talent' is just like anywhere else - ie most people are meerly compentent (but still getting paid £500 plus a day). Even graduates with zero experience are taken on at £300 a day or more.
He seems to better than competent, and so has managed to negotiate a much higher rate.
it is a new platform, hard to tell but I wouldnt go so far as to say linux is to blame, its more likely to be exchange software, they've gone from native to fix 5, so its using a new protocol, new os, new software stack and most likely new hardware stack
I don't think that "linux is to blame" or even that any choice of programming languages or other tools are to blame.
It's a large system with extra-ordinarily tight performance requirements, and a lot of integration with external systems, and by nature hard to phase in gradually. That kind of system is very hard to get right first time regardless of your toolset. I'd be interested to hear more about thier methods to deal with these requirements and risks, less about thier platform choice.
IIRC, both this and the previous failed system were outsourced, which doesn't exactly fill me with confidence.
Yes, I didn't mean to blame Linux. I was just referring to the system in the way it was made famous (replacing a Microsoft based system with a Linux one).
Close, you do need to stop the world every once in a while to GC some of the harder stuff.
But it's true, there's flavors of Java that are certified for hard realtime. If you stop the GC for 100ms every 2s regardless if you need it or not, that's pretty nice and predictable behavior for a realtime system.
That seems unlikely. F# only recently moved from being an experimental language from Microsoft Research to a supported .NET language. The previous system is old enough to predate that.
Maybe not if they were using F# on mono. Their PR would have to work very hard to put this in a way that doesn't say "LSE is using our language on a foreign platform using a completely alternative implementation, which has nothing to do with us apart some ECMA specs - and they're still successful without MS involvement. Yes, we just confirmed that you can trust your highly critical .NET environment to the alternative versions of runtime."
True enough. If the statement turns out to be not entirely correct, that would be news.
I suspect that if there is any F# code, it's in a 3rd party system that talks to the stock exchange. There are lots of those due to the nature of an exchange, though they should all talk over the standard FIX protocol.
OK, I wouldn't expect an official press release like this from Microsoft head office.
But the team in Microsoft Research are (a) keen to promote their language (people like Don and Brian often help early adopters personally) and (b) supportive of Mono (they've always released a .zip file for Mono at the same time as their Windows installers).
using a immature language (f#) over unproved software stack for an international multi billion dollar low latency transactions when there are dozens of proven highly stable functional alternatives that run well on a linux platform? I can hardly believe it
At this size, some obfuscation might be good. If you find a bug and crash some browser, it's not a big deal. If you find some problem in LSE that can be triggered from outside, you're dealing with realworld $M transactions.
Also, unless you have a huge farm of servers it would be useless to you - and basically anyone really.
Unfortunately sometimes fail over isn't a solution. Until they've at least narrowed down the problem, a fail over system may make things worse.
Personally, I'm curious about the testing that was done - given the complaints and problems that have been reported so far, did they even try to run the new and old systems simultaneously for any period of time to verify that the various components could all keep up properly? Or at the very least, did they try to test the system at full load for a decent period of time?
The LSE can't afford to compete with $1000/day salaries so they end up hiring second-tier developers. And unfortunately second-tier developers are a lot lot worse than first-tier developers.