The technical interview is overwhelmingly the primary screening method of choice at the world's most competent software development companies. The team that builds the match engine for the electronic exchange, the team that builds the boot ROM for your smart phone, the team that builds the Google crawler and indexing, the teams building kernel SAN storage code --- these teams are hired by technical interviews.
The "audition project" is a trend story with, from what I can see, very little empirical evidence to back it up. When the most notable example of a company routinely employing audition is Treehouse --- all due respect to Treehouse, but still --- that's a red flag.
The bigger red flag is that the paid audition project method has obvious flaws. The top 20% of software developers are almost uniformly employed. Prospective new employers for these people court them actively; in fact, the problem of companies luring top developers out of other companies is so challenging that large SFBA tech companies have entered into illegal compacts to stifle the practice. Companies are so hard-up for talent that they'll "buy" startups just to get access to their teams.
In this environment, why would a top developer, who has their choice among tens of different high-paying interesting jobs, moonlight for a prospective new employer just so they can make sure the relationship's going to work?
Most technical interviews are terribly flawed. They aren't standardized and they aren't rigorous so you can't compare candidates on any apples-apples basis and you can't correlate them to job performance to make them more predictive. Most of the developers tasked with conducting them suck at interviewing; many use interviews as a sort of hazing ritual, and most use them as opportunity to project their own subjective views about how software should be built onto candidates. Many technical interviews are trivia games punctuated by awkward attempts at working through code on a whiteboard or a piece of paper in a high-pressure environment.
The solution to this problem is to improve technical interviews. It's not to pretend that the whole market for devs has suddenly embraced temp- to- perm hiring. It hasn't. It's getting harder to find good developers, not easier, and the notion that companies have the luxury of inflicting "audition projects" on candidates is counter to reality.
I would agree with this in that the "audition project" would not be done by top candidates - but I'd like to suggest that they already do the audition project.
The group I'm in (all flash SAN storage) has gone from single digits to ~80 people in under a year, and basically the only predictive metric is referrals from people who have made other good referrals. We're writing a complete HPC OS, and it's sink or swim. The referrals are almost all swimming.
That's it. Almost all other hires have been completely random in quality.
So these amazing engineers that you're really struggling to hire are actually being hired because of a career long audition with their peers. It's wildly incestuous, and has some interesting effects. Some effects are bad: no one knows new tools/new development styles. Some effects are good: at the top, gender split is much closer to 50/50.
I'd love to figure out how to vet someone in under 3 months that is not part of this professional tribe, but even typing that out I'm almost suspicious they'd succeed and become part of it if they aren't in the tribe already. It seems like even my attitudes are a problem in opening up these kinds of candidate searches.
You make an interesting macro point. But there's a world of difference between an entry level full-time job and a 1-2 week "audition project".
The phenomenon you're describing is what makes recruiting such an interesting problem. The largest or wealthiest employers can often afford to constrain their candidates to people who have proven themselves in other settings. Which of course leaves a "Moneyball"-type opening to the employers who can figure out how to hire for aptitude instead of experience. Which makes hiring at those savvier employees a high-stakes, high-value process, instead of a tedious formality inflicted across the whole team.
> Which of course leaves a "Moneyball"-type opening to the employers who can figure out how to hire for aptitude instead of experience.
I want to call for tokenadult's canned "within the US, look for a work sample. Outside the US, use an IQ test" post. We figured out _how_ to hire for aptitude a long time ago (subject, of course, to the fact that prediction is never perfect); Malcom McLean apparently filled his new shipping company top-down by who, among the people who worked for his old trucking company, scored highest on an IQ test. But then we decided that that would be illegal in the future?
We know what the solution looks like, but not what its precise measurements are; the person who comes up with a universal, economical, predictively powerful work sample test for software developers is probably going to make a lot of money.
Really a family of such metrics because the moment the first one hits the ground in any earnestly it'll be gamed to death. Nothing ruins a powerful, economical, repeatable indicator for predicting economic activity faster than using it.
The point I was trying to make wasn't about hiring "for experience" but leveraging those who were around them to vouch for their aptitude. This is a distinct difference, and you have to work hard to get people to think in these terms instead of amount of experience. I may be naive in even thinking we're accomplishing this.
Without the public displays of performance that baseball has, there's another level of indirection that you must filter through to try and illustrate a candidates performance.
This makes it something like the "Moneyball of Moneyball of programming".
Technical interviews are perfectly good at doing what they are intended to do: screen out people who can't program. Somewhere between 50% and 80% of people I meet in first-round interviews literally just cannot program. Half of the people who say C++ is their strongest language can't tell me the difference between a pointer and a reference, or between a pointer to const and a const pointer.
I don't know if this problem is specific to our industry or not. Do people apply to be English teachers when it takes them an hour and a stack of reference material to form one English sentence? Do people apply to be carpenters who don't know how to hold a saw?
When I was interviewing people, phone screen with some basic concept questions would filter out 80% of applicants with nicely looking resumes; an hour-long whiteboard screen with FizzBuzz kind of problems would take care of the rest. Very quick and efficient process.
Pointer allows you to dereference a memory address. A reference is the physical memory address variable.
If you have a variable and you want to pass-by-reference, making any changes to the original variable effect the originally allocated value.
You use a pointer for instance if you want to dereference a physical memory address as part of an array of allocated memory addresses.
// reference example
int value = 0;
int value1 = &value;
value1 = 2;
// value should equal 2
// pointer example
int value[5] = {1,2,3,4,5};
*value += 1;value++;
*value += 1;value++;
*value += 1;value++;
*value += 1;value++;
*value += 1;
// should now be {2,3,4,5,6}
The syntax is different, but in terms of what you can do, the only difference I know is that references can be reassigned.
Both give you the guarantee that it's not pointing at the NULL address. In the case of const pointers, that's because you can't reassign it, and you'd never deliberately assign a const pointer to NULL. In the case of references, it's because there's actually no such thing as a null reference in C++.
That's not the case in Java and C#, which allow "null" to be a right-hand value for all types.
Sorry, somehow I managed to forget that primitives existed for that brief moment. I was reacting to the realization that "null" is a bizarre special case for the type system.
I was wrong on reinitializing references as well. Thanks for pointing that out. As far as const pointers, I meant that they were unlikely to be null, since they would have to be initialized to null. Of course, then I realized that you could easily assign them to another null pointer that you didn't realize was null.
Not a good day for me, but I don't think it warrants that kind of snarkiness.
To my credit, please reread the part where I said "there is actually no such thing as a null reference."
The best part of your reply is in showing how he failed his own interview, figuratively speaking of course. In real life, the candidate doesn't get to explain things the day after they were turned down because of bad answers.
I truly believe he knows his stuff, but even he made a mistake. He, however, didn't have to come up with his answer while people are staring at him.
A reference looks like a value but is actually a point (de-references are automatic). Most high-level languages prefer references over pointers as they are harder to screw up. E.g.
myref = otherref + 10
is equivalent to
* myptr = * otherptr + 10
Now, most languages also have a notion of values that are neither pointers nor references, and primitives generally are expressed as values, but hopefully you get the point.
I don't think I saw this mentioned but another technique that companies use it to subject you to formal tests. Usually produced by some 3rd party. I have huge experience with Excel and spreadsheets since Lotus 123 and have worked on and produced some enormously complicated, but successful spreadsheet/VBA solutions. I am almost invariably seen as the resident Excel expert in all the organizations I have been employed, yet have failed a test to prove my Excel "competence" because somehow the Dictionary Object was considered all important by this 3rd party test and yet in 15 years of spreadsheet is never something I touched.
I don't doubt that the Dictionary object is very useful in some use-cases but to fail an Excel competence test because some test producer decided to get cute is hardly an indication of real world knowledge and experience with a product.
That is just Excel, I shudder to think what a test in C++ would do to even a highly experienced C++ programmer if it decided to explore all the edge cases and obscure gotcha's that exist in C++.
This is not to say that some tests cannot be useful - just be very careful of what you are actually testing for and how you are doing it.
Often enough, just testing the things that everyone uses day in and day out is plenty enough to tell you about how a person can be as a programmer. Such as creating a table view that says 1 to 5 or animating things in iOS, and const pointers and references in C++
I've mentioned this before, but I have a trendy skill, and one that has a higher barrier to entry than eg web developer. If a potential employer says, "so... x0x0. What we're thinking is that you burn a week of vacation time doing a trial project for us" my response is something like "hahahaha...bzzt. next"
As mentioned in the article, after Google examined the data, they found that technical interviews were worthless at determining employee quality. Instead, they say, "what works well are structured behavioral interviews." This seems pretty damning for the technical interview. Would you just say that Google was doing technical interviews wrong?
It's funny - when I hear people in the press discussing technical interviews [1] I hear questions like this: "How many golf balls could you fit into a school bus?"
When I attend technical interviews I hear questions like this: "How would you implement an arbitrary-precision decimal number class, like Java's BigDecimal? Can you write out the add function on that whiteboard?"
When Google's Laszlo Bock [2] says "On the hiring side, we found that brainteasers are a complete waste of time. How many golf balls can you fit into an airplane? How many gas stations in Manhattan? A complete waste of time. They don’t predict anything." it seems fairly clear to me the idea he's dismissing is brainteasers, not whiteboard coding.
That's consistent with other reports [3,4] from a few years ago saying brain teasers were not used.
The author of the TechCrunch article seems to interpret Bock as saying whiteboard coding is an ineffective practice. I'm not convinced the Bock interview said that.
I'm suspicious of whiteboard coding as well. Why not give the person a computer and 20 minutes? Why make it less realistic by taking all reference material, editor and ability to test-run the program?
No, I read tptacek as agreeing with this person at Google. [1] That person is trying to develop a better interview formalism that works and is repeatable. So is tptacek, albeit probably in a slightly different direction.
What neither the Googlers nor Ptacek are saying is that the solution to the screening problem is to ask candidates to accept a deal that most full-time employees, and most clueful contractors too, would reject out of hand.
If your screening strategy would reject any sizable percentage of people who are currently successfully working for someone else, your screening strategy is bogus. Unless your express goal is to hire promising juniors who don't know their own value. Which does feel like the goal of many hiring managers: As market rates get bid up, they're desperately looking for people who are just clueful enough to succeed, yet just clueless enough not to ask for money in return.
----
[1] I do not presume that any single person speaks for all of Google. In particular, I would be surprised if that giant company of folks who were all hired via brainteaser interviews has actually entirely stopped using brainteaser interviews. Fixing interviewing culture is hard.
If I'm a top coder I'd do the 3 months evaluation contract in a heartbeat... why? Well because if I'm a top coder then chances are I know I'm pretty good, and by the end of three months I'll be able to better assess the value I can (maybe already do) bring to the company. That would in turn give me a ton more leverage for compensation negotiations than I would have had earlier. Also, if I'm hoping to find a place to park for a while, isn't it better for both sides to get a better feel for each other by actually experiencing that fit?
If it doesn't work out there's always the next 3 month interview...
It's getting harder to find good developers, not easier
This could be part of what's going on. Small companies are getting the leavings of the big ultra-competitive companies. Small companies notice no matter who they hire, they are never up to par. Small companies blame this on the technical interview.
Another compounding effect is that in order to get good at hiring, you need years of experience hiring, so you can find out what kind of interview is predictive and what kind of interview just wastes time. Most small companies don't have that kind of history to draw on.
Also, when you are able to detect and hire the best people you can put a heavy pressure on the competition by simultaneously depriving them of talent and attracting more talent yourself. E.g. if the company A that has 8 out of 10 top people in my field made me an offer then the B with 0 such people will have to make a much better offer for me to even consider them.
In such a situation the usual "false negative is okay, it cost more to hire a wrong person than to skip a right one" stops working. Skipping a right one will cost a lot in the long run.
I would say that a technical interview consisting of coding questions that test data structures and algorithms is about as standard as you can get. True that some interviewers will turn this into a hazing ritual, but hopefully there's some feedback mechanism by which that behavior can be discouraged -- a few ideas would be to have shadow interviewers or allowing candidates to submit feedback about the interview content and process etc.
IMHO, the ideal technical interview would be one where the interviewer is able to dig in deep into the candidate's experience and able to ask intelligent questions that reveal the candidate's technical depth. But this requires very strong interviewing skills as well as an appropriate technical background on part of the interviewer -- this will not scale.
The "audition project" is a non-starter for the reasons you mentioned and as you correctly point out, we have no evidence that it works in the large.
So back we are to the flawed coding interview. Like some wag said about democracy: it's a shitty system, but there's nothing better.
A standardized interview process is one where every candidate receives (roughly) the same set of questions, so that the company can look over the last 18 months of interviews, compare every candidate regardless of who gave the interview, and make observations across candidates.
A rigorous interview (for lack of a better term) is one where questions have either correct answers or quantifiable answers. "Best algorithmic approach" to a problem is usually not a rigorous question (unless the problem domain is very well defined and very well studied). But "number of valid algorithmic approaches offered" is, as would be "validated time complexity of best offered approach".
The idea that a tech interview should be digging deep into a candidate's background to assess their experience and derive intelligent questions that reveal depth is, I believe, the pervasive myth underlying all bad technical hiring. The people who conduct technical interviews are developers, not trained psychologists. Developers have a pronounced tendency to overestimate their own confidence in things like interviews. For lack of a more eloquent way to put this: developers suck at interviewing.
There's no better way I know of to interview devs than to have devs do interviews, because it takes a trained developer brain to accurately classify answers to useful tech questions (ie, "is this unexpected response incorrect, or is it a correct response that uses a more clever approach than the one we expected, or is it a correct response phrased in different terminology than we're used to", &c). You have to have devs doing interviews.
So the next best thing is to stop pretending that devs are capable of doing something that they're mostly not capable of doing. To me, that says "take a couple weeks to come up with a standardized interview that every developer will give".† Watch the responses to those interviews. Look at who those interviews select; if those people seem to be working out, stick with it, else adjust.
† One indicator I'm playing with: if anyone on your team looks forward to doing interviews, you're doing something wrong; for the interviewer, the process should be mostly very boring.
> In this environment, why would a top developer, who has their choice among tens of different high-paying interesting jobs, moonlight for a prospective new employer just so they can make sure the relationship's going to work?
I have my doubts that I'm a top developer (I have a long list of things I'd like to understand/do better in order to consider myself in that category), but I've done and would consider doing a paid audition project.
The job I have right now pays market rates, offers good benefits, and involves work for famous clients. It's not a bad gig, and there are enough people who've been there for over five years that it's clearly a good environment for some. On the other hand, I don't find the work itself particularly satisfying. Most of our projects themselves are a bit staid (as projects from established brand clients who feel they have more to lose than gain from risky moves can be), and a lot of what I've ended up doing is production front-end work rather than much problem solving.
This is different from the picture I got during the interviews. That could be as much about my own ability to read what day-to-day life was likely to be like post-hire as anything else, but whether it's uniquely difficult for me or for everybody, I'm left with the same problem -- how do I get a glimpse of what working there is really going to be like?
Seems to me a paid audition is one good answer.
I'd rather do 2-3 months on contract than a 2-3 day project, but I'd rather do the 2-3 days of real work than nothing.
And I think the 2-3 days of real work idea beats another practice that's becoming prevalent: the ~1 day (unpaid!) work sample/code challenge. I don't mind such challenges in and of themselves; a small exercise can be educational and an opportunity to use all the things you've been thinking would be good ideas to fold into a new project. What I dislike is that real feedback seems to be scarce. Even the people who've hired me didn't give me much feedback beyond "it met the requirements and we learned a thing or two from your work." By the time I've invested a workday of focused time in a company, I'd appreciate a "we were hoping to see" or "here's what we liked", and I'm finding I don't get much of that from challenges/exercises. Whatever the flaws of the conventional interview process or the paid real project may be, at least they're both generally more symmetric in terms of investments required and opportunities for evaluation.
I'm not saying that freelancing is bad. For a bunch of reasons, I'm very positive about freelancing and consulting.
You sound like someone who'd rather be consulting than working full-time jobs. Great! You'll probably also make more money and get more professional experience faster than people who take the less-risky full-time job path.
What I object to is the notion of temporarily converting full-time workers into consultants so that companies can mitigate their concerns about how inept their own technical interviews are. Consultants and full-timers are two very different things, and they aren't easily convertible.
> the ~1 day (unpaid!) work sample/code challenge [...] Even the people who've hired me didn't give me much feedback beyond "it met the requirements and we learned a thing or two from your work."
You need to be concerned about the employers who "learn a thing or two from your work" and don't hire you.
Some programming "team leaders" I've met wouldn't hesitate to bring in applicants to work unpaid for a day on coding challenges containing technical blockers from their own projects.
And of course much of what happens in the open source programming world is the same on a larger scale without even needing to supply office space for a day.
I have my doubts that I'm a top developer (I have a long list of things I'd like to understand/do better in order to consider myself in that category), but I've done and would consider doing a paid audition project.
The problem is that it's not always your choice to make. Noncompete clauses exist, and, if you're outside the state of California, are generally enforceable. I don't know about you, but I'd really rather have a system where taking an interview wasn't a fire-able offence.
To spot a top performer one needs no elaborate testing or, god forbid, standardized interviews. It is the same as it is in sports or music or arts - there are cues that identify a top performer. What standardized tests you want to apply to, say, Beethoven, Orwell or Muhammad Ali, or, good forbid, Steve Jobs, or Igor Sysoev?)
It is not about testing, it is about having a gift (or not) and a character - willingness to become a master just for the sake of ones love/passion/enjoyment of the craft, not any stupid job or position.
As a shameless plug - if anyone's interested in a completely in-browser experience where you and a candidate can both write and EXECUTE code simultaneously, I make a tool for that: https://coderpad.io
> these teams are hired by technical interviews.
The "audition project" is a trend story with, from what I can see, very little empirical evidence to back it up.
The problem with the "audition project" is precisely that it has the same inconvenient economics as dating, finding an apartment, or job hunting/hiring in general, only more so. There is greatly increased "opportunity cost." This alone can explain the lack of data. It might be like a lot of "better" tests -- it might be a better way to gauge suitability on an instance-ti-instance basis, but the economics preclude it from being widely used.
Since you are an employer and interviewer yourself, have you thought about writing a guide to doing technical interviews? Or do you think one of the existing guides, such as SPolsky's Guerrilla Guide, is good enough?
Also, if "audition projects" were to become the norm then you'd see rampant cheating. Simply farm out the project to a competent freelancer and then pass it off as your own. The freelancer may not even know they're working on an audition project. And they may not even end up being paid more than the candidate since they could likely finish the project in much less time (and thus work at a much higher effective hourly rate).
And how do you filter the candidates who you give audition projects to in the first place? That's very difficult without a technical interview.
Detecting a cheat should be easy -- just ask them to walk through the project and explain it. You should be able to expose a cheater with a couple of deep questions.
Sigh, this is what I was going to say (I tend to say it with more typos :-).
The interesting effects here for people building teams then are in the dispossessed. The folks who didn't graduate college (so shunned by many of the big names) or the folks who took an off beaten path. My concern is that if Google fixes their blind spot with respect to this talent pool it will get that much harder for startups. So far I've not seen any indication that they have but I worry about it.
[Edit: I can't even joke without getting it wrong.]
If you've found a superstar, then surely they can skip the process. Otherwise a test drive sounds perfect. It's what you'd do in any other transaction.
That's one reason. However, I have been in the position of being desperate to sell a home (many people were at the time and in some places still are, I'm sure), and "test drive" or any equivalent simply does not make sense in some situations.
There's a lot less overhead to driving a car for ten minutes than there is to living in a new place for a day or two... to get the right impression of a car, you drive it in a residential area, you drive it on the highway, and there, you have at least an overview. To do the same for a house? Move in, settle in, meet the neighbors, live in it, then move out, because someone else wants a test drive, too.
And, of course, neither applies well at all to interviewing a person.
The metaphors just don't work here, and they're not helpful. Maybe there are things that work while car/house/pet/food shopping that can be applied to interviewing, but that correlation does not necessarily indicate any similarity between those situations, and comparison isn't useful unless the activities themselves actually are meaningfully comparable.
Most startups aren't looking to hire developers that are desperate for jobs. The point is they have a job they like reasonably well, they have other offers on the table from other companies. Other companies here did not require them to moonlight for a week knowing full well it might not work out (keep in mind they would be moon lighting with you in addition to interviewing with other companies). So why would they agree to moonlight if -- even without moonlighting -- the job offer is already a tough sell?
There are two possibilities:
1) You're looking for a co-founder of for perhaps the founding engineer, e.g., Sergey Brin and Larry Page hiring Craig Silverstein. You've impressed them to a great degree (they like your product demo, your background), and they are already willing to pay potential opportunity costs (a job at an established company, finishing their Ph.D. program, etc...) to join you. Then this really isn't a hiring problem per-se, this is a "finding a co-founder or founding engineer" problem; you are less likely be hiring someone "from the cold" for this in the first place (this is why so many companies are started by graduate students or former coworkers...)
2) You're willing to hire people who are unable to find other full-time work, in other words someone with no formal experience or experience in a completely wrong area (e.g., someone who worked on internal business web applications, but who wants to get into embedded programming).
Then you're really looking to hire an intern (in case of no formal experience) -- where the process is essentially that, a contract to hire, or simply take a calculated risk. In other words, you are able to see potential in this person (which they've demonstrated by perhaps building an app in their spare time) and are willing to set them up for success (as oppose to trying to find the best candidates out of a pool).
The "audition project" is a trend story with, from what I can see, very little empirical evidence to back it up. When the most notable example of a company routinely employing audition is Treehouse --- all due respect to Treehouse, but still --- that's a red flag.
The bigger red flag is that the paid audition project method has obvious flaws. The top 20% of software developers are almost uniformly employed. Prospective new employers for these people court them actively; in fact, the problem of companies luring top developers out of other companies is so challenging that large SFBA tech companies have entered into illegal compacts to stifle the practice. Companies are so hard-up for talent that they'll "buy" startups just to get access to their teams.
In this environment, why would a top developer, who has their choice among tens of different high-paying interesting jobs, moonlight for a prospective new employer just so they can make sure the relationship's going to work?
Most technical interviews are terribly flawed. They aren't standardized and they aren't rigorous so you can't compare candidates on any apples-apples basis and you can't correlate them to job performance to make them more predictive. Most of the developers tasked with conducting them suck at interviewing; many use interviews as a sort of hazing ritual, and most use them as opportunity to project their own subjective views about how software should be built onto candidates. Many technical interviews are trivia games punctuated by awkward attempts at working through code on a whiteboard or a piece of paper in a high-pressure environment.
The solution to this problem is to improve technical interviews. It's not to pretend that the whole market for devs has suddenly embraced temp- to- perm hiring. It hasn't. It's getting harder to find good developers, not easier, and the notion that companies have the luxury of inflicting "audition projects" on candidates is counter to reality.