I answered "a few critical things" ... but, for the most part, testing is tedious, frustrating, and a time-sink for me. I recently paid someone $100+ an hour for some remote TDD coaching. It's helping a bit but hasn't really change my attitude towards testing (yet).
What bugs me:
- Testing frameworks and "best practices" change way faster than language frameworks and I simply can't keep up. What rspec version do I use with what version of Rails? Now I have to use Cucumber? I learned some Cucumber ... oh, now Steak is better. [rage comic goes here]
- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ...
- I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite
- More code is moving to front-end Javascript stuff ... so, now I have to write Rails tests AND JS tests? Sounds delightful
Feel free to try and convince me otherwise, but I don't ever see myself in the "test ALL the things" camp.
My approach to testing is not to be obsessed with the latest, greatest framework or 100% code coverage.
I try to start with just one or two tests to actually help do things that are tedious or require multiple steps. It takes some time to automate a good test but once you do it immediately starts saving time because you don't have to run the same sequence a thousand times while developing. You can think of it more like a macro that saves you time.
Once you write the main test it's easy then to run it with all combinations of good and bad input. By doing that you'll often wind up hitting a pretty good percentage of your code.
Then as bugs are discovered due to unexpected input you can just keeping adding more input situations.
Look at it this way: you must be testing code as you write it anyway. There's really no other sane way to do it. You make a change, you load the page and see that your change worked, or you call your new function from an interactive interpreter.
Smart automated testing just takes all that extra test work you're already doing and saves it as you go along.
No need to try to invent extra things to test. You just test what you would have tested anyway by hand.
You think like that until you hit your first serious regressions and discover it has been in the code base for several months and that the person responsible for it has left.
I used to work at a company where automated testing was sending you emails about what your commits broke. It does help at improving code quality.
But then you'd still do the manual test after you complete your code. Nobody (I hope) codes blind hoping it would work or caught later by a test suite. Test suits don't reveal everything. Only what you tested for.
If all your "test it" steps are being done manually, you're being very inefficient. A good unit test can actually make development go by faster with the added bonus of defending your code from changes down the road that might screw things up.
That's all fine unless you're exploring a solution space. Then the overhead of writing tests which are thrown away _in entirety_ is outrageous. AFAIK TDD really only works if you're either a) prepared to waste a huge amount of time writing tests which will later be completely redundant or b) working to a very clear set of requirements with tools and frameworks you already know intimately. </Rant>
I find I spend significantly more time refactoring/maintaining code than I spend writing exploratory code. It's silly to write tests for prototype work, but once you're actually close to having a working prototype, tests help. Having decent test coverage saves so much more time when refactoring/maintaining.
TDD isn't "THE" way, but test coverage helps. It's not fun (at least not for me), but it's less aggravating than breaking something 6 months down the road in some non-obvious way. I'm human, so I assume I'll screw something up eventually. Having test coverage helps keep me from shooting myself in the foot later.
Different levels of tests work here. I usually start with a very high-level test and then as I implement I do unit tests once I have a reasonably high confidence that the units are a good design.
You should often be able to at least create an automated acceptance test for what you're doing (e.g., "as a user I want to click this button and see XYZ"). This is usually extremely decoupled from the implementation so it should survive refactoring. So then do your exploratory code, get the test passing, and then refactor, introducing lower-level tests.
If that doesn't seem doable you might be taking on a task that doesn't have a good set of requirements. Writing code without any concrete use case in mind is fun and all but that kind of code should usually be limited to a prototyping sandbox.
"If that doesn't seem doable you might be taking on a task that doesn't have a good set of requirements." Or it might have perfectly good requirements which are very hard to write automated tests for.
Consider (for instance) a program to translate ABC music format to proper sheet music. It's easy to say the basic requirement: "The program has to accurately translate ABC format to legible and easy to read sheet music." But even a start at automating a test for that would require converting a graphical representation of sheet music back to the basic notes, and that problem is at least an order of magnitude harder than writing the original program, without factoring in the "easy to read" bit at all. (PS This is a real issue for me, if someone knows of a decent open source sheet music to MIDI/ABC convertor I'd love to hear about it.)
The mistake here is that what you're describing is a functional test, not a unit test.
A unit test for a piece of code like this might be "Given that the time signature for the music is 3:4, the software puts 3:4 in the appropriate place on each line of output".
You then might write a variety of cases testing that it deals correctly with (say) a changing time signature at some point in the piece.
The upside of this is when you try and fix another bug which has a knock on effect on this bit of code, lots of your tests are going to fail- immediately identifying where the problem is (or at least letting you know there is one!)
> A unit test for a piece of code like this might be "Given that the time signature for the music is 3:4, the software puts 3:4 in the appropriate place on each line of output".
Don't you still have the same problem colomon described here, though? Testing your stated condition "the software puts 3:4 in the appropriate place on each line of output" still implies some form of image recognition on the graphical output.
But okay, is it really so much easier to write a test which just tests changing the time signature? It's still going to require doing OCR. And if I'm really testing it, I've got to make sure the time signature change comes at the correct point, which requires OCR on the notes around it. Also the bar lines (to makes sure the time signature change is reflected in the notes per bar and not just by writing a new time signature out and otherwise ignoring it).
Now, it's perfectly reasonable to have unit tests that the code correctly recognizes time signature changes in ABC format. (And it looks like I forgot to write them: I see inline key changes in the tests but not inline time signature changes.) But that's only testing half of the problem; and it's the easier half by far.
Actually, I've written tests for half of that problem: Parsing ABC text into an internal format. That is a very clear problem and one just needs a representative set of input files (I now have around 500 of them -- the tests still run in less than a minute. It's true that I haven't been able to figure out a useful way to have an automated test for the drawing part. Here's my project: http://code.google.com/p/abcjs/
in TDD what you are talking about is called a "spike", just write a bunch of code to try out assumptions and find a direction to go that you are reasonably sure is a good one.
Prototype all you want without tests, but once you've settled on a design & are ready to make it production-ready you should spend time at least writing unit tests for your work or (ideally) take what you've learned & re-apply to a clean design written in a test-first manner.
You might think this is a waste of time, but putting code into production without tests is going to give you more trouble in the long run.
Of course you have to know your tools. I usually start by doing test cases, then writing the real features while spork and whatchr evaluates my new code every time I save. I rarely even open a browser, it's the final thing I do when my tests are green.
It doesn't slow me down, and I can be sure that my feature is there and works even when somebody refactors our software.
That's mostly a beginners problem, I know several people where it's mostly:
1. Write a lot of code
2. Test it
3. It works!
4. Test it
5. It works!
6. Test it
7. It works!
...
15. Test it
16. It works!
17. Are you done?
TDD in no way makes 17 any clear, because every test they thought of before writing the code works more or less the first time. And that's the core problem with testing, for a solid developer what fails is has nothing to do with the code it's always a question of edge cases they did not think. (Wait, some sales people are their own managers and outside consultants at the same time? well just bob) You can force these people to write tests, but it really does just slow them down.
Not groking TDD, I literally worked my way thru the book, doing each and every step, just to get the gist of the experience.
TDD works fucking great. If you know what you're doing.
Alas, that's a big IF. Most of the stuff I do, I'm just figuring shit out.
Mostly, like when designing a new library, I work outside-in. I imagine how I'd want to do something, writing the client pseudocodeish stuff first, and then trying to make the implementation support my idealized programming mental model.
I end up throwing away A LOT of code. Getting something short and sweet takes a lot of experimentation, most of which are duds.
Though my personal approach of outside-in is trivially like TDD, it's not nearly as rigorous. Were I to be as thorough as TDD, I'd be spending all my time writing tests. Which seems pointless, for code I'm like just going to throw away.
Anyway. Much respect for the guy who wrote that first TDD book. It's one of the few methodological strategies that works as advertised.
I do strict TDD when I can, and I consider spiking out things part of the process. If I need to approach a problem that I don't know how I'd solve just yet, I create some sample code I later trash and do a lot of work in the Ruby console.
Then, once I've gotten an idea of the problem, I can start writing out some pending tests that help me figure out structure, and then I'll start into the strict TDD loop of write a bit of test, watch it fail, make it pass, write more test, etc.
It's not about telling you when you're done a feature. It's about leaving step 17 with a set of tests so that the next person in the code can tell when he's done without doing steps 1-17. And you'd think it slows you down, but it really doesn't. Some advantages of TDD are less context switching (you can test your code without even leaving the code itself) and a high degree of focus (every atomic subtask has a very clear completion criterion: fix the failing test). Those are things solid developers love.
I have never seen any half way decent developer write code for more then a few minutes without some sort of feedback, automated test suite or not. You are right that it will work more or less, but they work out the "less" part of that statement sooner rather then later. In my experience, that is a place you get to over time, only the people out of college write code for multiple hours straight, then debug everything afterwards.
That is actually the primary goal of tdd, to free you from the more mundane aspects of the code/run/debug loop. The secondary goal is to give you a good base for changing the code later and finding out what broke, again without a ton of manual actions. But as useful as that is (and it is extremely useful), it doesn't hold a candle to the first benefit.
> I have never seen any half way decent developer write code for more then a few minutes without some sort of feedback, automated test suite or not
I do this all the time. Two reasons:
1. I can keep in "the flow" for an extended period of time. This is more important if the code is especially complicated. If I have to stop every few minutes to fix trivial errors, it's easy to forget important details of how everything is supposed to work.
2. Not having any feedback forces you reason about the code before writing it. It's very easy to fall into the trap of writing code, then waiting until it is tested to find the errors. Thinking before writing is the fundamental skill that TDD encourages, but you don't need TDD in order to do it.
> only the people out of college write code for multiple hours straight, then debug everything afterwards.
Knuth wrote TeX in a notebook and did not test it for a good six months afterwards, though I am not aware if he was out of college at the time.
> I have never seen any half way decent developer write code for more then a few minutes without some sort of feedback, automated test suite or not
Reading that again, I'm sorry if it came off as sort of attacky, but I really meant that as a "from my personal experience with the people I have worked with over my career" type qualification :)
I can buy #1, but only when it is something you've done a bajillion times before. When you are getting feedback every few minutes, you know exactly what introduced the problem, and don't waste time tracking things down. If you do miss something fundamental and have several hours work behind you, you tend to be more inclined to hack out something to make it work, where if you catch it a few minutes in, you can adjust your design to take it into account. I also find I can keep in the flow pretty easily with constant feedback, and I use simple todo lists to make sure I don't lose track of things.
As for 2, at least for me, I don't think there is any comparison between thinking about how things should work, and knowing if things do work before writing. TDD is definitely not a replacement for deep thought and planning, but I think that is a different beast then working out the details as you are writing them, which is where it comes into play
> only the people out of college write code for multiple hours straight, then debug everything afterwards.
I sort of did it again there, I should have qualified it more :) In my experience, the better programmers I have worked with, paired with, and watched code in videos will get feedback as quickly and often as makes sense, be it with tests or without them. I know if I wrote TeX in a notebook, it would be a guaranteed unmitigated disaster :)
I think most of you guys missed the point. Writing tests is very important WHILE writing code, to write it better.
We MUST write tests not only to catch regressions, to be sure that certain invariants will be manteined. But we write tests to check if we are writing good code.
I need to write a class to do some stuff. The test is the first user of this class. If I cannot write the test very fast, and I see that I'm spending a lot of time doing it, this means that my class is poorly designed, is not flexible, is not very reusable. Maybe I'm doing something wrong with my app design. If I'm writing good code, reusable and clean code, testing is easy and fast.
Testing help me to check immediately what's going wrong with the code, not only in term of bugs.
That's the point I was trying to make. The main benefit comes from eliminating the "run/debug" part of the "code/run/debug" loop. It then just becomes "code/test" where "test" takes all of a couple seconds each time.
A couple of seconds is too long. I run a small test suite in a couple of microsecs every time I save a file, and save my file at every change. When a task is done I run all the tests.
But surely if you have written a unit of code, you should at least know a. What valid input the code should have, b. what output the code should return, and c. What you want he code to do! If you know these things, then wouldn't it be easy enough to write tests for at least these conditions?
It's not that. I wrote a fairly complex piece of code in, of all things, TSQL, and as the logic was unfortunately in the stored procedures and functions I actually found that the unit tests I did for the more granular functions saved me a lot of time. This was because I would make a change to the logic of a function that other functions/procs relied on and the. All of a sudden I would find that a whole bunch of tests on functions that worked before started failing. I'd never have known this without the tests that DID work previously. Saved me a lot of time I can tell you :-)
Then you didn't write good enough tests. I have deployed code that thousands of customers see without manually testing it. If my tests are green, I'm confident in deploying my code.
Even if you test manually what you just changed, in a relatively complex codebase how can you guarantee that your changes haven't broken behaviour in a separate yet related aspect of the system?
This is what I find the major advantage of a comprehensive test-suite to be, I don't have to worry as much about breaking any part of the system as a whole - if my suite passes, then I know everything I've worked on so far works, not just the bit I think I changed.
Smart automated testing sounds amazing, until you realize that it's not smart at all. The dumb computer that you're ordering to do your bidding is the same dumb computer that is going to be running your tests, and chances are the programmer is invariant as well. In short, good programmers need unit tests less and bad programmers will write bad tests. You can't fix a personnel problem with technology.
Exactly. The computer will just repeat what you tell it to do. There is the chance that you will tell it wrong (a bug in your test code), and the chance that what you told it is not true anymore. An automated test basically saves you the work of doing the same thing over an over again as you develop.
But we must keep in mind that maintaining test code has a cost. Automated testing is not a holy grail and it isn't useful 100% of the time. You should carefully decide what code is worth testing via automation.
Unit tests in general don't catch any regressions, they only help you develop.
Functional tests might be useful, but only if you are testing something that is not likely to change much. E.g.: It is not worth to automate testing the UI if you are going to completely redesign it next month.
Manual testing can actually be cheaper than maintaining test code.
> - Testing frameworks and "best practices" change way faster than language frameworks and I simply can't keep up. What rspec version do I use with what version of Rails? Now I have to use Cucumber? I learned some Cucumber ... oh, now Steak is better. [rage comic goes here]
I think this is only in the Rails community, where all new things is quick to be pronounced "the new right way to do things", not just in testing.
> I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite
We do the same (tests for interaction with EC2, Github, and a few other providers). It is more expensive, but we find it more worthwhile too. Normally, 3rd party APIs are insufficiently specified, especially for error conditions. So when we have a failure in production, we can easily add tests to make sure we handle that edge case in future.
I'm working on a single page app that's about 30% Rails and 70% Coffeescript/Backbone.js. Test::Unit is practically useless for us since users never hit any plain HTML pages besides the login page.
Imagine the current HTML5 Pandora having bugs with one particular song in Chrome only. How do you test for that using Test::Unit?
there are a bajillion libraries out there for unit testing javascript (something test::unit-ish would be qunit). rails + backbone + JST style templating actually makes TDDing your js relatively painless
We're looking into integration-level testing for our Javascript since it'd give us full coverage of the bugs we see.
I forgot to mention we also use Socket.io for real time push updates to user Backbone models. A good number of bugs don't crop up until another user modifies data and those changes get pushed out to another person.
Testing with front end javascript is more difficult due to different interpreters, much more variation in environment, etc. That doesn't mean you can't have tests for your core business logic at least in something like Vows or Mocha.
I strongly feel you should try to add one test in each category. That adds a sanity check and lowers to cost to adding more tests when you really need it.
It's pretty painful to think "oh, this really needs a test, but I haven't got a test suite set up and besides, I don't know to write a test of this kind".
Writing tests for edge cases we see in production is the most valuable thing we do. We use Airbrake to find the bugs, and then we add a test for it, if possible (it's not always possible).
That gives us good confidence that other changes aren't fucking things up. It's also a pretty sane strategy for growing a test suite when you inevitably have some portion of your code which has no tests at all.
"- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ..."
This is why regression tests are my favorite type of test. The need for the test has been confirmed by real world usage and once you create the regression test to fail, fix the bug, and pass the test, you won't have to ever worry about users seeing that bug again :)
but I don't ever see myself in the "test ALL the things" camp
Good for you. Extremists on all sides are usually wrong.
Shoot for "test MOST OF the things" or "test the MOST IMPORTANT things" or even "test just enough things so that you know if change Y totally breaks MOST IMPORTANT feature Z".
I'm pretty far into the extremist side of TDD, and I'll say that there is a thing as TOO many tests. Your test suite needs to run fast to be really useful.
If you have thousands of full stack integration tests that takes an hour to run, you're not going to run them as often as you should be, if at all, ad might as well delete them.
Completely false. You need to know that your change doesn't break the build, and if you wait a long time, you will have mentally switched gears when informed of the breakage. This is a big productivity sink.
(What's worse is when changes come in faster than the CI system can run the tests. Then you don't know which change broke your build because many changes were tested at once.)
Anything longer than 10 seconds is too long, in my opinion. As soon as you can get up to get coffee while your tests are running, you've lost a lot of productivity. (I recently finished a project where the tests took about ten minutes to run. That meant I could only change code 50 times per day. If the tests had taken 10 seconds, I would have been able to be make 300 changes per day. That's a 6x productivity increase right there.
Fast running test suites are absolutely essential.
That's not strictly true: you'd still like to get reasonably quick feedback, especially if you're trying to make a release.
It's also nice to be reasonably confident that your commit won't break the build, for which you should probably run a good chunk of the tests before committing.
There are some workflows (Gerrit springs to mind) where you can let the CI server work on your code without breaking anything else, but even then there's a cost to the context switch when a test failure means you have to return to a piece of code you thought you'd finished.
My experience shows that tests is not very useful in protecting from "hard" mistaken (like unusual combination of inputs, missing condition branch coverage, etc) because even with 100% code coverage you don't actually cover 100% of input/state combinations. And things you didn't think in development are usually things you didn't think in tests too. Tests are, however, always been amazingly helpful for me in:
1. Protecting me from stupid mistakes like using wrong variable in parameters, etc. (yes, it is embarrassing to have something like this, but I better be embarrassed by test and fix it before anybody seen it than be embarrassed by somebody else hitting it when using my code).
2. Ensuring refactoring and adding new things didn't break anything.
3. After "hard" bug has been found, ensuring it never reoccurs again.
As for dealing with authentication, etc. - that's what unit tests are for, testing stuff that is under these layers directly. And I don't see it matters what you are using for tests - almost any framework would do fine, it's having tests that matters, not how you run them.
I think you can unit-test javascript too, though I never had to deal with it myself so I don't know how.
I think that the main advantage of unit testing is that you have to write testable, modular code. It ensures a sound design, which is the cheapest phase to catch bugs. The regression proofing is not a particularly big advantage of unit testing since functional and integration tests catch more bugs anyway.
You should check out QuickCheck for catching edge cases you did not think of. The idea behind QuickCheck is simple--you specify invariants in your code (called "properties") and the framework tests them with random inputs.
This tool is very widely used in Haskell, but it's been ported to a whole bunch of other languages and could make your testing more thorough. In Haskell it's also easy to use and more fun than normal tests, but I don't know what it would be like in a different language.
At first I was doubtful about testing my JS code, but nowadays I do enjoy it much more that testing the Rails backend. I use my own gem guard-jasmine that runs the specs headless on PhantomJS and it's a real joy! My whole spec suite with over 1000 specs runs in under 3 seconds. I use SinonJS for faking AJAX calls to the backend, but that's just a small subset of all specs since most stuff isn't interacting with the backend.
The point of testing/TDD for me is not (just) about preventing bugs, it is more about having quick feedback. Running a test is faster than waiting until it is deployed and manually clicking around in an application. It is kind of comparable to using a REPL.
"- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ..."
I feel that way often too but I write test more as a specification for how I want the code to work then as a catch all bugs thing.
"- I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite
- More code is moving to front-end Javascript stuff ... so, now I have to write Rails tests AND JS tests? Sounds delightful"
I feel your pain, I code stuff that use WebGL currently and I find it hard to test that stuff.
I've found tests very useful for refactoring. I can pretty much go wild, as long as the tests pass at the end.
About bugs in production, after you find a bug write a test that exercises that bug. Then make the test pass. That way, you're unlikely to ever have a regression on that bug.
For browser-side UI tests, selenium is very useful.
What bugs me:
- Testing frameworks and "best practices" change way faster than language frameworks and I simply can't keep up. What rspec version do I use with what version of Rails? Now I have to use Cucumber? I learned some Cucumber ... oh, now Steak is better. [rage comic goes here]
- Most bugs/edge cases I encounter in our production apps are things I'd never think to write a test for ...
- I deal with custom domains, authentication, and 3rd party API calls in almost every app we have. IMO, this adds 20% or more to the (already high) testing overhead just to get these things configured right in the test suite
- More code is moving to front-end Javascript stuff ... so, now I have to write Rails tests AND JS tests? Sounds delightful
Feel free to try and convince me otherwise, but I don't ever see myself in the "test ALL the things" camp.