Archive for category testing
I’m giving a Virtual Lunch and Learn talk this Friday, June 26, at 12pm Central Time. You may register here!
This has been my favorite talk for the last three years or so. I’m going through the content and updating it to reflect feedback I got during this time. I hope to see some of you there!
Testing in Agile: from Afterthought to an Integral Part
Many who try to start automating tests end up giving up the practice. Those tests seem really helpful at the beginning, but are abandoned over time. Even the practice of Test-Driven Development (TDD) faces similar issues, with many giving it up.
How do long-time practitioners do it? Or, perhaps more importantly, why do they do it?
Let me share my experiences in this area, starting with unit tests way back in 2004, navigating through lessons learned the hard way, and ending with my current approach to automated tests, code coverage, TDD/BDD, and how I use those techniques to bring together developers, QA, UX, Product Owners, and Business Analysts .
I remember years ago a person saying that “a QA’s job is to find bugs in the programmer’s code”. I’ve actually heard that from quite a number of people, as early as last year. I remember companies rewarding QA employees based on the number of bugs they find. I believe it’s kind of hard to keep a good relationship between QA and programmers in such environments.
In Scrum, both programmers and QA are developers, because…
- They both run the software and click around to make sure make sure it works. They test it.
- Programmers automate their tests. They write tests.
- QA automate their tests. They write tests.
The nature of the tests they write are different. The programming languages they write are likely different. But they are both contributing to the product’s development. That’s why they’re both called developers.
Yes, each one leans towards different areas of software development, but they’re not working against each other. They’re collaborating. Any and all efforts going towards improving such collaboration should be potentialized. Just a few ideas:
- If programmers are done implementing code, they can offer help with the QA efforts (testing features implemented by other programmers, writing automated tests, writing scripts that can speed up the testing process, etc.);
- If a QA personnel are done testing what’s available, they can offer help clarifying acceptance criteria to the programmers, they can help programmers write the specs for their tests (given-when-then!);
These are just a few thoughts that I’ve been sharing at my “Testing in Agile” talk, which comes from putting that approach into practice with teams I work with.
If you’d like to get more insight into this kind of approach, please check out this great video my friend Daniel posted recently.
I don’t think I’ve ever met a developer who hasn’t had to answer this question: “Why are you writing tests?!”. Some have given up the practice because they grew tired of that, others have moved on to places where they don’t have to fight this uphill battle. Fortunately, we also have developers such as my fellow Improver Harold, who believes in and follows the practice, and can articulate explaining many of the reasons why we test from a developer’s point-of-view.
I have heard many reasons why tests are NOT written and I plan on writing individual posts to tackle those at a later time. For this post, I’d like to offer you my thoughts and answer to the initial question.
Such as with most developers, my answer used to be along the lines of “I write tests to make sure my code works!”. That answer evolved into incorporating “…and it also allows me to refactor my code. Look at how clean my code looks now!”.
However, bugs would still show up with my fully-tested code. Other developers would also have trouble working on fixing it because they couldn’t understand my tests.
After several years of that, I started seeing why it was so hard to get people’s buy-in on testing. Did you notice the word I had bolded on the previous paragraphs? Yup, “my”. I was making it all about me.
Many people use the following analogy to justify writing tests: “Doctors scrub their hands before working with a patient, because that’s the right thing to do!”, or something along those lines.
Do the doctors do it for themselves? Nope.
So the short answer to our initial question here (“Why are you writing tests?”) should be: I am doing that for you!
Or, a slightly longer elaboration:
I am doing it to make sure what we are building reflects the needs of the business as we understand it now.
This inversion in the motivation changes the dynamics of the relationship considerably; if our practices bring value to others, we’re way more likely to get their buy-in.
This realization didn’t come to me overnight. As I check out my posts on testing, I realize the first one dates back to 2008 and in it I say it was 2003 when I first heard of unit tests. Maybe my motivation shifted when I went from Arrange-Act-Assert to Given-When-Then. From that, the next step had to be the “No GWT? No code!” approach.
To wrap up this post, I’ll drop the quote I have on my business card:
“What you do matters, but WHY you do it matters much more.” – unknown
“People don’t buy what you do, they buy why you do it.” – Simon Sinek
We normally hear that we should only be writing tests for our code, not someone else’s (external libraries or APIs). For example, if we are writing tests for code that calls out to an API, the test should mock the dependency on the API and only verify that the calls are made as expected and that the results are handled as expected. The test should NOT verify that the API produces the appropriate results for the given the input; that would be testing someone else’s code.
I agree with all of that; for unit tests.
However, I’d still consider writing integration tests against that API, NOT to test someone else’s code, but to document our assumptions about the API.
Why? Because our code relies on those assumptions. That’s a dependency.
What happens if the API implementors decide to make changes that break our assumptions? Without integration tests validating those assumptions, all our unit tests would still pass, but we could end up with either defects or inaccurate results in production (which could go unnoticed for a long time).
Another added benefit from the practice of writing such tests is that, should a new version of the API come out, evaluating risk levels of consuming the new version becomes much simpler: just run the tests against the new version.
Last but not least, say an API offers a large variety of features that could be used; having tests that describe how we use that API makes it much easier for developers to learn the details of how we depend on it. Such understanding, again, helps with both assessing risks when consuming different versions of the API, as well as assessing a potential replacement of the API.
Dependency management is very important!
Did I get your attention with that title? I hope so.
Let me clarify it: most people use code coverage for the wrong reason, making it worthless. I know I did that for a while.
Back when I first learned about writing tests, it didn’t take long until I heard about code coverage, and then the search for the magic code coverage percentage started:
“100% code coverage?”. Nope, that’s impractical
“50%, then?”. Nope, too low.
“92.35?”. Yeah, that’s more like it! Well… not!
Seriously, I’ve seen some crazy numbers as the required code coverage policy out there.
Writing tests for the sake of bringing up code coverage will NOT:
- make the code quality get better
- delivery better value to the business
- make refactoring easier
I have seen tests out there that have no assertions.
Those tests have hundreds of lines of code (usually involving some crazy, unreadable mock setups), and no assertions. Why? Simple: because developers had to satisfy the policy of XX% code coverage! The only thing those tests do is make sure no exceptions get thrown when those lines of code run. It’s pretty much a smoke test.
Such tests do NOT bring value. In many cases, the tests may be exercising lines of code for features that aren’t even used!
Think of new developers joining the project and having to go through all of that code trying to learn things, figuring out how things are done. Even existing developers after a while will have a hard time remembering why certain code is there.
So, when is code coverage worthwhile?
When writing tests for existing code!
Once a conscious decision has been made about what we should write tests for, start by writing a test that does a clean pass through the code (meaning, without it throwing exceptions). We’re likely to uncover dependencies we didn’t even know the code had. This will be a big integration test. I wouldn’t even fret about putting assertions in that test. Why? It’s very likely I don’t even know what the expect outcome of that code is at that moment.
With the first clean pass in place, look at the code coverage number. If we have about 30%, that’s too low, so we need to look into writing more tests that go through different branches of the code. Once we get to a number we feel comfortable with (that could be 70, 80, 90%… it really depends on the risks and costs of breaking changes), then we can start capturing the current outcome of that code, by writing assertions for it, bearing in mind that the outcome may not even be accurate, but it is what the code produces without any changes.
Now we can go ahead and start refactoring the code, making it more readable, without fear of breaking whatever it currently does. As we split it into smaller chunks of code, we identify opportunities to write new unit tests for those smaller pieces of logic.
Eventually, we’ll get to a point where that initially big integration test may either end up not being relevant anymore (and can be removed, replaced by the new unit tests), or, it can be refactored to something that more accurately describes the reason the code exists; the big picture.
Once the team starts using code coverage for the right reasons, then the metrics can be changed over from “Code Coverage” to “Feature Coverage”. Knowing what features are covered by test is a far more valuable practice.
If you choose to get one thing out of this post, may it be this: read Working Effective with Legacy Code, by Michael Feathers. It still one of my all-time favorite books.
Another common question I get from developers who are starting to get into testing (or even from devs who have been doing it for a while): “how do you decide what to write tests for?”.
This question normally applies to brownfield cases (existing codebase). There’s already a lot of code there. Where do we even start? Yes, maybe we write tests for the new code, but what about the existing one?!
Here’s my personal technique for it. When working with an existing codebase, I’ll ask the business:
What is the single most important feature of this product?
Think of the feature that, if broken, will either cause the business to lose money or not make money.
THAT is where we start. Those are our must-have tests.
Once the most important features have been covered by tests, the next question is:
What is the feature or area of the system that when you tell developers they need to make changes to it, they feel like running away?
That’ll usually surface areas where the code is a mess, complex, convoluted. Hence, it needs tests, so developers can feel safe making changes to it, refactoring it. Now, we only move on to this one when the features that came out of the first question above have been covered by tests.
A very common question I hear from developers is “How do I write tests for private methods?”. My immediate answer is “You don’t!”. Technically, if you’re in C# Land, you can instantiate the class and then use Reflection to call the private method. But please don’t!
Say you have some class like this one:
Of course, instead of comments, you’d have the actual code. You get the point.
You then decide to clean things up a bit and extract the “make sure all the ingredients are in” code into a separate, private ValidateIngredients method, like so:
That’s usually the moment when developers ask “how do I test that private method?”. If we have tests for the main method (DoTheMagic, in this example), then ValidateIngredients already get test coverage.
Quite often, when developers feel strong about having separate tests for a private method, there’s a clear indication that the private method should really be a public method on a separate class. Think Single Responsibility Principle.
Following the example above, we create a Validator class and move our validation logic in there:
And then we use that validator in the previously shown class:
You’ve probably noticed that we also introduced an IValidatePotion interface. Think Dependency Inversion Principle. One of the benefits here is being able to isolate tests for the AwesomenessPotion and AswesomenessPotionValidator classes.
I’m often asked about comments in code: when to do it, how to do it, what to put, etc. I’ve recently run into Steve’s post about when to comment your code, left a comment (!) there, and we got to expand our conversation in his podcast/screencast (link at the bottom). I’ve decided to create this post to consolidate the links and info shared during the interview, to make it easy for folks to find the material.
I remembered writing blog posts a couple of times over the years and it’s interesting to see how my opinion on this subject has changed over time.
The first post goes all the way back to 2005, with me asking “can you plesae put some comment on that Regular Expression?”. 15 years later, I still ask my smart friends to get me the RegEx I need, along some comments as to what each piece does!
In 2007, I was big into using Xml Comments, GhostDoct, and Documentor… I’m not anymore, as documented 10 years later with my post “XmlDoc Comments: Auto Generate and Hide the Clutter”. In nutshell, if we’re documenting a public API, yes, by all means let’s put in that documentation, but making it count: commenting the GetCustomers endpoint with “Gets the customers” doesn’t add any value to the effort!
My practice of making comments stick out as a sore thumb posted in 2010 still stands in 2020: I still set up all of my IDEs in that manner and it still produces exactly the outcome as I intended.
When I do want to drop a quick TODO comments in code (watch/listen to the podcast interview when it’s up to know why I might do that), I have templates on my IDEs to automate that: ReSharper in Visual Studio, User Snippet in VS Code, Live Template in RubyMine.
In regards to code that’s commented out, like so:
We should be using a source control system; if we ever want that code back, we have a way to bring it back. So just remove it!
Still using the example above, notice that each if-block is preceeded by a comment. Is it really necessary? How about removing the comment and extracting the expression into a method that tells us the question being asked?
There’s also the “narrator-style” comment:
Narrating every single line of code is very annoying (by the way, I think I wrote the code above many moons ago). If the comments were written initially as a placeholder for the steps that needed to be implemented, let’s make sure to get rid of it when we’re done.
Last but not least, some people say (I’ve said it myself) that comments should document “why” the code was written in such manner. I’d propose a variaton to that: how about documenting the why with some good specs (or tests, if that’s how you prefer)?
Now, I’m not refering to tests that look like this:
Such test doesn’t tell us the “why”; it tells us the “how”. I mean this kind of test (now you’ll see why spec fits better):
Summing it up, this is how I prefer to “comment” code:
- Given-When-Then specs
- Writing code English-first
If I do have a real need to drop an actual comment in code (“why do we have this query in the code that has to potential to perform badly”), I’ll probably drop a quick comment, with a link out to the issue tracker, where I’ll put more context about why the code was left like that, and where a Product Owner can decide when it’s appropriate to address the situation.
Any comments? 🙂
And for audio-only version:
My “Testing in Agile: From an Afterthought to an Integral Part” is becoming a hit: I’ve been receiving great feedback and compliments from attendees and many requests to deliver it as a Lunch and Learn at their companies (drop me a note if you’re in the Houston or surrounding area, and I’ll come to your company, too!). I’m so pleased with the response I’ve been getting that I really feel like working on polishing the presentation further (better title, better description, etc.).
One of the points I bring up on this presentation is my “No GWT, no code!” movement. 🙂
As it turns out, people are responding well to that! At some conferences and user groups, when I mention the movement (which initially just came out as a funny remark), I hear attendees saying out loud “YES!!!”. But now, the coolest thing happened… check this out:
I just got to the Improving Houston office and had an envelope that came in the mail for me. What is it?
Yes, “No GWT, no code.” stickers!!
An attendee to my talk at one of the conferences felt inspired, got these made, and mailed it to me. How awesome is that?!
The realization that you’re inspiring others with your work and attitude brings so much joy, while it also keeps the flame burning, providing energy to keep pushing forward. You should try it, too!
I’m going to be speaking at the AgileShift Conference in April!
I’ll be delivering my current favorite talk, for which I’ve been receiving great feedback from people who have seen it (ranging from software developers, QA, business analysts, product owners, etc.):
Testing in Agile: From Afterthought to an Integral Part
Testing cannot be an afterthought; it has to be an integral part of software development. Is it something that QA teams do? Or is it part of a developer’s duties? Do business analysts play any role in it? What is test automation? Unit test, Integration test, Test-Driven Development, Behavior-Driven Development… what do those mean?! This session addresses all of those questions, as we talk through the importance of tests, the collaboration among team members, the techniques, and practices around different kinds of automated testing.
I hope to see you at the conference! 🙂