Respons to "Estimation Isn’t Agile" article

This is a response to a blog post written by Craig Buchek: http://blog.boochtek.com/2014/03/23/agile-estimation
I claimed, on twitter, that I had this opinion once as well, but have "changed my mind". He then asked me to write a blog post about my thoughts on it. It's a great idea. So, here it is.
I'm going to quote his blog in here, for easier reading.

I don’t believe that estimation should be part of any Agile practice.

Estimates is a powerful tool. Like any powerful tool, you must learn to use it first.
If you can be agile with or without estimates is an opinion, I don't question that :-)

One of our managers recently mentioned that we hadn’t met the “contract” that we had “committed” to in our last iteration. This was complete nonsense, because A) we hadn’t made any such commitments, and B) we completed many more story points than the previous iterations (and without inflating story points).

The text makes me think that the estimates was delivered as "single point numbers"? Estimates should come with a range and probability. There are lots of ways to do this.
Secondly, I think it's time to talk to this manager. She needs to learn how to use estimates as well. It's not only developers that needs to improve. Solve the root cause first, don't "blame" estimation. Her behavior will not get better if you (or anyone) stop doing estimates.

But her language made me come to several realizations. First and foremost, estimates are contracts. Sure, they’re not supposed to be treated as commitments, but they almost always are. And what does the Agile Manifesto say about this? It says that we should value customer collaboration over contract negotiation, and responding to change over following a plan. So it’s pretty clear that treating estimates as commitments is completely counter to the Agile values.

The reason estimates are treated as commitments or promises is because how we use them and present and communicate them. If they become contracts is up to you.
E.g. if a bunch of developers take half a day (or a couple of hours or maybe days) to come up with a, probably "accurate", number, no wonder the estimates are treated as commitments. Who wouldn't? They don't know how you came up with that number. They think you are professionals!
Instead, deliver a range with probability. Some say this doesn't work. I disagree. It's not uncommon to hear the doctor say: "It's a 70% chance this operation goes well". You can handle that, everyone can handle that. And then you trigger another discussion, an important one: "What's the 30% risk?"
And, some kind of "promise" must be made. The ones paying us to write their software should be able to know, in advance, roughly how much they're expected to pay and when to get it. Why should we not answer that?

Why does this matter? What benefits do the Agile values bring us? I think the biggest benefit they bring is changing the way that we work, so that we can better deliver value to our customers. Without Agile, we’d just keep working the way we’ve always done things. And that didn’t seem to be working out so well. If we follow the Agile values and principles, at least we’ll have a fighting chance of improving our ability to deliver value.

Yes, agile is about seeking improvement. And doing things better. Improve how you use estimates. Improve how you use your tools. That goes for any tool in your toolbox. As I said, estimates is a powerful tool. Learn to use it. Improve, yes.

Ask yourself — have you ever seen a software development project that was on time and on budget? Where the estimates were spot-on? Of course not. For one, we’re terrible at estimating. For another, our plans change — either from external factors, or from what we learn as we go.

This is a red herring. No project should be on time and budget. Then you're doing it wrong. If you hit time/budget, then you A) Was very, very lucky or B) Adjusted things to hit the time and budget. I'd rather have something great a bit late and more expensive than something bad on time and budget. That is, if budget and/or time isn't crucial. But then, I'd have no problem cutting scope.
This goes for everyone. Put on your customers glasses - they'd probably have no problem if you signal, in time, that the estimate seems to be wrong. It should actually have been a part of the risk discussion before project was started. But still, adjust estimates when you know better. Remember: customer collaboration.

Improved Estimation

To me, Agile is also about facing reality — and embracing it. It realizes that we’re terrible at estimating. It realizes that plans change. Most Agile methodologies have some tricks to counteract Hofstadter’s law. Generally, we use relative story points instead of hours, and then use an empirical factor to convert points to hours.

Yes, plans change. Estimates change. Embrace change. Don't lock estimates, keep the conversation going. If you must lock them? Get good lawyers :-)

When this works, it is better than any other estimation I’ve ever seen. But it doesn’t work very often. People have trouble with relative estimation. How do you set the basis for what a point means without relating it to actual hours? Affinity estimation could work, but then you have to remember what the basis was. We’ve got a large distributed team, and when we tried this, we couldn’t all remember what the basis was.

I agree. Story points are "lousy" when talking to the ones paying you money to write their software. But this is only my opinion. Some use story points, and they work great. Good for them.

Since we couldn’t get affinity estimation to work, we tried changing to perfect hours (only powers of 2). But then people thought of them as time. When we took longer than the estimate on an individual story, managers and team members thought we were taking longer than we should have. So our estimates ended up causing problems.

Don't be optimistic when delivering estimates. This is a common mistake. Why not deliver your "worst" estimate? Who will be sad if you deliver faster? Probably the opposite - they'll be glad! Until they've eventually seen through your "scam" :-) And if your'e trying to "win" a project, or if customer wants to decide if a project is "worth doing" then you should *not* provide your "worst" estimate.
So, start by trying to deliver your estimates with ranges and probability.

What Can We Do Instead?

Managers want estimates so that they can have predictability. They want to know when new features will be available. Is there a better way to get what we need?

I don't think so. If we can, great! Please, let me know! :-)
If managers use estimates to create "predictability", they should improve. Nothing will fix "stupid". Other than stop doing stupid things. Stop estimating will certainly not fix that.

I believe there’s a better way — prioritization. If you work on the most important thing first, then the most important thing will get done first. We should always be working on the next most important thing.

Agree, but this doesn't exclude estimates. You can do this anyway. But, how do the ones paying us to write their software know (roughly) when the thing you're building for them is "shippable"?

What if there’s more than 1 thing that’s most important? Then you’ve failed. You’ve failed at logic if you can’t understand that only 1 thing can be most important. You’ve failed at prioritizing the customers’ needs. You’ve failed at project management.

Agree. Nothing to do with estimates though.

Arguments 

1. Why can’t you just tell us how long it will really take?

Because we don’t know. Because we can’t know. This is the first time we’ve ever implemented the functionality you’ve asked for. If we’d done it before, we’d just use that existing code. As Glenn Vanderburg pointed out in his excellent talk on Software Engineering, we’re not building software, we’re architecting it.

No, you don't *know*. That's why we're *estimating*. If someone takes your estimates for "knowing", rethink how you deliver those estimates. Again, you can't refuse the ones paying us for writing their software the right to get a sense of how much and when.

2. But we have to tell our customers what to expect.

Why? Is the product so bad that you can’t keep customers around without leading them on with future enhancements? And why do customers need exact dates? A general roadmap telling them what the priorities for upcoming features should be sufficient.

No, they don't need exact dates. Why do you think they do? Instead, stop delivering exact dates. But they do want to know roughly how much and when. And, estimates comes with a probability.
And everyone should/must have a common understanding on what "done" looks like. No chance of success otherwise.

3. But we have to have messaging about new features.

OK. Then send out that messaging once the feature has made it to Staging. Or even after it’s been rolled out to Production.

I don't understand this point.

4. But we’ve promised these new features to the customers by this date.

Ah, so you’ve made promised to the customer that you don’t have control over. Have you ever heard of “under-promise and over-deliver”? That’s how you create happy customers. Yet you’ve done just the opposite, haven’t you? And then you want to blame someone else.

Put on your customers glasses. If someone has promised to deliver something to you and they don't, what would you do?
And, you are actually at it. Don't be optimistic when delivering estimates. Common mistake. Also think about how you deliver your estimate, they should not be interpreted as promises.

Risk

Estimates are risk. But the risk doesn’t come at the end, when the estimates are shown to be incorrect. The risk was in asking for the estimates in the first place, and placing trust in them. Don’t do it. Don’t promise things that you can’t be sure of.

This is a bit of a confusing point. Yes, estimates are risk. Talk about it with the ones paying us to write their software *before* the risk becomes a problem. Customer collaboration. The risk was *not* to ask for estimates. And don't put trust in them. But that depends on how you deliver them, how you communicate them - customer collaboration.
By the way, who asks for estimates? I've never heard: "We want to buy some estimates!" What they want to know, roughly, is the cost and time. How you provide it is up to you. If you can do it without estimating, good for you! But you can't "blame" the ones paying you to know roughly, in advance, how much and when to expect it. Usually/always you must estimate that.

Embrace this reality. Embrace this uncertainty. Always focus on what’s most important. That’s how you make customers happy.

Agree. Nothing to do with estimates/no estimates though.
Usually, in all other situations you handle "unknowns"and uncertainties by having margins or "buffers". If you want to buy a TV, you have a "roof" price, but you might still adjust it depending on what you find :-) And if you go on a trip, don't you bring "a little extra", just in case? And if you remodel you kitchen, don't you have margins for certain "unknown" costs that might occur? Well, you should at least. And wouldn't you want to have someone, an expert, to point out these "uncertainties"?
Same thing with software. Talk to the ones giving you money to write their software. Let them know your "unknowns" and your uncertainties. Maybe they want to create "buffers" or margins as well. It's called risk handling. And let them decide later if to use them or not - in collaboration with you.