Agile Development: why it rocks, who it helps, and why it's failing
The Agile method was the best thing that happened to me in my career as a developer. Before I came across Agile, one of my biggest frustrations was the difficulty I had with getting large assignments and breaking them down in little pieces. Usually, in the beginning, I'd be at full steam, really motivated, really excited, and rearing to go. This would last for a little while then the chunky middle part of the project came, some of the more boring parts, and before you know it I'm at a crawl. Then, when things are due in a week, full panic mode would commence and there would be ridiculously late nights followed by incredibly sleepy mornings, then sleepless nights all because I thought I had a handle on the scope and how long it would take to complete.
Agile was a new awakening for me. For those of us who aren't aware the concept is to plan your software development X amount of weeks in advance then break up that time into still smaller sections before you get started. The reason for this change is that seeing how dynamic and fickle the world of software and software development is it just isn't prudent to plan that far ahead. You go with the brightest and best new tools and concepts and three years later when your plan is complete it's mostly obsolete. Planning only small parts at a time allows for the ever changing industry. The unit of weeks that you choose to use is called an iteration. Each iteration you plan out exactly what your team will deliver when it is complete. Then as a group you break down your plan into projects. Then personally you break down your projects into tasks that are anywhere between .5 and 3 days long. You then have daily stand up meetings to discuss how you are doing on your tasks, if you are ahead, or possibly running into issues you can cut them off at the pass. There are all kinds of permutations like Scrum and XP, but you get the drift.
I myself took to this like a fish to water. No longer did it feel like I was sitting in front of a ton of unpeeled potatoes, I had something to accomplish today. Just one thing. Then tomorrow I have something else that will take me two days, then after that another day on something else. Honestly, I haven't looked back since, I could never go back to the old way of doing things.
Frankly, the industry hasn't been the same. Agile development lends itself to huge wins for well designed, awesomely developed software. However, nothing has surprised me more then the rumblings I've been hearing lately about how “Agile sucks, everyone said it was so cool and it just doesn't work.” “We have an Agile shop and they are four months behind and no one gets anything done ever.”To me, that's like saying “Water is a solid” or “Aaron Heilman is a dependable reliever,” it just doesn't make any sense.
So, I've done a little investigating of my own. An insightful email from a reader who goes by the name Abhishek Parolkar really got my brain churning. I asked myself what was consistent about the complaints I have been hearing. I had my Eureka moment a few moments later (sans bathtub)
See, what was making these Agile shops so successful and productive was not purely Agile at all it's a combination of some very key traits that allow for unfettered growth in technology and development. What the initial adopters of Agile had in common was that it was a “start up” type of idea. In other words, it was the small companies adopting this methodology, not the bigger firms. And this makes sense, because it's a lot easier for 5 guys to make a process change then 150 people of varying skills and specialties. We see this everytime a new software or tool that is popular among industry professionals. Like, right now it's Git and RoR the majority of the shops you will see focusing on these things are the progressive, smaller companies.
So, what else did these smaller companies have in common then the fact they were using Agile? Well, the time from approx 2003 on I think we will grow to view that as the re-awakening of the start-up. A little of the sting of the “bubble burst” had passed, people got some of their money back, and had some new ideas. The perks and the relaxed atmosphere and the focus on employee satisfaction retuurned. The developer as a species appreciates these type of places, it's our natural environment and the condition in which we thrive. Where our ideas are welcomed, where there is no pre-approval paper work, where there aren't 2 week long QA processes, where the hard part of implementing a great idea together is walking from the big white board to your desk without getting another one.
See, what's happened lately is the bigger IT departments have attributed this success and prosperity to the Agile process, when really they need to look at the minds that recognized this process as a great idea and what other things they think are important. Making a process change as big as a new methodology is huge, so most places with > 50 people are either slowly switching over or seriously considering it now. The kind of complaints I've been hearing are followed by statements like “Plus, my boss is such a dick he makes me subtract out my cigarette breaks when I bill” or “I just can't stand it everything they ask me to do just makes no sense and there is nothing I can do about it.” That's where the larger companies are loosing it, being an Agile shop isn't good enough. When your employee dreads Mondays no methodology is going to make them want to work hard for you, and therein you get no similar productivity boost as the other Agile shops, you may have better architected software, but the huge measurable profitability successes just aren't there.
What about Google you say? Well, they are the perfect example of how you do it correctly when you have a lot of people. They are a large company, however they run their development teams as a bunch of smaller more intimate groups. They take the time to make sure every employee loves being there, that their immediate needs are met and all their individual ideas are considered and encouraged.
So what's the moral of the story? Implementing the newest ideas is simply not going to get you the incredible department you have been waiting for. Sometimes, you can get caught up in pomp and circumstance about your “sophisticated and professional” team. However, your “sophisticated and professional” team of developers is hating every second, so you get behind, and you miss deadlines, and you get stuck in backlogs. Not because Agile doesn't work, but because when people aren't happy they don't care if your company succeeds or not. They care about their Saturday nights, or where they are going after work, or if they remembered to TiVo Heroes. So it's simple: when you care about your developers, they care about you and that ALONG with the best practices and tools are the keys to success.
Similar Posts
- Three Great Reasons Why Even Lonely Developers Need Source Control
- The Hitchhiker's Guide to Refactoring: Why, Where, When and How
- Five good reasons to stay away from third party applications (even open source ones)
Comments
rowland on on 8.26.2008 at 11:15 PM
I would contend that Agile methodologies are a new craze - their take up maybe, but DSDM has been around since at least 1995.
The success or failure of using an Agile methodology is partly based on the attitude of management (supporting project managers, team leads, and developers) and how the methodology is employed within that organisation. SCRUM is great, in that it provides a framework that you can use to supplement existing processes that already exist (continuous integration, QA, etc.).
Stakeholders *must* be brought on board at all levels, so they understand the risks and benefits and can themselves be agile when the Agile methodology does not fit well with the business strategy. If one part of the Agile process doesn't work, don't use it - possibly embrace agility from somewhere else.
Ix on on 8.27.2008 at 7:11 AM
I would have to say that I agree, regardless of what process you use, if you aren't taking care of the needs of your staff, you aren't going to succeed. It seems a lot simpler to most members of management staff, "We can use this new process and it will change things. Our production cycle will increase, and profits will go up".
While that could be true they don't seem to pick up on the same thing that workers have been trying to relay to management for as long as management has existed...Treat me fairly or find someone else who doesn't care.
It's simple math really, a given methodology of production can only give you "X" percent performance increase before you have added too much stress to the individual producer. At this point you really only have two options, push the producer harder, possibly endangering your current "X" percent production increase, or augment the producer in some way.
Whether that is in an incentive package to ease the mental or physical burden to said producer, or by adding "X" numbers of producers to offset the workload, performance numbers can only go so far before you need to change something greater than a meeting or a memo.
So to sum it up, I blame lackadaisical management staff for their paltry implementation of what could have been a great productivity tool.
Daniel Pietraru on on 8.27.2008 at 7:26 AM
I would say that in every situation out there people make processes work and not the other way around. Good people will make any process work because they cut to the spirit of that process and do the right thing. But there is always another group that applies any process in letter. And then it does not matter if the process is called Agile or Waterfall or whatever. A change in process has to come from within in order to be successful. Imposed process changes will end up just changing the label while people are doing the same thing underneath, a thing that is not the new process, wasn't the old process, but just something that helped them go through the day. Companies have their own culture established in time and this culture determines what kind of people are hired there and how they are educated after being hired. And they in turn reinforce the culture of the place. So I guess Agile can be successful or not based on the culture of the place. A while back I wrote a post about Agile adoption in the enterprise: <a href="http://littletutorials.com/2008/06/24/the-agile-800-pounds-gorilla/">The Agile 800 Pounds Gorilla</a>. You might find it entertaining... or not. Good reads on your site, by the way.
ncloud on on 8.27.2008 at 8:01 AM
"When your employee dreads Mondays no methodology is going to make them want to work hard for you..."
For the win!
Slackmaster K on on 8.28.2008 at 7:31 PM
This is how I see Managers' mindsets:
1. Steal Underwear.
2. ?
3. Profit!
"What the hell, let's plug Agile into #2 and see what works."
Actually, I believe the reasoning behind breaking code into small iterations is so you always have a compilable, shippable product - Even if it doesn't do anything.
Meetings suck the life out of me. How can you possibly not feel the same way? Every hour wasted in a meeting is an hour you don't get to spend doing something fun - like coding.
Something to accomplish today... So if you can only see three potatoes, you don't care about the ton of them you don't see?
As for being four months behind... The way I understand it, if a shop is really Agile, they don't have deadlines. They estimate the time they need based on current velocity and give that to the customer. The customer does not make the dates in Agile.
I worked in an Waterfall shop. There were great perks and a relaxed atmosphere; employee satisfaction was easily accomplished because when you have good people building a good product, it doesn't matter what methodology you use - You'll succeed and feel great because of it.
What 2-week QA processes are you referring to? In the Waterfall shop I came from, elevates to Test were as simple as logging into the server and copying the files when the Testers were ready for code. How quickly that happens is a simple matter of how good your Testers are and how big your changes were.
At my current position in an Agile-ish shop, such elevates take a week of time, six hours of meetings, PM involvement, and half a bottle of Tylenol. After all, one must organize with so many teams (Dev, Test, Environment, Project Management) and document so much of the process. We actually have Pre-(Pre-(Elevate Meeting) meeting) meetings (http://www.zi255.com/?Req=Post&PID=560). Whether the Company or Agile is to blame remains to be seen, but it's clear that you can have too much organization.
Maybe I'm more Cowboy than Waterfall, but I never had a problem implementing an idea before I joined an Agile company. Now I'm nothing more than a code monkey. The engineering and problem-solving have flown the coop, and what's worse: Agile has taken all the fun out of it. At my last job, I felt free to implement great idea whenever one occurred to me. I could whiteboard it and consult with my peers, if I felt like it.
Last job, Waterfall, small team (1-2). Completed project early and under budget. Current job, Agile-ish, huge team (60). Two years behind, 105,000 man-hours to go, and enough money left over for a tenth of it. They're now moving back into a "Traditional" methodology - they're afraid to use the word Waterfall - because Agile "just isn't working". Make of that what you will; but these are strong Agile proponents saying it.
So I guess maybe you're right in that Agile works better in small shops than it does in large ones. I'll have to see if I ever work in a small Agile shop. I do dread Mondays now. This project would have been done two years ago if it were just architected right in the first place. They could have done it with six people.
So I guess the moral of the story is thus:
1. Start with a good idea.
2. Get smart people to do your Design and Development.
3. Make your people happy.
4. Methodology doesn't matter; but most Devs are strongly in favor of one type.
5. If you're not Google, you're probably doing it wrong.
James van Leuvaan on on 8.29.2008 at 9:36 AM
I agree with you, however sadly have to play devil's advocate.
It is one thing to know that projects should be properly scoped, and organized in such a matter that all nuances of the project are defined, with reasonable time lines, and functional build progression.
As a matter of fact, my business partner and I just last night were stating quite simply, "you know what? it would be awesome to actually have the time to properly scope out a project, and build it without it falling into that last minute chaotic, "oh by the way, can you do this?" and, "Oh and we forgot to tell you..." or, "oh. Well we actually visualized something completely different, will it still be ready on time even though we've completely changed what we want now, in comparison to what it was we initiated?"
Yes folks... that is the nature of business in the real world. Gone are the days of the professor assigning something, and then we get marked on how well we do something - on a time line that is far too long to accomplish a task.
The fact of the matter is, I am a full on fan of the agile methodology - however... very few organizations - beyond perhaps Microsoft, Intel, Sun Microsystems - have the staffing, availabilities or man power to fully portion a project in chunks to ensure it returns on time.
It is the nature of IT to be up late, tired in the morning, and running at Mach II with our hair on fire.
It is the age old complaint of the technologist. It isn't likely to change with any new breed of technician, no matter how idealistic or optimistic we all (were) or may be for any given generation.
Glad you're enjoying it though... hope you can maintain.
Abhishek Parolkar on on 9.03.2008 at 10:45 AM
Thanks Sara , for sharing your thoughts about conversation that we had.
I feel the term "methodology of development" help software salesmen more than the natural coders like us. While implementing any idea every thing to me looks fresh and naturally agile (not Agile as in Agile Methodology), Every engineering problem brings a question mark on how to solve it and I rarely found a process to attack any innovative problem.
I certainly agree with Slackmaster K's points.
I do respect and use the concepts of Agile @ work. But point to note is , the real essence of agile isnt conveyed to many who claim to be agile shopkeepers for only reason that majority of developers still choose agile because its turning out to be buzz word. I wonder how many devs can understand Agile without having failed many-many times in delivering solutions in other methods. On the other hand, how many can deliver good quality software in name of Agile without being ever failed even once. I am a software craftsman and none of my work ever evolved 100% perfect at first instant hence I believed in fast iterations (Am I Agile?)... At another times... I crafted innovative idea and didnt involve anybody and sat in black box till I had qualitative prototype (with buggy code)to speak for itself...(Am I not Agile?)
For a great surprise, every favorite piece of software I use is crafted by atmost one or two passionate programmers every body else just followed their steps... There was no process of communication/ development / QA. Because all that just complicate the innovation... We got to agree "Software is a complex material" when it comes to communicating ideas and explaining things to fellow mates when innovation is in "pre-mature" state. So the question isnt about agile methodology but about "When does a methodology gains enough respect such that developer starts using it for his/her own software craft?"
Isnt that the "question" that answers when team starts loving the methodology?
Julian Simpson on on 9.07.2008 at 5:05 PM
In my experience (I worked Agile consultancy for 4 years) you tend to get little outbreaks of Agile in an organization. There will be team of developers, business analysts and testers all cranking out good software and it will mostly be under the radar of the rest of the organization. Those projects can be successful.
Trying to impose something like that on a large scale is fraught with difficulties: some people don't like to collaborate at work. Engaging them to pair program is hard. Running an Agile project is hard anyway. Poor developers can take a lot away from a project when they are given the high degree of trust that an Agile project has. And it's not clear how to interoperate Agile development teams and operational teams.
(my pet peeve is the last one. We don't often think about things like deployment until it's too late)
However, with the right team and the right conditions, Agile can be pure magic.
Devo on on 9.26.2008 at 4:35 PM
Agile in concept is much better than Waterfall because there are very few projects where you can know everything before coding and put that down into a big detailed spec that is supposed to be perfect and isn't supposed to change.
I work at a company that has a mostly Waterfall methodology and it is very inefficient.
Problems or not enough detail in a spec require change control documents, which get voted on by the product team, most of whom have no idea what the issue is. This discourages developers from trying to make changes to a project for the better. If I have to write a change control, argue my case, and have members of a team from marketing, customer service, sales, etc. who know nothing about the issue, how big the change is, or what constitutes good software, vote down my change control, then no thanks, I just won't bother anymore. I've seen this deflate the enthusiasm of many developers who just want to develop good software without hassles.
Waterfall also has problems with how estimates are done. If you said (or were just given) 2 months for a project, then you are expected to have the project done in that time no matter what. This encourages developers to overestimate and then not do any extra stuff they might have wanted to do to make the product better if it requires any extra time. It also encourages developers to dog it at the end if they do happen to finish early, so they are not called out for overestimating.
Another problem with Waterfall is the fixed cycles that aren't supposed to overlap.
You can't start programming at all until a spec is finalized and you write the technical documents. Testing can't start testing until the programming is completely done and a complete product is delivered to them. And you can't start on the next project until this one is done and shipped.
So with all these issues I have with Waterfall, you would think I'd love Agile. Well, not really.
In concept Agile is much better than Waterfall. It's agile after all, how is that not a good thing? It's more flexible and condusive to needed changes. It's more efficient in that many parts of the SDLC can occur at the same time. And of course it's more fun for everyone. Yeah.
I have a printout of the Agile Manifesto on my cube wall. It sounds great. "Individuals and interactions over processes and tools".
"Responding to change over following a plan."
Thats all good stuff.
Then you discover that Agile is implemented in companies in much the same rigid manner as Waterfall.
There are processes that must be adhered to for no good reasons. There are procedures that are complicated and offer few benefits.
The problem is that people read about or hear about software methodologies (Agile or other) and automatically think that these must be good ideas that should be applied to all companies and all employees within a company, just because somebody who wrote a book told them so.
I think this also occurs because there are alot of people (mostly clueless a-holes who can't think for themselves) who just want to tell others what to do. It doesn't matter if it won't work well, this isn't a good situation for it, or it will make others unhappy.
For example, one company I heard about forces all developers to do pair programming. I've actually never tried this myself (except for an accasional 10 - 15 minutes of looking at code with another developer), but I don't have to eat shit to know that I won't like it, and this shure looks like shit to me. I was reading a book on pair programming where the author was saying how great it was and how it actually increases productivity. Two developers pair programming are more productive than those same two developers working by themselves. That was his claim, but not much was offered to back up that claim. I know I would never work at a company that forced me to do pair programming. I'm not saying it's bad for everyone. It could work well with the right people - they have to want to do it though. I could certainly see it working in a situation where one developer is very familiar with a product and another developer is new.
So basically what I'm advocating, is that the people who are in charge of putting a software development methodology in place should not get their ideas directly from a book, a speaker, or a can and force it on developers. Developers need ownership of their own methodologies and flexibility to work the best way they know how.
Usually the developers know what they need and want and would be best much more than those in charge.
It's time we stand up and give anyone who tries to force feed methodologies on us a special new assignment - flipping burgers.
Devo
-The pissed-off programmer