Software Development Life Cycle - You're probably doing it wrong.

The premise of this post is that real world Agile inspired SDLC in the form of Scrum or Kanban miss some important engineering practices and this actually looks like a process breakthrough until tech debt, failures, complexity, and slow release cycles challenge the very core of an organization.

First, a story from the project management side.

This summer I gained some painful insight into project management.  I remodeled an investment property and went over budget and over time. I found it very difficult to get accurate estimates and finish dates. I realized, while writing a check for more than I expected for things I thought I had already paid for, that this must be what it feels like to own a software company or have an IT department. And I felt so much empathy for all of my previous product owners that tried so hard to bring stability and predictability to my software projects. In the case of my remodel, there were personal issues with workers, a major shortage of certain skilled craftsman, regular low ball estimates, and a smattering of no show rescheduling. How in the world could I have planned on so many failures and delays? I thought I would bring good money to the table and get good work, but money clearly wasn't enough. Some people even worked feverishly and around the clock yet I still felt like the train could not be set back on the rails.

I can watch this everyday and still laugh like the first time!

 

Maybe it's wishful thinking or getting oversold but I have experienced many situations where process was responsible for more than it should be.

• It does not:
  • Make you good at what you do
  • Give you a vision and achievable goals
  • Ensure that you make good decisions
  • Stop mistakes or bad designs from happening
  • Ensure timelines are met
  • Ensure that everybody is happy
• It does give you scheduled events to:
  • Get feedback to help make you better at what you do
  • Get feedback to increase the accuracy of your vision and goals
  • Get feedback to help make better decisions
  • Get feedback to help correct mistakes
  • Get feedback to help predict and meet timelines
  • Have a predictable and balanced work experience that can lead to happiness and job satisfaction

Here are the problems, causes, and solutions to agile inspired software processes as I see them at the time of this writing. *I reserve the right to change my mind!

• Problem - Trust has broken down between the 'business' and the 'developers'. The business thinks the software is taking too long and has too many bugs. The developers think the business doesn't know what they want and are always changing their specifications.
  • Cause - Because of the complexity of software and the high rate of technology churn, developers are constantly learning new skills and tools. Imagine striving to be a professional tennis player, but every couple of months the official racket shape, weight, and size change.
    • Solution - Hire and pay for the best software architects. Depending on the size of your shop you might not need many, but talent and skill cannot be understated when turning ideas into software.
    • Solution - Give time for training and account for learning new technologies and patterns in the estimates. This is really expensive and not to be taken lightly.
  • Cause - Because the business is often creating something that never existed before they usually change their minds about one thing or another as new information is discovered. When the business changes a specification, it's usually a very good thing for a very good reason. The business just paid some 'conceptual debt' and increased the requested value of their software product. Unfortunately to a developer, it can seem like working on a house construction site and then the word comes down that they are actually supposed to be building a touring bus, but we are still going to refer to it as a house. To the business, it was a minor discovery and good, to the developer it's a scary game changer and they wonder how they worked this long without knowing there was supposed to be wheels on the 'house'.
    • Solution - Pay conceptual debt up front. Make sure your business models exist completely and separately than any technical implementation. Exercise in detail the model and order of actions independently of of the software process. This boils down to knowing your business process.
    • Solution - It is also just fine, if budget permits, to allow the developers to help pay conceptual debt with conceptual software, often referred to as proof of concepts or POC. This should never be confused with releasable hardened applications ready for production. The purpose of a  POC is to prove that something works and or discover more decision points that will increase the final product's value. It's a natural mindset for the business to think of their abstract idea from two minutes ago as a shiny, polished, and perfectly shaped widget. Developers, corrupted by facts, technology, and acronyms, hear the idea as 'Give us a widget of undefined shape and size (perfect = undefined), but that is in fact shiny, that can fly in 'any environment' including water, but mainly space'.
  • Cause - Developers or (and) Business do not understand the concept of 'Iteration'. Iteration is like an incremental end state. It serves many purposes including managing expectations, defining a small amount of work, prioritization and delivery encapsulation of features, and it generally establishes some important event points of the software life cycle.
    • Solution - Learn what an iteration is and how to define one that both devs and businesses can agree on. Hint, it's all about scope. Scrum sets you up with a one size fits all approach and if you are bad at iterating, it can be useful.
• Problem - Estimates are always wrong.
  • Cause - You can't estimate what you don't know.
    • Solution - Don't estimate before you design. This is controversial and certainly poses a conundrum but I'll totally defend it. Pick your pattern. Choose your tools and libraries. Model the data. Make some prototypes or proof of concepts. You can even time box the design if you have it clearly scoped. If this sounds like waterfall than immediately reduce scope.
  • Cause - Developers and project managers tend to be optimistic and want to put their best case scenario forward.
    • Solution - Compare the work to similar projects that are finished and extrapolate from there. I concede that some evergreen projects can't be compared.
  • Cause - You asked for an estimate.
    • Solution - Don't estimate. Kanban wins hands down on this factor.  If you believe as I do that estimates are wildly inaccurate under the very best conditions, then it stands to reason that you should not base your entire software life cycle around it. There are other ways to predict software development, like a complete understanding of the scope of work and evaluation of work finished. Think of the way they use analytics to predict business decisions instead of a team members best guess. Nobody asks factory workers how long it takes to make 200 widgets, they time how long it takes to put their portion of the widget together until they have some reality working in their favor.
• Problem - Our iteration went fast and then slowed way down.
  • Cause - Tech debt was accrued either knowingly or unknowingly in order to deliver previous sprints or cycles quickly.
    • Solution - Long term speed requires refactors and code quality. Code quality and maintainability is rarely part of done criteria and often developers fail to properly advocate for it. Do so early and often or else trust will be used up by the time everybody figures out there is a ball of yarn that needs to be unwound.
• Problem - We adhere to our process but still don't seem to be getting ahead or going in a smart direction.
  • Cause - A process doesn't have a brain, an opinion about talent vs incompetence, or have any way to ensure wise decisions vs bad decisions. There is absolutely no substitute for talent, vision, or a deep understanding of the problem domain.
    • Solution - Good old fashioned study and understanding of the problem, as well as relentlessly describing, explaining, and conveying the problem to those designing solutions. I can't count the number of times I have started coding against a problem that seemed to have little to zero connection to my understanding of the company's larger goals. In fact many times I knew that my code assignment was about solving an edge case failure, or the fact that some internal big wig 'noticed' something, rather than meeting the stated long term goals of my organization.

Last random thought. Don't be religious about it. It's just a series of communication events that can assist your actual discipline in a team environment.

Summary

I'm still learning about what makes great software but would give process preference to Kanban at this moment. Even though it's questionably agile, having been around for a lot longer, its process can serve a wider set of practices if Scrum doesn't meet your goals. Another important consideration is that over analyzing the process could be a trap when truly mastering the business and problem domain is what will bring the most rewards.

Skip this entire article and just watch this :)

Resources

http://mikehadlow.blogspot.com/2014/06/heisenberg-developers.html

http://www.jamesshore.com/Blog/The-Decline-and-Fall-of-Agile.html

https://lostechies.com/jimmybogard/2012/09/12/why-im-done-with-scrum/

http://www.agileadvice.com/2011/12/05/referenceinformation/24-common-scrum-pitfalls-summarized/

https://www.atlassian.com/agile/kanban

http://blogs.msdn.com/b/cdndevs/archive/2015/10/15/the-path-to-no-estimates.aspx

http://blog.jbrains.ca/permalink/your-daily-scrum-is-killing-your-team

https://www.targetprocess.com/blog/2014/07/why-agile-estimates-dont-work-part-1

https://www.targetprocess.com/blog/2011/04/5-reasons-why-you-should-stop-estimating-user-stories/

http://serialseb.com/blog/2016/02/25/agile-is-dead-intro-podcast/

http://techbeacon.com/uncle-bob-martin-agile-manifesto-15-years-later

http://geekswithblogs.net/Optikal/archive/2014/01/14/155191.aspx

https://michaelochurch.wordpress.com/2015/06/06/why-agile-and-especially-scrum-are-terrible/

https://www.mountaingoatsoftware.com/blog/my-primary-criticism-of-scrum

http://steve-yegge.blogspot.com/2006/09/good-agile-bad-agile_27.html

http://www.agileoverflow.com/t/why-do-some-developers-at-strong-companies-like-google-consider-agile-development-to-be-nonsense/107

Problems arise when companies take the project management stuff and do not take the technical stuff

https://www.aaron-gray.com/a-criticism-of-scrum/