Proposal to Adopt Agile Development Methods
Chapman University College
Proposal to Adopt Agile Development Methods
Welcome to the Twenty First Century. We have amazing technology, a complex ecosystem of global industry, and a systems-thinking learning organization. However, we are using antiquated software development methods. Our home-grown methodology based upon a manufacturing model of large up-front detail designs, construction, then verification, and finally production is a phased or largely sequential software development process. This process does not work well for software - a very malleable product. Our project success rate of 32% is slightly better than the industry average of 28% (small companies) reported in the CHAOS Report (Johnson, 1995). However, if we continue to fail two-thirds of the time, we may find ourselves at the back of the pack, and possibly in the bankruptcy courts.
Agile software development is a family of methodologies based on iterative and incremental adaptive development with lightweight process controls featuring highly collaborative environments with disciplined technical practices. It focuses on delivering frequently and embracing changing requirements. Many more companies are adopting Agile software development practices. The trend is no longer a fad. The results of Agile Transformation are clearly higher productivity, greater product quality, greater business customer satisfaction, with equal or lower cost of development (Appendix A) (Ambler, 2008). The success rates of Agile teams exceed 80% (Ambler, 2008). This is a drastic change from one in three successful projects to four out of five project successes.
An Agile Culture
An Agile organization, at the company level or at the project team level would exhibit the values of the Agile Manifesto:
Manifesto for Agile Software Development
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more. (Beck et al., 2001/2001)
The 12 principles of the Agile Manifesto (Appendix B) could be used to assess the organization as it transitions into a more Agile organization.
Business cultural changes are notoriously difficult. It is one thing to pronounce that the company will adopt an Agile method or even that we wish to have an Agile philosophy; it will however be a completely different thing to manage a longterm transition toward those goals. One lens through which we may view radical changes of this nature is the Bridges Transition model. This model reminds us that changes may be as simple as announcing the goal. The transition, however, is more complex. The transition is the path of growth that will lead to the goal, “composed of (1) and ending, (2) a neutral zone, and (3) a new beginning” (Bridges, 2004, p. 4). Viewed through the lens of the Transition model the first phase is the ending. That would be the point at which the organization decided to change to an Agile organization, and announce the vision. To end the waterfall of phased development. The neutral zone is the gap between our old well known process and the new beginning when our new Agile processes are well known. A feeling that for a period of time the company may be lost, and misdirected, a feeling that may be greatly amplified as we learn and grow into this new organization and become the new company. Bridges (2004) notes that the neutral zone is a time of learning. The new beginning, is marked by the point at which we naturally start to think of ourselves as an Agile organization, not questioning if we are, but knowing that we are Agile. The beginning will mark new and unknown opportunities for the organization. The ability to quickly recognize market shifts and respond to changing requirements by delivering high quality applications and software services that meet customer’s current needs. Along with an ability to evolve our solutions as new needs arise. Responding to valuable feedback from customers and incorporating that into new features while rapidly delivering those solutions will make our company successful.
Behaviors of an Agile Organization
Delivering frequent customer value is the hallmark of an Agile company. To continually provide working software to our customers we must have processes that allow for last minute changes that add value. Development processes that embrace change, rather than resist change, are goals of an Agile organization; for life is the act of constant change.
Agile development practice allow for evolving designs rather than planning for static designs implemented in construction phases. Spending less time in up-front requirements gathering and specification elaboration phases requires trust. Trust that the team can and will turn a requirement, even poorly specified, into actual working software. Trust that the business will manage the resources well. Build the most valuable assets early, not wasting energy on bells and whistles that customers do not desire. Our development teams must collaborate with the business groups on a daily basis throughout the design. This will require commitments on both sides, as detailed plans will not be developed; therefore constant input and feedback must be part of the process. Mistakes must be allowed to happen, even encouraged, for it is in errors that we learn the most. Risk of large mistakes will be mitigated with short duration iterations (time-box) of design and development. The result of each iteration will be a functional product, demonstrated to the project stakeholders for evaluation and feedback. Adjustments will become the norm; adjustments in project scope, features, release plans, and timeframes. These adjustment may be as large as the cancelation of a project; but when successful these adjustments will be the minor corrections that one makes to navigate the highway while driving a car.
Leadership in an Agile Organization
Agile development requires that the management assemble a group, give the group some clear goals, support the group, negotiate with the group on all aspects of constraints, and then leave the group to self-organize, and allow the power of motivated individuals joining with other individuals into a team resolving and solving the problems required to achieve the goal. In these work groups leaders will emerge. Given clear concise goals the group will form into a team. A typical solution for team leadership and Agile process control is the Scrum framework. Scrum is a lightweight process framework, based in empirical process control theory, which uses an iterative and incremental approach to “optimize predictability and control risk” (Schwaber, 2009, p. 1). The Scrum framework defines only three roles for the process, a team of developers responsible for the product, a Product Owner ultimately responsible for the project return on investment (ROI), and a Scrum Master responsible for the process. Scrum is one of many Agile processes. It is recommended as the cornerstone of our Agile transformation because it is widely known, well defined, simple, and easy to learn. Although it may be considered very hard to master; like riding a wild mustang on the range, it requires constant attention to the nuances of the team, processes, and product. A Scrum Master may play a leadership role for the team, however there is no direct authority over the team members except for process authority. The Scrum Master will be a facilitator for the Scrum process teaching other team members, the Product Owner, and stakeholders their responsibilities and roles. The Scrum Master does not lead the team, they allow the team to develop its own leadership. Emergent leaders in self-directed teams have greater emotional connection with the group, and exhibit empathy for the context surrounding the group. This context allows emergent leaders greater effectiveness in directing group performance (Cohen & Bailey, 1997).
Scrum can be successfully applied outside of a software or product development environment. We may find upon a successful transition of the development practices that Scrum is a valuable tool in higher levels of our organization. Aspects of empirical process control with its emphasis on inspecting actual progress and adapting the next iteration to be incrementally better is inherent in organizational development methods also. Opportunities await us along the path to Agile Transformation.
The Feel of an Agile Organization
Our current perception of a world where technologies change so fast as to leave us all behind will change when we can keep pace with the industry. When we have made significant progress toward our goal of becoming an Agile software development group we should be capable of taking rapid advantage of emerging technologies. The strategic vision of transitioning our legacy products to a Service Oriented Architecture (SOA) will benefit from the ability to adapt to rapidly evolving standards in the new paradigm. The development teams will be capable of using new Web 2.0 techniques that allow our User Experience designers more freedom in user interface interaction delivering a richer experience to our customers.
Product design will become a richer collaborative process, using the broad skills of many team members across our diverse organization. A more cohesive, inclusive, and participatory product development environment should result in higher quality and insanely great products.
One principle of Agile is the sustainable pace. We need to maintain a development pace that does not require late-nights, weekends, and a “death march” (Brooks Jr, 1995) to production. With the addition of Agile skill sets from the Extreme Programing (XP) methods we expect developer job satisfaction to increase. The XP practices of Continuous Integration (CI), Test-Driven Development (TDD), and Pair Programming (PP) will change the nature of the daily work of the software development group. This will require training, coaching, and experience for our people. The physical space will also change. Working alone in a cubicle from specification documents written 16 months prior is a practice of past eras. A project team room that concentrates the software development group along with the product development group will increase just-in-time communication. Teamwork will build tacit knowledge of who knowns what details and problem solving skills. We will use techniques such as pair programming to build knowledge, skills, and abilities (KSA) within our Scrum teams. The use of TDD will create a suite of tests (a safety net) that provides value to the development team (including the embedded quality assurance specialist). Tests will prove that the product functions as desired. The CI system and the automated build system will allow push-button deployment of our products to System Test Servers. Allowing frequent (almost anytime) deployment of our SOA products allowing the Product Owner to quicken the return on investment (ROI).
Dr. Dobb’s Journal article reporting on Scott Ambler’s Agile adoption and success rate survey (Feb. 2008) concluded: “In short, becoming more agile appears to be low-risk decision for senior IT management” (Ambler, 2008).
The Value Bottom Line
The value of a transition from our waterfall process to an Agile process will deliver both intrinsic values and extrinsic benefits. Our recent effort in Triple Bottom Line (TBL) accounting, summarized by SustainAbility’s “people, planet, profit” phrase, will resonate with Agile’s people-centric view of our human capital. A list of Agile’s intrinsic values are:
•Employee job satisfaction (motivation, sustainable pace, enjoyment)
•Humanistic values (trust in people and collaboration)
•Maximize project success rates (78% overall success rate; 83% co-located)
•Quicker time to market (releasable “barely sufficient” products)
•Reduction in the Cost of Change (less requirement defects, less bug fixes)
•Quicker time to project ROI (via more frequent release to production)
•Greater Productivity (82% of respondents perceived higher productivity)
•Higher Quality (77% of respondents perceived higher quality)
•Higher Stakeholder Satisfaction (78% of respondents perceived higher satisfaction)
•Lower Development Cost (37% of respondents perceived lower cost)
(Ambler, 2008, statistics from survey).
The extrinsic benefits will be realized as we transition to an Agile organization, these may include: customer loyalty due to great products, ease of recruiting due to improved work environment, status in the industry and the Agile community, etc.
The Agile transformation will require time and patience, as well as skilled coaching. It is expected that results in team work environments and project quality will be the first benefits to be realized. New Scrum teams learn quickly but must be given the opportunity to make and correct their own mistakes along the Agile path. At Yahoo Sutherland (2009) reports using a technique to jump start Scrum teams when “properly coached delivered 300-400% improvements” (p. 1). “The best Scrum teams in the world average 750% gains over the velocity of waterfall teams with much higher quality, customer satisfaction, and developer experience” (Sutherland, 2009, p. 1). A team at Jayway in Sweden “doing two week sprints achieved 375% of initial velocity in six sprints” (Sutherland, 2009, p. 4). Other benefits of Agile adoption will take longer to materialize, and largely depend upon the attitude of the company at large. “Scrum was institutionalized at Systematic over a period of approximately six months” (Jakobsen & Sutherland, 2009, p. 1). Systematic was found to be CMMI level-5 compliant in 2005, and adopted Scrum in 2006 (Jakobsen & Sutherland, 2009, p. 1). Many of the corporate level benefits are only seen in the later stages of complete transformation to an Agile corporation.
Consequences of Inaction
The status quo of failing projects after months and even years of development are in store for the long run, unless, we act to change. Some project are bound to fail, we take risk and judge the rewards worth the cost. However, even the Agile concept of a success when we fail fast, saving the cost of further investment in a poor idea, is a benefit that we do not currently have with our software development lifecycle. We must invest upwards of 80% of the total project cost before we can make go no-go decision based upon functional working software.
Training is the first step after a decision by corporate management. We must all speak a common language regarding the Agile philosophy and lexicon. Scrum training is the easiest Agile training obtainable and is offered at reasonable cost for ad hoc attendance of two day seminars, or a Certified Scrum Trainer may be engaged to come on site. This would be considered initial training for key personnel and valuable for further organizational change development planning stages. Educating ourselves on the Agile values and methods and then developing our organizational change action plans within a Scrum model of inspecting and adapting will be a great experience for our management team. A further step will be to hire an Agile coach, a person to guide the organization, to mentor future Scrum Masters, Product Owners as well as Team Members. This person will be tasked with oversight of the change initiative, and report to the VP of Product Development.
The Map is Not the Terrain
In the Robert DeNiro film “Ronin”, DeNiro's character Sam expresses the need to actually see the place, to physically be in the space: “The map, the map, the map. The map is not the terrain.” An action plan is not the Agile path we will traverse. Dwight D. Eisenhower said, “The plan is useless; it’s the planning that’s important.”
Kurt Lewin’s Action Research model of organizational development and change will be to overarching approach. It meshes well with the Scrum framework. Action Research is a cyclic process model facilitated by an Organizational Development (OD) Practitioner, which relies heavily upon the collaboration of a client group (transformation team) for exploration of data, problem space diagnosis, action planning, implementation, evaluation of interventions, and further cycles of planning, actions, and post action assessment (Jackson, 2006).
In the cyclic process of Action Research we have reached step (3) preliminary diagnosis. A problem has been perceived (step 1), and collaboration with an OD practitioner (step 2) has commenced. The preliminary diagnosis (step 3) is to institute an Agile transformation of the software development group. With a possible larger transformation of the company as an organization in the future. The future steps are: (4) gather data from the transformation team, (5) data feedback to the team, (6) team exploration of data and problem domain, (7) team action planning, (8) team takes action, (9) post action evaluation and data gathering, (10) new action planning by team, (11) new action taken by team, (12) post action assessment, then a repeating of the cycle (Jackson, 2006, p. 140). However, let us take first steps first. The Scrum training of key personnel would be the preliminary recommendation of an action. The transformation team and executive management should take this proposal under consideration and decide if this is the proper course of action. In other words, will they commit the company to this Agile course of action? Knowing full well that our plans may need to be adjusted, and being responsible for evaluation of the action, and any future adjustments necessary.
Resonance with Organization Development Theories
The cultural change that an Agile transformation requires may be analyzed via Force Field Analysis. “In planning for change, it is advantageous to identify those conditions that will help or hinder the change initiative” (Jackson, 2006, p. 145). The transformation team may use this tool to identify the driving and restraining forces for the Agile transformation initiative. After identifying all forces both pro and con the forces are assigned relative weights (or points). The weights allow some subjective measure of the effort the force applies to the current situation (status quo). Analyzing these forces the team will try to strengthening the driving forces while weakening the restraining forces, and possibly devising strategies to flip a force from restraining to driving. This will allow the team to identify impediments to the initiative and seek management support in removing the impediments (a role of the Scrum Master).
Action Research Becomes Scrum
The team members trained in Scrum will recognize may of the aspects of Organizational Development methods such as Action Research. Scrum and many of the Agile processes draw on years of research and studies of team dynamics, process workflow, continuous improvement, systems-thinking, and organizational learning (Sutherland, 2004). Scrum uses very small iterations and feedback loops on many levels to plan work over a short duration, inspect the progress of the work done, to expressly define the standards of completed work, and to reflect upon the work, processes, and techniques to constantly improve and move forward.
Methods to Facilitate Transition
Many Agile development groups have embraced more humanistic values for work, attempting to capitalize on knowledge workers innate desire to enjoy their work and their internal motivation. Herzberg’s Two-factor theory is applicable to this situation. Also referred to as the Motivation-hygiene theory, it describes the factors that lead to worker satisfaction (motivators) and factors that lead to dissatisfaction (hygiene) if not attended to well. Herzberg concluded that the satisfaction scale was not a single continuum. The factors that lead to dissatisfaction (work conditions, salary, relationships, supervision) when poorly managed, did not lead to satisfaction when expertly managed. There are very different factors that lead to satisfaction. These motivating factors are: growth, advancement, responsibility, recognition, achievement, and work itself (Robbins & Judge, 2009). Agile methods support these motivational factors in the work place. Additionally many Agile teams use Appreciative Inquiry during retrospectives. Appreciative Inquiry “seeks to identify the unique qualities and special strengths of an organization, which can then be built on to improve performance. That is, it focuses on an organization’s successes rather than on its problems” (Robbins & Judge, 2009, p. 632). Scrum requires a team retrospective each sprint (iteration), this is a process inspection and a chance for adaption. Using Appreciative Inquiry builds on teams ability to learn and improve, recognizing that perfect execution may be a goal, but rarely attainable on the first several attempts.
Organizational Change was studied by Kotter, who developed an eight step change model largely from the failures of companies engaged in change initiatives. “A few of these corporate change efforts have been very successful. A few have been utter failures. Most fall somewhere in between, with a distinct tilt toward the lower end of the scale” (Kotter, 2007, p. 96). Kotter’s recommendation are: “(1) Establish a sense of urgency; (2) Create the guiding coalition; (3) Develop a vision and strategy; (4) Communicate the change vision; (5) Empower broad-based action; (6) Create short-term wins; (7) Consolidate gains and produce more change; (8) Anchor new approaches in the culture - make it stick” (Jackson, 2006, pp. 166-167). Kotter (2007) warns that skipping steps or not given proper attention to a step is the typically mistake made by failing corporations. There are valuable lessons to learn from these mistakes.
Recognizing Success - Metrics
Scrum uses very simple tools for measuring iteration success. These metrics are the Scrum task board with task to be done, the in-process task, and the done task. The task board will be the center of activity during stand-up meetings each morning. A quick look at the board allows everyone to see progress being made and the work remaining during the sprint. It is the Scrum team’s job to manage this and radiate the information to all. Another information radiator is the release burn-down chart showing the completed work and remaining work required for a product release. The physical changes in Agile adoptions will be easy to recognize. These will be groups of strangers forming teams, teamwork and group discussions, “big visible charts” used as information radiators (burn-down chart), stand-up meetings, product demonstration, application test suite pass/fail monitors, project build charts, broken build email notifications, the list goes on and on. One specific example; on some teams the engineers wire-in a flashing red light to the build machine, if a build fails the flashing light alerts everyone in the room. Quick action by the developer that broke the build by committing the defective code is the result. This “stop the line” behavior will become obvious to management that takes the time to understand the Agile process.
Agility of our development organization may be measured by several instruments. Cautions using these instruments should be used as they are not as well studied as may psychometric test such as IQ test or Myers-Briggs Type Indicator for example. The simple 10 question Nokia Test is one example. “The Nokia test is a similar to a maintenance check on your car. It looks at whether your tires have air, your tank has gas, all cylinders are firing, and makes sure there are no critical missing pieces to your car. You should perform it before you go out for a drive with your Scrum team” (Sutherland, 2009). A more comprehensive measure that allows organizations to perform longitudinal studies of improvement in Agility is the Comparative Agility™ survey by Cohn and Rubin (2008). This survey is available free of charge on the web (http://www.comparativeagility.com/). However, it has not been studied for reliability and validity. Of course, the Action Research transformation team will evaluate every intervention, and use any instrument that is deemed appropriate.
More subtle successes will be rates of project success, measured over much longer periods of time. Greater return on investment for projects, as well as quicker cost recovery and lower investment cost. These will come after Agile process are in place and the corporation begins to take advantage of Agile’s many secondary benefits. The measures will be typical business metrics already in place at our company.
Agile transformations do not happen overnight. They are not without trials and tribulations. In a systems-thinking view of our organization a change in the development process will ripple outward effecting other departments. Scrum does not offer to fix our problems, it is no silver-bullet. A core value of Scrum is transparency. “Transparency ensures that aspects of the process that affect the outcome must be visible to those managing the outcomes” (Schwaber, 2009, p. 1). We will not be able to solve problems of which we do not have knowledge. We must trust others motivates when they raise issues and collaborate on solutions. Our Agile coaches will have faced many of the issues our transformation will conjure up, some may be unique to our organization. An Agile philosophy is that making decisions and then monitoring the outcome, even if a mistake, is much better than postponing the decision past the last responsible moment.
Many Agile adoptions start with pilot projects and with tentative steps toward half measures. These attempts have been labeled “Scrumbut” processes. Agile practices work as an orchestra producing a symphony, to remove one section of the orchestra will mute a portion of the symphony. The expected results will not be achieved with half measures.
Cost for Certified Scrum Master training is $18,000 (plus expenses) for up to 20 participants on site. Agile coaches salaries are approximately $130,000. We may lose some employees who do not wish to transition to Agile work methods. There are also indirect cost associated with every change. Infrastructure cost will be incurred to retrofit our software development work bay of cubicles into team rooms with modern pairing stations and a more flexible organic work space. Details will be studied by the transformation team and evaluated when the time comes for those decision.
Change is the nature of our business. If we do not change our business methods and adjust then change will be force upon us. Adopting an Agile stance will allow us to change with the industry, to seize opportunities before challengers, and to outperform competitors. It will provide for a people-centric development practice, that is at our companies core beliefs. Many other companies have seen higher productivity, higher quality, greater job-satisfaction and overall cost reduced using Agile methods.
The first step towards this cultural transformation is for the Executive Management Team to approve this proposal, to join the Agile community, to communicate the new vision, and begin the transformation into an Agile organization.
Appendix A. Agile Adoptions Rate Survey Results
Agile Adoption Rate Survey Results (February 2008) published in “Has Agile Peaked?” Dr. Dobb’s Journal (June 2008) by Scott Ambler: (http://www.ambysoft.com/surveys/agileFebruary2008.html used with permission).
Appendix B. Principles of Agile Manifesto
•Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
•Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
•Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
•Business people and developers must work together daily throughout the project.
•Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
•The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
•Working software is the primary measure of progress.
•Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
•Continuous attention to technical excellence and good design enhances agility.
•Simplicity--the art of maximizing the amount of work not done--is essential.
•The best architectures, requirements, and designs emerge from self-organizing teams.
•At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
(Beck et al., 2001/2001)
Appendix C. Definition of Terms
Agile software development - A family of methodologies based on iterative and incremental adaptive development with lightweight process controls featuring highly collaborative environments with disciplined technical practices focusing on delivering frequently and embracing changing requirements.
Burn-down chart - a information radiator depicting the progress made toward a goal.
CMMI - Carnegie Mellon, Software Engineering Institute’s Capability Maturity Model Integration - a measure of process maturity (5 being the highest level).
Lightweight process - generalization of Agile processes; relative to highly formal processes or heavyweight processes.
Methodology - “a system of methods used in a particular area of study or activity” (Jewell, 2001).
Product Owner - a role on the Scrum team responsible for prioritizing the work to be done, to optimize the return on investment.
Scrum - a lightweight process framework, based in empirical process control theory, which uses an iterative and incremental approach to “optimize predictability and control risk” (Schwaber, 2009, p. 1).
Scrum Master - a role on the Scrum team responsible for optimize the process and productivity.
Scrum meeting - a short focused daily plaining meeting for the team, also called the standup meeting (because no one is allowed to sit).
Sprint - a time-boxed iterations of product development (typically one month or less).
Time-box - a fixed time duration in which a task is performed.
Velocity - the amount of work a Scrum team performs in one sprint (may be measured in unit-less value of story points).
Waterfall - a generic term applied to heavy-weight formal sequential phased development methods.
Ambler, S. W. (2008). Has agile peaked. Dr. Dobbs Journal: The World of Software Development, 3(6), 52-54.
Beck, Beedle, Bennekum, Cockburn, Cunningham, Fowler, et al. (2001). Manifesto for agile software development. Retrieved november [Web page]. (Original work published 2001)
Bridges, W. 1. (2004). Transitions : Making sense of life's changes . Cambridge, MA : Da Capo Press.
Brooks Jr, F. P. (1995). The mythical man-month (anniversary ed.). Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA.
Cohen, S. G., & Bailey, D. E. (1997). What makes teams work: Group effectiveness research from the shop floor to the executive suite. Journal of Management, 23(3), 239.
Cohn, & Rubin (2008). Comparative agility survey. [Web page]
Jackson, J. C. (2006). Organization development : The human and social dynamics of organizational change (illustrated ed.). Lanham, MD : University Press of America.
Jakobsen, C., & Sutherland, J. (2009). Scrum and cmmi--going from good to great: Are you ready-ready to be done-done?. Agile 2009.
Jewell (2001). The new oxford american dictionary (illustrated ed.). Oxford University Press.
Johnson (1995). The CHAOS report (1994). Standish Group. Retrieved September 24, 2009, from http://www.standishgroup.com/sample_research/chaos_1994_4.php
Kotter, J. (2007). Leading change: Why transformation efforts fail. Harv Bus Rev, 96-127.
Robbins, & Judge (2009). Organizational behavior . Upper Saddle River, N.J. : Pearson Prentice Hall.
Schwaber (2009). Scrum guide. [Pamphlet]
Sutherland (2009, August 25). Nokia test: Where did it come from? Scrum log (Jeff Sutherland's Blog) [Web page].
Sutherland, J. (2004). Agile development: Lessons learned from the first scrum. Cutter Agile Project Management Advisory Service, 5(20).
Proposal to Adopt Agile Development Methods
Wednesday, October 14, 2009