Coaching in the Agile Context
David Koontz
Coaching increases an organization’s ability to transform in any circumstance. Modern organizations are turning to coaching as an effective means of learning and developing their key resources, people. As our industries move into the information age, knowledge creation becomes a key competitive advantage. Coaching facilitates knowledge creation and organization adaption to new capabilities. The software industry is undergoing a transformation to new methodologies and many of these newer lightweight processes fall under the Agile umbrella, or family of processes. Agile software development methodologist recognize the need for training and development in these new techniques. Coaching at many levels in the organization is paramount for successful Agile transformation to occur. Organization transformation have been studied and demonstrate a life-cycle of phases. Kotter’s model of organizational transformation and Agile transformation are overlaid in this paper with an emphasis on coaching. Coaching at all levels in the organization is most advantageous to the software development group’s transformation into a learning organization.
Coaching in the Modern Organization
“Coaching is the process of equipping people with the tools, knowledge, and opportunities they need to develop themselves and become more effective” (Peterson & Hicks, 1996, p 14). Seen as a process, coaching is not an occasional conversation, or a single event, it is a continuum of events and conversations that guide people to learn how to perceive their current reality and the tools, resources, opportunities they have to produce a desired outcome in a future reality. The tools a coach engages are the tools of inquiry, reflection, analysis, and imagination (Peterson & Hicks, 1996).
Coaching in the modern organization is a hot topic. Many of our leading companies are training managers by the thousands to become coaches (Hargrove, 1995). This proliferation of coaching is a means to an end. The goal of organizations training coaches and supporting coaching throughout the ranks is to achieve success for the company. Leaders recognize the imperative of creating new knowledge in the Twenty First Century information age. Coaching fulfills an important development force in the knowledge worker’s environment. A coaches sphere of influence is a just-in-time method of development training.
Hargrove (1995) describes three rites of passage that businesses have experienced in the evolution of companies and our market places. The change from crafts to high-volume production lines around the turn of the Twentieth Century is the first passage; standardization. Production lines required standardization to achieve high-volume throughput of parts assembled into a whole product. In this era the assembly line worker was seen as just another replaceable part in the machine. The second rite of passage is the quality era, lead by W. E. Deming in the mid Twentieth Century. In this era the company had to produce high quality goods as well as high volume. Deming’s techniques for achieving consistent quality resulted in social changes in the workplace. These quality techniques empowered the line workers. The workers became more that just a means of production, they became an integral part of the quality of production. In Hargrove’s third rite of passage the company must concern itself with innovation and creating new products. This creative process requires knowledge, complex problem solving skills, and collaboration. Hargrove (1995) sees the successful companies in this new era with “a special kind of management culture, one that is based on creating new knowledge” (p 6). Further the “crucial player in this new management culture is the transformational coach” (p 6). There is no doubt that a software development company is operating within this third era.
An innovative company traditionally contains a research and development department to create products giving the company a competitive advantage. In the knowledge era the company must also maintain competitive advantage with its human capital. Coaching and development are “integral to leveraging an organization’s strategies and core competencies” (Peterson & Hicks, 1996, p 9). This leader “translate[s] the rhetoric of ‘people are our most important asset’ into real investment in people's growth” (Peterson & Hicks, 1996, p 9). Peterson and Hicks argue that every leader in the organization must close the gap in perceptions of the lip service given to human development, if the goals are to become reality. Their argument is that; “change is inevitable,” “people must learn and adapt quickly,” “employees want to grow,” and “people are the real source of competitive advantage” (Peterson & Hicks, 1996, p 11-12).
Coaching others is a discipline that can be enormously powerful in personal satisfaction and pride for the coach. However, there are other benefits for the coach. Personal motivations are important to recognize, and lead to self-sustaining models with virtuous cycles. The personal payback for the coach are stronger teams with highly capable people. Dedication and excitement are beneficial byproducts of a person growing in their role. Talented people seek growth opportunities. A reputation as a leader that helps people grow and learn creates an influx of talent. As talented people influenced by the coach move on to new opportunities the coach’s network expands. Cultivation of this social network, of people with uniquely skilled competencies has value for the coach and the business. This social network creates an exponential growth in knowledge that may be tapped by the coach. The coach does not need to personally know every problem domain to be effective within the solution space. One secret of effective and efficient coaches is, only a small amount of energy is required if focused on the highest priority opportunities. This creates extreme leverage for the coach and the company (Peterson & Hicks, 1996).
Peterson and Hicks (1996) give many strategies to target the common barriers to development. The basis is trust. Partnerships are founded on trust, and coaching is a partnership. All parties must understand the motives of the others. Understand that the coach also has something to gain from the partnership. The coaches motives should be made visible to the team. Inspiring commitment to self development becomes easier when people understand how the development will personally effect them. Developing new skills requires practice. Coaching helps grow new competencies by disciplined consistent practice. Measuring progress is a vital aspect of growth. Recognizing barriers to learning and suggesting methods to overcome these barriers, and teaching the skills for future meta-cognition is coaching at its best. Observing misalignments of personal and organizational goals and making these observations visible to all parties is a key competency. Finally coaches are concerned with the team environment. Shaping the environment to encourage and reward new behaviors makes change regenerative.
Coaching the Agile Transformation
Augustine (2005) describes the Agile manager as a change agent who aspires to transformational leadership. The Agile manager believes in the people with whom they work, such that delegation and shared responsibility is a natural outcome of the relationship. They maintain high moral and ethical standards. Lifelong learning is the Agile manager’s way of dealing with the constant change, complexity and ambiguity in the industry. They are experimenters, exploring and analyzing the effects of actions. Agile managers are visionaries, capable of articulating a shared vision, and influencing others.
The Agile software development movement is redefining the methodology for producing software. Agile development is a systems approach of managing a creative sometimes chaotic process. Agile transitions require time and energy, and a shepherding of the transformation. With any organizational transition there is risk of never achieving the goal. Coaching significantly increases success rates of Agile transformations. Agile transformation is an organizational change from command-and-control, highly rigorous processes with detailed plans and up-front designs to methods of collaborative work environments, lightweight processes, evolving architectures and designs intended to embrace changing requirements.
The Agile Context
The Standish Group’s (1995) famous Chaos Report noted that only 9% of software development projects were successful in large companies, with slightly higher rates for medium and small companies (p 3). In this era of software development companies were using highly formal linear design then construct methods. The foundations of the Agile software movement are the various “lightweight” processes of the 1990s. This new development process, resulted from an era of failed “heavyweight” software projects utilizing process methodologies that required large investment of time and energy in planning, design, and documentation of the project goals as well as the process steps. The new alternative was a response by some in the industry to correct systemic problems. Scrum, Extreme Programming (XP), Feature-Driven Development (FDD), Pragmatic Programming, Dynamic Systems Development Method (DSDM), Adaptive Software Development, Crystal Clear, and others were the processes that grew out of the stressful failures of the large number of failed software projects of the 1980s and 1990s. In early 2001, seventeen leaders of these various lightweight methods came together to define their common characteristic and values (Fowler, Highsmith, 2001). What resulted from this meeting was the Agile Alliance and their manifesto:
Manifesto for Agile Software Development
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
(Fowler, et al., 2001, p 35)
The alliance of methodologist use these core values and principles to redefine the software development industry. Coaching is an organizational process that will support these values.
Coaching the Agile Development Team
Coaching is recognized as a key role within the development group. Extreme Programing (XP) identified a specific role on the development team, the XP Coach. Among other duties, the XP coach “helps the individuals with less experience find directions to grow in that are relevant to the business and the project unfolding” (Cunningham, 2005, p 1). Extreme programming utilizes specialized skills such as; pair-programming, consensus decision making, test-driven development and continuous integration. Developers new to Agile practices will need to learn these XP skills. These XP skills rely on well developed interpersonal skills that many software development personnel will need to develop. Development of team building and group dynamics, as well as training in the technical competencies require coaching and development effort.
The Scrum methodology does not define a formal coach role. However the role of the Scrum Master is to ensure the Scrum process is followed and the various team members are functioning to the best of their abilities. Ken Schwaber described one aspect of the Scrum Master’s role: “Improve the lives of the development team by facilitating creativity and empowerment” (Schwaber, 2004, p 7). Much like a coach the Scrum Master has no authority or hierarchical control over the team. A Scrum Master uses their influence, knowledge of the Scrum process, the general knowledge of the software development best practices, and the systems thinking embodied in the Agile values to guide the team. The Scrum Master to Scrum team relationship is based upon trust, respect, a commitment to learning and courage for inquiry that manifest in well functioning Scrum retrospective meetings each sprint (development iteration).
In Yahoo’s Agile transformation, the role of the coaching group was recognized, as playing a vital role in the change process (Benefield, 2008). Yahoo describes its transformation as a grassroots effort; although the Agile coaches had executive support and VP level budgetary oversight. In hindsight Benefield saw the benefit of concurrently introducing XP engineering practices such as pair-programming and test-driven development at the team level, while engaging the Scrum process, which is focus at the team organizational level (Benefield, 2008). A large challenge for Yahoo was growing the coaching team at the acceleration rate of teams wishing to adopt Scrum. The demand for training and coaching far exceeded the supply. A significant portion of the coaching team’s effort was diverted into justification of their own existence, and collections of metrics to justify the transformation. Both management and teams desired proof of success before they were willing to commit. Therefore a large amount of coaching energy went into measurements and surveys of the transformation to Agile. This behavior is indicative of lacking vision. The vision of Agile transformation was not urgent and shared by all the organization. Groups of people needed proof to buy-in to the change process. With extra effort the small coaching team was able to overcome these disfunction and grow the adoption from a few teams to over 150 Agile teams with 81% of the individuals happy with the Scrum process (Benefield, 2008, p 272).
Coaching through the Eight Steps to Transformation
In John Kotter’s paper “Leading Change - Why Transformation Efforts Fail,” he describes the lessons learned from observing hundreds of companies in transformation processes. From these successes and failures he draws the conclusions that there are a series of phases that require considerable time for the company to progress through. Short cutting the phases (or steps) harms the overall process. A critical mistake in any of the phases has far reaching negative affects reducing the chance for the long-term successful transformation. One factor that pressures businesses to short-cut phases is the lack of capable people experience in transformations (Kotter, 2007).
Many software development organization are adopting Agile processes to improve their companies development groups. To be successful the development group must negotiate a transformation, from existing practices typically rooted in “waterfall” process mentality to Agile practices and mentality. Modeling Agile transformations using Kotter’s eight steps for transformation gives insights into how experienced Agile coaches improve the chances for success. For the purposes of this paper we will use the Agile process of Scrum as a management framework with XP engineering practices for developers, a typical conglomeration of process methods.
Kotter’s primary step in transforming the organization is to “establish a sense of urgency” (Kotter, 2007, p 99). This step is the initiation of the change. Creating this sense of urgency is the primary responsibility of the company’s leadership. Kotter sees that 50% of companies fail with this first phase. The reason: “executives underestimate how hard it can be to drive people out of their comfort zones” (Kotter, 1995, p 97). Yahoo found this to be true, in their bottom-up approach to Agile transformation. The coaching staff at Yahoo worked to overcome the rut of traditional development practices (Benefield, 2008).
At a team level Scrum creates a sense of urgency to complete the sprint and deliver working software in just weeks. Agile processes such as Scrum, reduce the developers timeframe of concern from years until project completion, to weeks for sprint completion. This sprint deadline creates Kotter’s sense of urgency at a team level within the organization. To be successful a Scrum team must deliver working product in a matter of weeks. There will be no time to waste.
Leadership is required for transformation change, not management. Management by definition is about control. Agile processes are about empowerment, self directed teams capable of self-organization to solve complex problems. This requires a servant leadership style of interaction with the team. Agile coaches empower the organization to overcome the fear of change. Creating a sense of urgency that change is required. Support and renew this understanding that business as usual, allowing a large majority of projects to fail is unacceptable.
The second step is to form coalitions capable of leading the change. Empowering the group with the authority required to make changes in the organization. Kotter notes that this coalition consist of managers and various key individuals from all levels of the company. The coalition may be small at first but it must grow in size and influence as the transformation progresses. This is a top down vision of the transformation change. In the bottom-up Scrum model there is also a coalition of the willing. A Scrum team is a cross-functional, self-organizing and empowered group. However, a team does not exist within a vacuum. It must forge relationships with other teams and groups within the organization to achieve the goal of producing working software. Typically the relationships at the boundary area of a Scrum team are the responsibility of the Scrum Master. This is a coaching role for the Scrum Master. Coaching the team to work well with other groups, to set expectations, and to met their commitments. Also coaching the organization and the other groups in how to effectively interact with a Scrum team. These coalitions are foundational to building trust across people and teams.
The third and forth step in Kotter’s model are the steps of creating a vision and communicating that vision. For Agile transformations this vision is one of creating new ways to develop software that meets the customer current needs and is capable of evolving to met their future needs. The focus for the Agile transformation is on processes that allow the organization to create quality software effectively, while embracing the changes required by market forces. The Agile coach fulfills a vital role in creating the vision of better processes, developing strategies for achieving and measuring progress toward that vision. They are the evangelist for the Agile process, the exemplar of Agile philosophy in action.
The fifth step is to empower others to act on the vision. This is the essence of the role of Scrum Master. Kotter observes that a major error in transformation processes is not removing obstacles to the new vision. “Too often, an employee understands the new vision and wants to help make it happen. But an elephant appears to be blocking the path” (Kotter, 2007, p 101). Scrum has recognized this disfunction and has specifically charged the Scrum Master with responsibility to remove these impediments. Many impediments are resolved by just raising the visibility of the blocking issue to the level of awareness, then creating a plan to focus attention and changing behaviors. Quite often these impediments are of an organizations nature and cannot be resolved without the intervention of other department leaders. This is when the Scrum Master must facilitate understanding of the issues and offer coaching in seeing the cost of the organizational impediment, and the benefits of resolving these systemic issues.
Kotter’s sixth step is to create short-term wins. Success breads success. Planning for and recognizing the small success toward the larger vision is a powerful tool. Agile processes create this opportunity through the iterative process of incremental changes to the software application within the Scrum sprint. The sprint is a short timeframe in which the development team plans the work to be accomplished and then executes the plan. Resulting in a small increment to the overall feature sets of the application. This increment may be releasable to the customer, resulting in speedy return on investment. In these short timeframes (one to four weeks) the teams and stakeholders begin to see real success.
Consolidating improvements and institutionalizing these practices while still allowing and encouraging continued change is the seventh step required for transforming the organization. The increased credibility that Agile teams with successful processes create in changing system structures and policies will enhance the progress. Kotter warns not to declare victory too soon. Time is required for the new system structures and ways of doing to become the norm. Rituals such as the daily Scrum meeting help to reinforce new habits and build practice of key disciplines into the work life and social fabric. However too frequently change efforts revert to the old ways; therefore it is imperative that change agents reinvigorate the process by celebrating achieved milestones and then setting new goals.
In the eighth step, Kotter notes that articulating the reasons for new behaviors and tying them to corporate success, institutionalizes the new approaches. Helping people to see the right connections between cause and effect requires communication and vigilance. Institutionalization also requires an eye toward succession planning. Promoting from within will anchor the change in the culture. Insuring that new players coming into the company are committed to the transition especially in the top ranks is vital.
Just as Agile process of software development are iterative in nature, so to are Kotter’s phases of transformation. One is never done with transformation, for there is always more to be done to secure the transformation within the organization fabric.
Conclusion
The Agile software movement is a transformational method of correcting a culture that has allowed high rates of failed projects. Agile development is a systematic view of managing a chaotic process through empirical process controls. The transformation to Agile processes require time for people to internalize new ways of acting. The Kotter model of organizational transformation phases are facilitated by a coach that has experience in Agile transformations. Agile coaching at both the team level and above are key factors of success in transforming the organization. Agile coaches develop the team and their skills to perform within new frameworks that deliver quality constantly while adapting to ever changing conditions. Coaches develop processes that encourage innovation and reward knowledge creations. Agile coaching is a very important factor in success rates of Agile transformations, and thereby software development projects.
References
Augustine, S. (2005). Managing agile projects. Upper Saddle River, NJ: Prentice Hall.
Benefield, G. (2008). Rolling out agile in a large enterprise. Hawaii International Conference on System Sciences, pp. 462, Proceedings of the 41st Annual Hawaii International Conference on System Sciences.
Cunningham (2005). The coach. Cunningham & Cunningham, Inc. Retrieved June 17, 2005, from http://www.c2.com/cgi/wiki?TheCoach
Fowler, M., Highsmith, J. (2001). The agile manifesto. Software Development, 9(8), 28-35. Retrieved July 12, 2009, from http://agilemanifesto.org/
Hargrove, R. A. (1995). Masterful coaching: Extraordinary results by impacting people and the way they think and work together. San Francisco: Pfeiffer.
Kotter, J. (2007, January). Leading Change. Harvard Business Review, 85(1), 96-103. Retrieved July 24, 2009, from Business Source Premier database.
Peterson, D. B., & Hicks, M. D. (1996). Leader as coach: Strategies for coaching and developing others. Personnel Decisions International, Minneapolis, MN.
Schwaber, K. (2004). Agile project management with Scrum. Redmond, WA: Microsoft Press
Standish (1995). Chaos report. The Standish Group. Retrieved July 18, 2009, from www.projectsmart.co.uk/docs/chaos-report.pdf
Comments