Agile software development

The Need for Agile methods
@todo

The Agile Manifesto
On February 11-13, 2001, at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah, seventeen people met to talk, ski, relax, and try to find common ground and of course, to eat. What emerged was the Agile Software Development Manifesto. Representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes convened. Now, a bigger gathering of organizational anarchists would be hard to find, so what emerged from this meeting was symbolic a Manifesto for Agile Software Development signed by all participants. Taken from: "History: The Agile Manifesto"

So why bother with agile methodologies? Alistair Cockburn did research about how a process influences project teams. He concludes that the larger a project team the more a process can influence the effectiveness of the project development. However it is the people in the project teams that repeatedly argue the process that is proposed to use. Alistair Cockburn summarizes this behaviour in his quote: “People trump process”. It is true that good enough people in a team can deliver a solution where no process can improve the quality of the outcome. The most interesting part that contributed to the manifestation of agile methodologies is the fact that people working together with good communication and interaction have the ability to perform at significant higher levels than operated on an individual basis. According to Cockburn this becomes apparent time and again in brainstorming and joint problem-solving sessions. This the reason agile methodologies can contribute to project teams by focusing on increasing both the individual competencies and the collaboration levels.

For further reading visit: URL:http://agilemanifesto.org

Authors of the Agile Manifesto
On February 11-13, 2001, seventeen people met at the Lodge in the Wasatch mountains of Utah, they initiated an agile manifesto.

They were :
 * Kent Beck
 * Mike Beedle
 * Arie van Bennekum
 * Alistair Cockburn
 * Ward Cunningham
 * Martin Fowler
 * James Grenning
 * Jim Highsmith
 * Andrew Hunt
 * Ron Jeffries
 * Jon Kern
 * Brian Marick
 * Robert C. Martin
 * Steve Mellor
 * Ken Schwaber
 * Jeff Sutherland
 * Dave Thomas

The Agile manifesto encompassed the idea of several pre-existing methodologies including:
 * Extreme Programming
 * SCRUM
 * DSDM
 * Adaptive Software Development
 * Crystal,
 * Feature-Driven Development,
 * Pragmatic Programming
 * Lean Development

These authors all work for the software industry, and did a great distribution on agile software development. That’s the reason why they became participates for the Agile manifesto.

Defining traits of Agile Methodologies
Agility is the ability to both create and respond to change in order to profit in a turbulent business environment - Jim Highsmith, Agile Software Development Ecosystems

Such a general description allows a lot of room for debate as to whether a particular methodology is Agile or not. Where there is debate, it is fair to say that a methodology which doesn't have the following principles cannot consider itself Agile.


 * Least Design Up Front
 * Accommodates changing requirements, even late in the development cycle
 * Iterations are measured in weeks, not months
 * Low bureaucracy and administration overhead
 * Continuous customer feedback and involvement
 * Complete buy-in from all stakeholders

It's also fair to say that an Agile team that fails to realize any of these principles will struggle and probably fail.

Adaptive Software Development
Adaptive software development is a management to control software project for changeable requirement and short period project.

The three key point of adaptive software development are:
 * speculation
 * collaboration
 * learning

“The overriding, powerful, indivisible, predominant benefit of the Adaptive Development Life Cycle is that it forces us to confront the mental models that are at the root of our self-delusion. It forces us to more realistically estimate our ability.” - Jim Highsmith

Crystal Methods
To view the Crystal Methods section, please click the following link:

Crystal Methods

Dynamic Systems Development Method (DSDM)
What is DSDM?

DSDM is an example of an iterative (Agile) development model. DSDM approaches every step in the process to be completed only enough to be able to move to the next step. Since agile development is applied in situations where business requirements are likely to change, any further work would not benefit the overall process of delivery. The DSDM Consortium defines DSDM as a framework based on best practises that can be used to deliver new systems, which combines the most effective use of people's knowledge, tools and techniques to achieve tight project delivery timescales. The DSDM framework serves as a basis for a development and implementation process whether it is an IT project or a business change. The framework includes people (staff and skills), the technology that supports them (IT) and the processes that bind them all together (the business strategy).

Why DSDM?

The goal of DSDM is to target a demand for organizations to deliver working systems in shorter timescales. A typical DSDM project delivers an operational system within six months. The main emphasis however is that, according to the DSDM organization, most projects fail because of people issues rather than technology. Therefore the focus is providing people with guidelines to work together in a way that business goals are achieved successfully.

Because of the agile approach, DSDM build systems focus on the current needs in contrast to the traditional approach of going through all the perceived possibilities. Therefore, the resulting system is expected to better fit the business needs, be easier to test and be more likely to be accepted into the users' working practices.

The DSDM Framework Principles

The framework is based on nine initial principles which are applied in the different phases of the methodology:

Source: DSDM Consortium, underlying principles
 * 1) Active user involvement is imperative
 * 2) DSDM teams must be empowered to make decisions
 * 3) The focus is on frequent delivery of products
 * 4) Fitness for business purpose is the essential criterion for acceptance of deliverables
 * 5) Iterative and incremental development is necessary to converge on an accurate business solution
 * 6) All changes during development are reversible
 * 7) Requirements are baselined at a high level
 * 8) Testing is integrated throughout the lifecycle
 * 9) A collaborative and co-operative approach between all stakeholders is essential

The DSDM Framework Phases

The framework defines phases that apply to the process in the case of a new system development or the change of business processes. The core project process is preceded by the Pre-Project phase and end with the Post-Project phase:


 * Pre-Project
 * Feasibility Study
 * Business Study
 * Functional Model Iteration
 * Design and Build Iteration
 * Implementation
 * Post-Project

Within DSDM the phases “Feasibility Study” and “Business Study” are firstly done in sequence. Because these phases set the foundation rules for the rest of the iterative and incremental development in the remaining phases, they must be before any other work is done in a project. With these first two core phases done, the other three phases can overlap and be merged in any other order appropriate to the project. When the project solution is delivered the project team will hand over the solution to the Post-Project phase. This final phase includes the process of maintaining the solutions operations and monitoring the performance to make sure that the business requirements have been met.

Within each phase the goal is to produce a minimum set of products. These products are often specific documentation with a defined criteria and purpose. As a result the achievement of every purpose can be assessed. The way these products are produced and what the content will be, is left open for the concerning organization or project to determine. Table 1 provides an overview of the activity and products that apply to the DSDM phases.

''Table 1. DSDM activity and products''

For further reading visit: URL:http://www.dsdm.org/version4/2/public/default.asp

Overview
Feature Driven Development(FDD) is a model driven process of short iterations (two weeks). An overall model is established at the beginning of a project after which the project continues with a series of design by feature/build by feature iterations. A feature is a small and “useful in the eyes of the client” result.

Processes
FDD consists of five processes, three of which are per project, the other two being per feature.

Develop an Overall Model
This process involves the domain experts, chief architect and chief programmers. The main goal here is to establish the overall shape of the system. Classes and their inter relationships are defined.

Build Feature List
This process involves the domain experts, chief architect and chief programmers. Its goal is to produce fine grained feature list. Essentially the model developed in process 1 is functionally decomposed into subject areas. Each subject area contains business activities and each business activity can be broken into a number of business steps. The categorisation of these steps forms the feature list.

Plan By Feature
The feature sets are sequenced into a high level plan and assigned to chief programmers.

Design by Feature
A small set of features are designed by a feature team. Each feature is represented by a sequence diagram in UML. This diagram is the design deliverable and is peer reviewed by the feature team against the requirements.

Build By Feature
The code for each feature is written by the members of the team who own the business classes affected by the functionality of that feature. The completed code is peer reviewed. Once the code has passed the review and been unit tested it is marked for inclusion in the system build.

Best Practices
FDD is built around a set of eight best practices each of which complements the other. The practices are not exclusive to FDD, but are used in a unique combination in FDD.

Domain Object Modeling
@todo

Developing by Feature
@todo

Individual Class (Code) Ownership
@todo

Feature Teams
@todo

Inspections
@todo

Regular Builds
@todo

Configuration Management
@todo

Reporting/Visibility of Results
@todo

Milestones
For each feature of the feature list there are six discrete milestones. Three of which belong to the Design by Feature process; Domain Walkthrough, Design, Design Inspection; and three which belong to the Build by Feature process; Code, Code Inspection, Promote to Build.

Domain Walkthrough
@todo

Design
@todo

Design Inspection
@todo

Code
@todo

Code Inspection
@todo

Lean Development
Although Robert Charette originally defined the principles of Lean Development, he has not been it's primary proponent nor has he contributed much to the body of work relating to Lean Software Development. Instead, Lean Software Development has mainly been popularized through the work of Mary and Tom Poppendieck. They describe in detail how to apply the principles of lean production to the software development process. They specify a set of seven principles and 22 tools to help realise those principles. (Charette lists twelve principles )

Lean Production
Since Lean Development derives it's principles and practices from Lean Production it is important to have an understanding of Lean Production (sometimes referred to as Lean Manufacturing).

The concepts and tools of Lean Production were laid out in Womack, Jones and Roos book "The machine that changed the World: The Story of Lean Production". Lean Production draws on several production processes including Toyota Production System (TPS), just-in-time manufacturing and inventory management as well as adding it's own ideas to the mix. One of the key ideas in Lean Production - eliminate waste - is drawn from TPS. TPS has an extensive definition for waste, breaking it down into seven categories and suggesting how to deal with each one. Lean Software development then is the application of the principles of Lean Production to the software process.

Benefits of Lean Development
Lean Development allows companies to manage their risk and turn that risk into an opportunity. Robert Charette refers to this as "risk entrepreneurship" [Agile Software Development Ecosystems, Highsmith, 2002]. By proactively dealing with risk and adapting to changes in the market place brought about by circumstances and competitors a company can gain a competitive advantage. Lean Development enables software systems to adapt to this change rapidly, something that software systems are generally known for.

Principles of Lean Development
There are seven principles of lean development (though sometimes twelve are listed). Also, the naming of some of the principles has changed with time.

In Lean Software Development : An Agile Toolkit the following principles are listed
 * Eliminate Waste
 * Amplify Learning (Create Knowledge)
 * Decide as Late as Possible (Defer Commitment)
 * Deliver as Fast as Possible (Deliver Fast)
 * Empower Teams (Respect People)
 * Build Integrity In (Build Quality In)
 * See the Whole (Optimise the Whole)

(Titles in brackets are apparently newer names used in Implementing Lean Software Development )

Eliminate Waste - The Core Principle
In TPS, Taiichi Ohno analysed the concept of waste in great detail and categorised different types of waste, and ways of addressing them. In explaining Ohnos concept of waste, Poppendieck uses the phrase "anything which does not create value for a customer is waste" [ASDE, Highsmith] and when applied to software development this translates to implementation of features that are not core requirements, poor or missing requirements, administrative overhead and bureaucracy. Poppendieck maps Ohnos concepts of waste to software as follows:

Tools: Seeing Waste, Value Stream Mapping

Amplify Learning
Take advantage of the short iterations and feedback from customers to learn how to improve the product and the process. Present a choice of solutions as working prototypes rather than explore possible solutions. This way you limit choice as well as get a head start on the final solution.

Tools: Feedback, Iteration, Synchronization, Set-Based Development

Decide as Late as Possible (Defer Commitment)
In traditional software development processes, important decisions are made far in advance of implementation. Any mistakes made at the distant planning stage are amplified in implementation and their effect on the whole process. By deferring decisions to as late as possible the implications of mistakes are limited and you also can deal with change more easily.

Tools: Options Thinking, The Last Responsible Moment Making Decisions

Deliver as Fast as Possible (Deliver Fast)
Business success is often achieved by responding rapidly to change. A customer who needs a system for commercial applications wants that system as fast as possible. The faster you can deliver that system the greater the value to the customer and the greater the long term benefit to you. The concepts of this principle are related to the theory of Just-In-Time delivery.

Tools: Pull Systems, Cost of Delay, Queuing Theory

Empower Teams (Respect People)
"An organization that respects software developers as professionals will expect them to design their own jobs with proper training, coaching, and assistance. It will expect them to improve continually the way they do their work as part of a learning process. Finally, it will give them the time and equipment necessary to do their jobs well. In a lean organization, the people who add value are the centre of organizational energy. Frontline workers have process design authority and decision-making responsibility; they are the focus of resources, information and training." [Chapter 5]

Instead of treating software developers as cogs in a wheel, respecting their needs and abilities will give them a sense of responsibility and increase their motivation, and ultimately benefits everyone. Tools: Self-Determination, Motivation, Leadership, Expertise

Build Integrity In (Build Quality In)
Perceived Integrity - the ability of the software to "delight" the customer by functioning well, and Conceptual Integrity - a sense of cohesiveness and integrity between different parts of the system are key demand of customers. Refactoring and Testing ensure that these goals are achieved.

Tools: Perceived Integrity, Conceptual Integrity, Refactoring, Testing

See the Whole (Optimize the Whole)
By choosing the right measures and interpreting the measures correctly we can find the root cause and source of defects in systems. The assumptions are that defects are caused by individuals when in fact they are the caused by the processes and procedures in the system. Also, most projects are made up of many sub-teams, vendors and contractors. A shared agreement and understanding of goals is of huge benefit for the project and company in the long term.

Tools: Measurements, Contracts

Extreme Programming
Extreme Programming (XP) was developed by Kent Beck in mid 90’s. As projects reduced in size and became more dynamic with rapidly changing requirements, there was a need for quick project development life-cycle. Since 70’s widely used approach had linear structure characteristic known as “Waterfall Model”, project scope was planned out at the beginning of the project life-cycle, in which changing requirements in the middle of the development, made it very costly. Frequently changing requirements by customers required for more agile approach.

XP is known as a methodology of Agile development. This is used in small, collocated project teams up to 10 members, more is probable. Development is flexible and lightweight. XP is based on twelve practices and four groups - collaboration, feedback, revision and respect. Code is the most important part of XP, complete documentation is considered as a waste of assets. Simple code and design form the basis; this is understandable to programmers who join the team at the later stages. Peer programming complements it – what is obvious for one, is not for another, following the decisions is easy to understand the thought of the colleague.
 * Collaboration – active participation of all stakeholders of the project in which customer is a part of the development team. Project starts with creating simple design and ‘hands on’ coding. Test Driven Development approach is applied by adding new functionality to the code. Focus is on today tasks and requirements rather than looking at long-term. Regular meetings are held and frequent verbal communication between programmer team is practiced. Feedback acquired immediately through iterative structure of the development.
 * Feedback – feedback is effective if it is instantaneous. System feedback is achieved through Unit Testing. Unit Tests are created for the software throughout development phase on which complete software is built. Any changes to the requirements is done through collaboration process with customers who give the feedback to the existing model, changes are taken into account and presented as new requirements. XP exercise ‘Use Case’ diagrams and ‘Story boards’ through which requirements are presented.
 * Revision – XP development focuses on current tasks and problems. This makes written code often unstructured as it should be. Since the structure of the code is not planned out at the beginning of the project, regular refactoring is recommended and even compulsory for further development and maintenance. This takes courage to redesign the system and even sometimes discard completely parts of the code to implement a new functionality.
 * Respect – XP practices programming in pairs. Programmers split tasks, one is typing code (person is known as a ‘driver’), while the other programmer is following the code and working on an algorithm to solve the problem. Roles change in time. This is a good practice because every decision of the team matters which leads to the team loyalty.

There are twelve core practices in XP stated as a standard:
 * 1) Coding Standards – mutually agreed standard by the team throughout the development on language used, vendors, rules and format.
 * 2) Collective Code Ownership – every programmer in the team has a right on the code. Suggested to use repositories to store the code, every programmer has access rights.
 * 3) Simple Design – problem should be tackled and the simplest outcome should be applied.
 * 4) System Metaphor – classes and functions should do what they say on the ‘label’, naming should be given in mutual understanding for the functionality.
 * 5) Pair Programming - two programmers work on one task, one is typing and following the lead of the other who focuses on the functionality and algorithms to solve the problems, also reviewing written code.
 * 6) Planning Game – regular project meetings are held and are divided into release planning in which customers request necessary changes and new functionality, and iteration planning which is done after completing the iteration by presenting functional software.
 * 7) Test Driven Development – creation of a test case following by writing a code that satisfies the desired test result.
 * 8) Whole Team – all project stakeholders are actively involved and work together.
 * 9) Continuous Integration - team members should work on the latest release of the software, some team members who work on the delayed parts should integrate the code into the main release repositories as soon as possible.
 * 10) Refactoring – refactoring should be performed on a regular basis.
 * 11) Small Releases – functional releases should be built iteratively that form the project. These visible modules encourage customers, also used for evaluation purposes.
 * 12) Sustainable Pace – also known as a 40 hour week. Well being of the team is important, burnt out programmers do not perform at their best. Productivity is achieved by these means.

Scrum
See also full article on Scrum

Basic concept of Scrum
Scrum is a lightweight framework that helps people, teams and organizations generate value through adaptive solutions for complex problems. It uses iterative and incremental practices on the software product. It is raised in 1995 by Advanced Development Methodologies,Inc, and has become popular after the Agile Alliance is created since 2001.

The hypothesis of Scrum is, at the beginning of the product development, we could not definite the product’s final requirements. During the development duration, we need creation, develop, and get experience from mistakes, so there is no fixed procedure we can warrant the success. It seems like the Rugby football team, they have highest priority goal, familiar with the development process, have high self-control authority, and cooperate tightly, to ensure the develop team is going ahead to achieve their goals on every phase, every day.

One iteration lasts 30 days, it starts from the beginning of the new product’s requirements. The develop team must implement all functional parts which they select from the beginning. They have daily meeting, spend 15 minutes, to check the progress of every team number, to know what difficulties they met and try to resolve them as soon as possible.

Scrum’s advantage/ disadvantage
The most advantage of Scrum would be its changeability. Scrum can take action when something changes, e.g. requirement change from marketing.

Because of the changeable requirements using Scrum development process, it makes a great challenge to feature testing team especially when they are under developing test case. Because of unfixed and often changed requirements in every unit Sprint, testers often need to re-work on develop test case in a short time.

Scrum’s framework
 Glossary of Scrum terms 

(Material in this section is based on information available at Scrum Alliance)

Backlog: Backlog is the requirements for a system, expressed as a prioritized list of product backlog Items. These included both functional and non-functional customer requirements, as well as technical team-generated requirements.

Sprint: An iteration of work during which an increment of product functionality is implemented.

Sprint backlog: Defines the work for a sprint, represented by the set of tasks that must be completed to realize the sprint's goals, and selected set of product backlog items.

ScrumMaster: The ScrumMaster is a facilitator for the team and product owner.

Time-box: The duration time of daily meeting.

Sprint planning meeting: The Sprint planning meeting is a negotiation between the team and the product owner about what the team will do during the next sprint.

Daily Scrum meeting: A fifteen-minute daily meeting for each team member to answer three questions:


 * 1) "What have I done since the last Scrum meeting?"
 * 2) "What will I do before the next Scrum meeting?"
 * 3) "What prevents me from performing my work as efficiently as possible?"

The ScrumMaster ensures that participants call sidebar meetings for any discussions that go too far outside these constraints. The Scrum literature recommends that this meeting take place first thing in the morning, as soon as all team members arrive.

Sprint review meeting: The team shows their achievement to their product owner during this sprint. It lasts 4 hours.

Sprint retrospective meeting: The sprint retrospective meeting is held at the end of every sprint after the sprint review meeting. It lasts 3 hours.

Introduction of Scrum process

 * 1) Divide product backlog into Sprint backlogs, ensure Sprint backlogs can be fulfilled based on the current resource.
 * 2) Hold a Sprint planning meeting, to make sure the works should be finished in this Sprint. Assign work to every team number with priority.
 * 3) Start the Sprint development lifecycle, hold the daily Scrum meeting.
 * 4) After the whole Sprint lifecycle ends, show their achievement to their product owner in the Sprint review meeting.
 * 5) Hold the Sprint retrospective meeting at last, the team and ScrumMaster meet to discuss what went well and what to improve in the next sprint.
 * 6) The next Sprint starts.

Related Material
Ken Schwaber provides an informative and entertaining overview of SCRUM in this Google Tech Talk

Choosing an Agile Methodology
The methodology chosen needs to be on a level that the company is willing to work with. Different levels have different requirements such as:


 * Planning
 * Design
 * Programming practices

Extreme Programming (XP) is far more detailed than the other methodologies and defines programming practices, whereas the other methodologies are mainly involved with the project management and planning issues

The agile methodologies have different degrees of support for the different parts of the software development project's life-cycle.

The methodology to choose depends on:


 * The project size
 * The effort involved
 * Is the project mission or life critical
 * What is the company culture like
 * What build environments are in place(build everything before test allowed, etc)
 * How many senior developers in the team
 * What size is the development team
 * Are requirements going to change often (e.g. Nuclear Power Plant vs Company Website)
 * Order vs Chaos

All the above will determine firstly whether or not to take work with an Agile Methodology and secondly, which agile methodology to choose.

Tool Support for Agile Methodologies
While only a few tools are exclusively applicable to Agile, many tools have been developed to meet the needs of Agile teams and have gone on to find wider audiences. Many of the tools listed here are either specific to Java, or were developed for and with Java but now support other languages. This is mainly due to the popularity of Java during the time that Agile methods were widely adapted. Many tools exist for other languages also.

Scheduling and project management

 * OnTime Axosoft's OnTime is an award winning project management tool for software developers developing in the Agile Scrum methodology.
 * XPlanner - XPlanner is a web-based project management tool for XP projects. XPlanner is probably the most widely used open source project management tool and can be used on other Agile projects, not just XP.
 * AgileTrack - A commercial project management tool for Agile projects
 * TargetProcess TargetProcess is a commercial project management tool for agile projects

Version Control Systems
Version control is a central configuration management activity in any Agile project. Code is written frequently and checked in every day at least. A reliable version control system is central to any agile project. There are dozens of version controls systems to choose from, only the most widely used are listed below, but even that doesn't approach a complete list
 * cvs - A client server system, until recently the dominant VCS system in use. Still very popular.
 * subversion - Seen as a successor to CVS and addresses many of the shortcomings of CVS
 * mercurial - A nascent distributed version control system
 * clearcase - A commercial client server source control system owned by IBM. See here for details of how it can be used in Agile projects.
 * perforce - A popular commercial VCS, used by Google amongst others.
 * Visual SourceSafe A widely used version control system from Microsoft

Build and Continuous Integration

 * CruiseControl - CruiseControl is an open source continuous build and integration tool.
 * buildbot - Another open source automated build and test tool

Testing
Unit Testing 

Unit Testing is one of the principles of XP and other Agile methodologies and a whole ecosystem of unit testing frameworks has sprung up around Unit tests test code functionality at the smallest possible level of granularity. In object oriented programming this is usually at the level of a single method. There are simply too many unit testing frameworks to list here however the most widely known is probably JUnit.

 Mock Objects  Mock Objects and unit testing are intrinsically linked. Mock object allow you to create simulations of dynamic complex objects so that dependent code can be reliably tested without having to manage the complex objects state and complexity. There are many mock objects libraries for each language, the list below is a brief selection of what's available:
 * pMock for Python
 * EasyMock for Java
 * NMockLib for C#
 * RSpec for Ruby

Tips and Alternatives
The agile technique works well in collocation environments, for relatively small developer teams (four to ten people). Unlike the Waterfall model, which is more oriented at e.g. software contractors performing projects from conception to maintenance; Agile development uses iterations, which churns out new features and provides maintenance in a continuous cycle, with no set final plan and little documentation. This is not an approach suited for "reliable" software, such as security-critical projects.

Meeting weekly with the customers or other stakeholders will positively impact the success of Agile development. Collaboration is very important. Agile principles propose face-to-face meetings; this is easiest accomplished when team members are collocated in the same premises. There may be significant setbacks if all members work from remote, separate locations, which depends on the type of software, team dynamics, or other factors.

Part of Extreme Programming practice is to refactor regularly. Recoding existing code or changing the structure of the system, even discarding and writing brand new code. This is suitable for rapid prototyping, but may not be the most responsible usage of resources. Excess resource usage is preventable if the architecture of the system is planned from the very beginning.

Agile development works well with experienced developers who are comfortable in their position and skills.

Current Status and Future Directions
‘Agile’ development is a very dynamic and elastic technique that could be applied in a variety of ways to different kinds of projects. Radically evolving technology creates a need for flexible ways of dealing with a new set of environments and rules. There is much scepticism in ‘Agile’ development, however the technique works well for many and they don’t want to discard it or apply heavy-weight development style instead. The set of ‘Agile’ tools and processes provided may be improved. Saying that current ‘Agile’ method is not final and it will be most definitely enhanced.