All material is Copyright © 2010 by Gertrud & Cope and John Wiley. All rights reserved. This page reflects content from an early version of the manuscript rather than being a verbatim replica of the current publication.

 

“Individuals and interactions over processes and tools,” and “Customer collaboration over contract negotiation” comprise half of the values of the Agile Manifesto. These values are no less suitable to architecture than to any other facet of Agile development, and they lie at the heart of this chapter. Perhaps we hear much about the other two principles of working software and embracing change because they’re closer to the programmer, and because popular Agile is a programmers’ movement. Maybe half of software development is about nerd stuff happening at the whiteboard and about typing at the keyboard. But the other half is about people and relationships. There are few software team activities where this becomes more obvious than during architecture.

This chapter is about the people in a software development effort, the roles they play, and a little bit about the processes that guide them. To summarize the chapter, architecture is everybody’s job. We’ll maybe have some surprising things to say about customers and architects, too.

Going back to that saw, “Customer collaboration over contracts and negotiation” brings to mind another human-centered artifact of development: requirements, or whatever term one chooses to ascribe to them. Though the values of the Manifesto don’t mention end users, Agile practices tend to focus on the informally communicated needs of users, commonly called User Stories. Now you’ve picked up a book about architecture—one that even dares to use the word requirements—and perhaps you’re afraid that all of that is going to go away.

No, don’t worry: in fact, we’re going to push this perspective even further. We’ll venture beyond just the paying customer to the even more-important end user, who is often a different person that the customer. In fact, there are many stakeholders who can derive value from your product. Some of these stake-holding relationships come down to economics: Both customers and end users can benefit from low incremental cost and fast incremental time to market. End-users are stakeholders who want a well-tailored solution to their problem as well, including ease of use. Both Lean and Agile can help us out here. Lean teaches us that the value stream is important, and that every activity and every artifact should contribute to end-user value. Agile’s “customer collaboration” lets the value stream reach the deepest recesses of the software development process so user needs are explicit and visible to the vendor community.

There are important stakeholders outside your market as well. Your company benefits from a high return on investment (ROI), which comes from low cost and good sales. While good sales come from meeting market stakeholders’ needs, low costs relate to how vendors meets their own needs. Designers, coders and testers have more fulfilling jobs if they feel they are contributing to the value stream: reducing time to market by shortening the development interval, and reducing cost by reducing frustrating rework. We can reduce rework by carefully watching and listening to our customers and to the market on one hand, but to history and standards on the other. So both Lean and Agile can help on the vendor side, too.

Lean adds to the value stream by keeping the system consistent, and a good architecture is a de-facto standard that helps system parts fit together. Lean supports the value stream by reducing waste: a good architecture based on historical knowledge can greatly reduce the rework of re-inventing the wheel. Agile keeps the links in the value stream consistent with each other through “individuals and interactions over processes and tools.” The feedback in Agile supports Lean’s goal of consistency, while Lean’s trimmed and fit processes make Agile change more flexible.

Along with eliminating waste and smoothing out flow, overall consistency completes the three fundamental goals of Lean. For a software team, that means consistency between end user needs and what is produced. That implies that we must have consistency between the end user mental models, the end user actions, our Use Cases, the code, its architecture, the tests, the human interface—everything, everyone, everywhere. If everyone is communicating at just the right time, then we can avoid inconsistent actions that lead to waste and irregular production. And it’s hard to schedule the “right time” in advance because the future is full of surprises. You could wait until you were done coding and testing to engage the sales people, but maybe they want to develop an advertising campaign so they’re ready to sell when you’re done. You could wait until you’re done coding to make friends with the testers, but engaging them early smoothes later communication and builds the environment of team trust that is crucial to success.

The key to success in both Lean and Agile is the Lean secret: Everybody, all at once, from the beginning. For Lean architecture and Agile production to work, we must engage the right stakeholders and ensure that the process keeps them on the same page, optimizing value to the end user.

We identify five major stakeholder areas:

·    the end users,

·    the business,

·    customers,

·    domain experts,

·    and developers.

You might be tempted to partition these roles by process phases. However, Agile’s emphasis on feedback and Lean’s emphasis on consistency suggests another organization: that these roles are more fully present in every development phase than you find in most software organizations.

This doesn’t mean that everyone must attend every meeting, and it doesn’t mean that you should force your end users or business people to write code. It does mean breaking down the traditional organizational walls that hamper communication between these roles, particularly at those times of crucial design decisions.

A good development organization is organized around roles that have a solid place in the lean value stream. In the Organizational Patterns book (Producer Roles, [CopHar2004]) we discussed producer roles, supporting roles, and deadbeat roles. You might think that your organization has no deadbeat roles—until you map out your value stream.

We probably miss some of your favorite roles in our discussion here. We discuss the roles that have distinguished contributions to the value stream, and a few of the key roles who help lubricate the value stream. For example, user experience people open doors to the end user role. The roles here should be interpreted broadly: so, for example, “developer” covers designers, coders, and testers; “domain expert” may cover your notion of architect, system engineer, and maybe business analyst or internal consultant; “business” includes sales and marketing and perhaps key management roles; “end user” has the user experience role in their corner; and so forth.

Since this is a book on architecture you might expect more conscious focus on the architect. A true software architect is one who is a domain expert, who knows how to apply the domain expertise to the design of a particular system, and who materially participates in implementation (Architect Controls Product and Architect Also Implements in [CopHar2004]). Such an architect can be a valuable asset. We still avoid that term in this book for many reasons. First, having a role named “architect” suggests that the people filling that role are the ones responsible for architecture, when in fact a much broader constituency drives system design. Second, the word too often raises a vision of someone who is too far removed from the feedback loop of day-to-day coding to be effective. This drives to a key point of Agile software development, and can be traced back to the Software Pattern discipline and to Christopher Alexander’s notion of the architecture as “master builder” rather than “artistic genius.” The term itself has dubious historic value. As the best-selling architecture critic Witold Rybczynksi writes in his book The Most Beautiful House in the World,

… But who is an architect?… For centuries, the difference between master masons, journeymen builders, joiners, dilettantes, gifted amateurs, and architects has bee ill defined. The great Renaissance buildings, for example, were designed by a variety of non-architects. Brunelleschi was trained as a goldsmith; Michelangelo as a sculptor, Leonardo da Vinci as a painter, and Alberti as a lawyer; only Bramante, who was also a painter, had formally studied building. These men are termed architects because, among other things, they created architecture—a tautology that explains nothing. [Ryb1989, p. 9]

Third, most valuable contributions of good architects are captured in the other roles mentioned above: domain experts, the business, and developers.

We don’t separate out system engineering as a separate role. It isn’t because we find systems engineering boring or useless; it’s more that few people know what it is anymore, those that do don’t need much advice. For the purposes of this book, systems engineers can be viewed as domain experts (which are too commonly called architects) who can translate architectural idealism into stark reality. Done right, an architect’s job looks like great architecture; a coder’s work looks like great craftsmanship; and a system engineer’s job looks like magic.

Let’s explore the roles.

1.1.1      End Users

End users anchor the value stream. The buck stops there. Their stake in your system is that it does what they expect it to. By “expect” I don’t mean a casual wish, but a tacit expectation that goes beyond conscious assumptions to models that lie partly in the user unconscious. Most customers can express their wants; some can justify their needs; and when using your product they can all tell you whether it does what they expect it to. Most software requirements techniques (and some architecture techniques) start by asking users what they want or need the system to do, rather than focusing on what they expect it to do. Too many projects collect lists of potential features, driven by the business view of what the customer is willing to pay for (in other words, what the customer wants). Just having a description of a feature alone doesn’t tell us much about how it will be used or why he or she will use it (we’ll talk more about the why question in Section 6.2.1.

I learned a lesson in user expectations from a client of mine. The client builds noise analysis systems for everything from automobiles to vacuum cleaners. One thing I learned is that their clients for vacuum cleaners insist that a good vacuum cleaner make a substantial amount of noise when on the high setting. For the German market, this is a low roar; for other markets, it is a high whine. The goal isn’t to minimize noise, even though it’s possible to make vacuum cleaners very quiet without reducing their effectiveness. The goal is to meet user expectations.

One way to uncover expectations is to start a conversation with User Stories and proceed to goal-driven Use Cases that include scenarios and user motivations for the functionality. User Stories help end users think in a concrete way, and if we feed back our understanding of their expectations, we can gain foresight into how they will react if the system were deployed according to our understanding. If we do that job well we can build a system that technically meets end user needs.

Once you’ve gotten the conversation started, make it concrete quickly. One thing I learned from Dani Weinberg many years ago, from dog training, is that dogs learn well from timely feedback. They have difficulty associating delayed feedback with the associated behavior so they are at best bewildered by your praise or criticism. If you can turn around a feature quickly at low cost to deliver to the end user for a test drive, you can concretely calibrate your interpretation of their expectations. Prototypes can be a good vehicle to elicit the same feedback at lower cost and in shorter time.

One problem with requirements is that they never anticipate all the scenarios that the user will conceive. So we need to go deeper than scenarios and explore the end user’s perception of the system form. This is called the end user cognitive model, and it has everything to do with architecture. And it closely relates to what the user expects from a system. We’ll explore the mechanics of User Stories and Use Cases more in CHAPTER 6, but here we’ll help set the context for those practices.

Psyching out the end users

Use Cases capture user/system interactions that end users can anticipate. If end users can anticipate every interaction for every possible data value and input, then the specification is complete. It also makes it unnecessary to build the system because we’ve delineated every possible input and every possible answer, and we could just look up the answer in the spec instead of running the program. Of course, that is a bit absurd. Good software systems have value because they can, to some degree, handle the unanticipated. For example, let’s say that my word processor supports tables, text paragraphs, and figures. One Use Case captures the user/system interactions to move a table within the document. I can describe the possible scenarios: moving a table from one page to another; moving the table to a point within a paragraph; moving the table just between two existing paragraphs; or even moving a table within another table. The possibilities are endless.

Instead of depending on an exhaustive compilation of Use Case scenarios alone, we instead turn to something that matters even more: the end user’s cognitive model. Users carry models in their head of the internals of the program they are using. They trust that the program “knows” about tables, paragraphs, and figures. The end user trusts the programmer to have paid attention to the need for white space between the table and any adjoining paragraphs. These elements of the end user mental model, while quite static, are useful for reasoning about most possible Use Case scenarios.

If the program doesn’t have an internal representation of a text paragraph, of a table and of a figure, then the program must work hard to present the illusion that it does. Otherwise, the program will endlessly surprise the end user. This is true even for scenarios that the end user might not have anticipated while helping the team compile requirements. Another way of talking about user expectations is to note that, unless we are designing video games, the end user rarely finds surprises to be pleasant. Therefore, it is crucial that the system architecture reflect the end user cognitive model. That helps us design the system so it not only meets end users anticipated wants and needs, but so it is also resilient when asked to support an un-anticipated by reasonable scenario.

If we capture this model well, the system and its initial code will handle most reasonable scenarios as they arise. Users will of course invent or stumble onto scenarios that the system doesn’t yet handle quite right—perhaps, for example, creating a table within the table of contents. As programmers we then need to extend the system to support the new Use Case. But the system form is unlikely to fundamentally change. Such business domains models remain relatively stable over time.

When it comes to testing and to end-user engagement in ongoing feature development, we want the end user’s feedback about how the system behavior matches their expectations. System tests, usability testing, and end-user demos all help. But when we are laying out the product architecture, we want the end user’s cognitive model of the system. Because the architecture reflects that model, the end user is a stakeholder in the architecture. In fact, the original goal of object-orientation was that the code capture the end user mental model (we’ll speak more to this in Section 6.1 and again in Section 7.1); to do that, we need to elicit the model.

It can sometimes be difficult for programmers to separate themselves from their engineering world enough to grasp the end user perspective, and it can be difficult for end users to objectively introspect about what their internal world models really are. This is where user experience people bring value: they are often the key to the stake held by the end user. User experience people are also trained to recommend interfaces and approaches that, while natural to end users and their mental models and behaviors, look crazy to a programmer. (If you don’t believe us, just read Raskin’s book on interaction design [Ras2000]).

Don’t forget behavior                                                                  

Yes, of course we still collect Use Cases, even if they handle only the most common cases and even if they can’t be exhaustively enumerated. Software ultimately is a service, not a product, and Use Case scenarios help us keep that fact in focus. The more Use Cases, the better? Well, but moderation is a key virtue here. Gathering requirements is expensive and hard, and it’s easy to go too deep into requirements too early, because end users may behave differently when faced with a delivered system than they envision in the abstract. To gather detailed requirements about ever-changing behaviors is waste, so that’s not Lean.

Also, it’s important to strike a balance between domain modeling (CHAPTER 5) and behavior modeling (CHAPTER 6). Behavior has form, too (ever hear the phrase “form follows function”?), is part of the end user mental model, and should be captured in the architecture. In the long term, the domain structure tends to be more stable than the behaviors we capture in Use Case scenarios, and its forms are the cornerstones of a good architecture. Object-oriented design techniques traditionally have been good at capturing the domain model but really bad at capturing the behavioral models.

There’s another good reason to capture Use Cases: testers are also stakeholders in the system’s quality and then need something to exercise the system behavior. In fact, one good way to drive system design is to gather the domain structure and the system behaviors in parallel and to let both drive the design. This is called behavior-driven development (BDD [TODO: BDD Cite]). Note that this means that the end user, the developer, the tester and the user experience specialist should be engaged together from very early in the project.

Focusing more on form than on function will help drive you in a direction that supports what users expect. Do that well, and you’ll easily be able to provide what they say that they want and need.

The End User landscape

Many systems have multiple end users and potentially multiple value steams. Think of a banking system. Simple exercises in object-oriented design courses often present account classes as the typical design building blocks and account-holders as the typical users. Yet bank tellers are also end users of a bank computer system. So are actuaries: the folks who look at the details of financial transactions inside the bank. So are the loan officers and the investment staff inside the bank. Does the concept of “account” suit them all?

Identifying such end user communities is crucial to a sound architecture. Different user communities have different domain models. Derivatives and commodities are potential domain entities to the investor, but not to the teller nor to the loan offer. Mortgages are domain entities to the loan folks but not the investment people. Yet somehow banks seem to run with a single common underlying domain model. What is it? In most complex financial system, the basic building blocks are financial transactions: those become the “data model.” From the user interface, different end users have the illusion that the system comprises accounts and loans and pork bellies. We’ll find that accounts really fall into a middle ground called Contexts: stateless collections of related behavior that behave like domain objects but which are richer in functionality and closer to the end user than many domain objects are. The Agile world has started to recognize the need for such user community differentiation, and that realization is showing up in tools like Concept Maps [TODO: Cite Jeff Patton].

Much of the rest of this book is about learning to identify these layers and to create an architecture that will allow the most common changes over time to be encapsulated or handled locally. Object-oriented architecture isn’t just a matter of “underline the nouns in the requirements document” any more. Before a project can build a resilient architecture, its people must know enough about the end user communities and their many mental models that they can form the system around long-term stable concepts of the business.

1.1.2      The Business

While end users ultimately wants the software to provide some service for them, the business has a stake in providing that service, as well as a stake in the well-being of its employees. That usually means paying the employees a wage or salary and/or bonuses, and that usually implies making money from the software. Good software businesses usually have a stake in growing their customer base and giving good return to investors—which means that the enterprise will have a diversity of end users or customers to support in the market. We usually think of business stake-holding as lying with sales and marketing, or with the Product Owner in Scrum.

If the business can serve more customers, it both grows its stake in its customer base and likely increases revenues. Good revenues are one way to be able to pay employees well; another is to reduce costs. Architecture is a means to hold down long-term development costs.

The business itself provides key inputs to the architectural effort that can hold down cost. One of the most important business decisions is the project scope: the business owns this decision. Nonetheless, it must be an informed decision. Scoping has to balance the expectations of all customers and users against financial objectives (should the scope be the union of the entire market’s expectations? Or should the business focus on the 20% of the market where 80% of the revenues lie?) The product scope can’t take development into an area that depends on technology that won’t yet be mature in the product’s lifetime (are we building electric cars? Business forecast software that uses artificial intelligence?) Such insight comes from domain experts and developers.

Just who is “the business”? The board of directors, executive management, business management, marketing, and sales are all key roles in this broad view of the business. Again, you probably don’t need all of these people at all meetings. But if you are discussing scope, or the what-the-system-does part of the architecture, invite selected representatives of these areas to the table.

The business may also hold down costs by making a buy-versus-build decision. Such decisions of course have broad and lasting influence on the architecture. Such decisions should be informed by customers and end users, who desire selected standards, and by domain experts who can advise the business on the feasibility of integrating third-party software. And don’t forget the developers, who actually have to do the work of integrating that software, and the testers, who have the burden of testing a system that contains a potential “black box.”

A special note for Managers

It’s perhaps noteworthy that though Scrum is a risk-reduction framework, or a framework to optimize ROI, it has no role named manager. The ScrumMaster has all many of the characteristics of a good servant-leadership style manager, and the Product Owner has the business savvy and the command-style attributes of a good Product Manager while lacking all of that role’s control-style attributes.

Remember that two keystones of Agile are self-organization and feedback. For your team to be successful, you should use the influence and power of your position to help make that happen. One of the most important roles of line management is to remove impediments that frustrate the team, or that slow the team’s progress. A line manager’s attitude can make or break the esprit de corps of a team. That is worth more than any methodology or tool.

A good way to think about managers in an Agile context is as members of a team whose product is the organization. As such, managers aren’t preoccupied with the production process for the enterprise product; instead, they influence that process by putting the right organization in place. Good organizations support effective communication through group autonomy, collocation, and group functions.

While developers should set their horizons on end users who care about the product, managers can focus on customers. This helps free the development team from “pure business” issues where they have less experience, insight, or responsibility than managers. Because most software these days is sold as a commodity, customers are more concerned with revenue streams and delivery dates than the actual product itself. That means that they may be more interested in the process than the product. Managers are a good entry point for these concerns—much better than the development team.

All that said, don’t forget: everybody, all together, from the beginning.

1.1.3      Customers

Customer is an almost emotive word. To not sign up to be customer-driven, or to strive for customer satisfaction, is to be a heathen and to be “not a team player.” If we look beyond the mythical associations of the terms we find that it’s useful to separate the Customer role from the End User role. Simply put: End Users are more about products and services, while Customers are more about process.

… as contrasted with end users

Customers and end users are interesting links in the value stream. Agile uses the word “customer” and we find the word featured both in the Agile manifesto and in much of the original Scrum vocabulary. Yet the Agile Manifesto says nothing about the end user, and we don’t find either role formally in today’s Scrum framework!

Customer and end users are very different stakeholders. The end users’ stake is relatively simple by comparison. They seek service from the software you are developing; that is the value that you supply for them. The end-user value to the development organization is that they are usually the source of revenues that ultimately feed your team members. It’s a good deal for them if your software in fact supports services that increase the end user’s quality of life.

The customer is in general a middleman. In general terms, customers are not consumers of the service that your software provides; they treat your software as a product that passes through their systems the same way that gold in a Japanese martini passes through the digestive system of its consumer. It may come out the other end in different packaging, but it is still the same product. When engaging such customers, consider their stake in opportunistically developing products for which there are yet no end users. While all stakeholders want to reduce risk, customers in this position are particularly averse to risk. They are much more interested in delivery times and in your development costs (because those become their costs) than they are in functionality. Therefore, the customer has a larger stake in your development process than in the service that your software provides. You may be engaging customers more in the area of process improvement than in development enactment. It is important to accord such activities a place in your enterprise using retrospectives—and here, retrospective means a serious activity that encompasses business scope and issues of trust. What passes for a retrospective in the two-hour “check-ups” at the end-of-sprint is inadequate. See [Ker2001] for more on retrospectives.

Customers have at least one other key stake-holding relationship to the architecture, and that relates to the market segments that they serve. In general, when it comes to customers, the more the merrier: Customers are a path to markets and therefore to revenues. However, different customers often represent different constituencies and bring the power of negotiation or market leverage to the negotiating table. If customers want to distinguish themselves from their competition, they will want their own configuration of your product. So one size does not fit all. Such configurations may extend beyond simple algorithms to variations on the system form: its architecture. A good architecture can be a tool that helps the Business cater to the needs of individual customers and market segments by supporting plug-and-play substitution of system modules.

Of course many combinations of customer and end-user are possible. They are sometimes one and the same. Sometimes you have the luxury of working directly with end-users, achieving the ultimate Agile goal of short feedback loops that avoid requirements misunderstandings.

It is common that a Scrum team delivers to another software team developing code in which your code is embedded. These projects are challenging to run in an Agile way. It is rare that such a team has or even can have meaningful discussions with end users. If your software has repercussions on end users (and what software doesn’t?), then your own customer is likely to introduce delay that makes it difficult to receive timely customer feedback before starting your next sprint. Testing, and, in general, most notions of  “done,” become difficult. In these situations it is much better to extend the scope of “done” to include such customers and to effectively enlarge the scope of development to engage the party as a development partner rather than as a customer.

Sometimes you have customers who yet have no end users because they are striving to develop a market, trying to develop a service that they hope will sell. In an Agile context, be wary of the possibility that your customer will look to you as a vendor to be the source of the requirements!

Other times you yourself are both the developer and the end user, such as might occur when developing tools that support the development team. That’s great! We encourage you to continue the discipline of separating your customer role from your developer role. Dave Byers relates:

Because in the developer role you're trying to get away with doing as little as possible, but in the customer/user role you want as much done as possible. Separate the two and it's possible to find a decent balance. Don't separate them and chances are you'll drift too far to one or the other. [Bye2008a]

Indeed, the possibilities are endless. Use common sense, guided but not constrained by Agile and lean principles.

 “Customers” in the value stream

Sometimes our “customers” are just politically or organizationally separate entities on the production side of the value stream. If you build framework software and sell it to a company that embeds it in their product, then there are no real end-users of your product in your customer. Your end users are on the other side of your customer—and that can cause a break in the value stream.

If you are in this situation, look carefully at the interfaces between the organizations and look for waste, delay, inconsistency, or boom-and-bust production cycles. If you find such problems, then Lean has some answers for you. (If you don’t find such problems, then that’s great! Like my grandfather said: If it ain’t broke don’t fix it.)

Lean’s main answer to these problems is to integrate both parties more fully into a single value stream. Remove obstacles to feedback between the teams. Leverage standards as a supplement to communication, and as a way of reducing changes in dependencies.

Remember that the same principle applies if you are taking software from another vendor and embedding it in your project. Try to close the gap. Toyota did this when they found they needed a new battery for their hybrid car. They didn’t have the expertise to build one in-house and couldn’t find one from any supplier. They solved the problem by partnering with Matsushita to jointly design a battery uniquely suited to the design of the Prius. [Lik2004, pp. 208-209]

1.1.4      Domain Experts

Domain experts are usually the grey-haired folks in the organization who know stuff. Domain experts are often the most direct and most explicit source of insight and advice on how to structure a new system. Most new systems in a domain look—from the perspective of form—very much like previous systems in the same domain.

It’s important to understand that everyone in an organization is probably an expert on something; otherwise, they wouldn’t be there. Software development is rarely a matter of having enough muscle to get the job done, but rather of having the right skill sets there. It’s about having diversity of skill sets, not just that one can overtake the market using Java muscle.

On the other hand, the folks commonly called domain experts have a special role in architecture. Over the years they have integrated the perspectives of multiple end user communities and other stakeholders into the forms that underlie the best systems.

Such knowledge is a priceless asset. Consider the alternative. With no knowledge of the best long-term structure of the system, designers would have to start with first principles—end user domain models at best, but more likely Use Cases—and try to derive the objects from those. It becomes more difficult if the team must deal with Use Cases from several different kinds of end users (e.g., both savings account holders and actuaries for a bank), and becomes even more difficult if the scope covers multiple clients or customers. The knowledge of the form suitable to such a complex landscape assimilates over years or decades, not over sprints or months. If you have your domain experts handy, they can relate the forms that they have already integrated, and in any case can point out areas that have been particularly challenging in the past.

Domain expert engagement is to architecture as end user engagement is to feature development. You should find that end users and domain experts are your most treasured contacts in a lean and Agile project. It is difficult to establish good working relationships with both of these roles (with end users because of organizational boundaries and with domain experts because of their scarcity), but make the extra effort. It’s worth it.

No ivory tower architects

Domain experts often bear the title of Architect. In the Organizational Patterns book [CopHar2004] we find patterns such as Architect Controls Product and Architect Also Implements, that use the name “architect” exactly in the sense of this common title. But both patterns suggest the architectural principles and domain expertise embodied in the role.

Today, we prefer the term “domain expert” more and more and the term “architect” less and less. The reason? In practice, “architect” isn’t a very distinguishing title. Interaction designers and coders have as much or more influence on the overall form of the system—its architecture—as titled architects do. In an Agile framework we value everybody’s contribution to the architecture, and to have a titled “architect” can actually disempower other stakeholders with deep insights. Differentiating the role of “domain expert” along the lines of expertise and experience, instead of along the lines of contribution to product foundations, better captures the stake-holding relationships.

Experts in both problem and solution domains

Don’t forget solution domain experts! It’s easy to get too caught up in value stream thinking that insists on tracing all business decisions to the end user as stakeholder. The business is also a stakeholder, as are the developers, and they want to use the best tools and design techniques possible. In the most general and theoretical sense, this eventually contributes to the value stream in a way that will benefit the end user, but it’s easier to think about it in terms of the more direct benefits to the team: benefits that help them do their job better.

Innovation in the solution domain goes hand-in-hand with long-term experience from solution domain experts. A good object-oriented expert can tell you not only where OO will give you benefits, but can also tell you where it won’t give you benefits. (In fact, a good rule of thumb is to trust someone as an expert only if they are good at telling a balanced story. An advocate is not always an expert, and an expert is not always an advocate.) So you want good dialog on the team between the innovators (which can be any role on the team) and the solution domain experts.

Keep your architecture team balanced so that both problem domain experts and solution domain experts have an equal say. One problem with stovepipe development is that needed dialog between these two perspectives turns into a war, because the earlier one in the process over-constrains the other. It is often a problem for a business to be driven too much by technological innovation, but it is even more problematic to be driven by the MBAs. To read a depressing case study about how this imbalance can go wrong, read Richard Gabriel’s post-mortem of Lisp innovator Lucid [Gab1998].

1.1.5      Developers and Testers

Developers are where the rubber meets the road. Their main job in architecture is often to rein in the grand visions of the business and architects with grounded domain expertise. As in Scrum, the developers should own the development estimates—after all, they’re the ones who will do the actual work of implementing. As we mentioned before it’s even better if the architects also implement—or, turning it the other way in terms of this book’s terminology, if at least some of the developers also have deep domain expertise.

As such, developers are the main oracle of technical feasibility. They are the primary solution domain experts. They should be active experts. For the tough questions, opinion isn’t well informed enough to make a long-term business decision or architecture decision. It’s important to gather empirical insights. Developers can help by building prototypes that compare and contrast architectural alternatives that are up for discussion. At PatientKeeper in Massachusetts, the Scrum Product Owners might spend months building prototypes and workflow models to refine their understanding of the design space. Developers are taxed to support them with prototyping tools and in building the actual prototypes.

Remember that developers are the primary channels of interaction between teams. If you believe Conway’s Law, that says that the team structure mirrors the architecture, then you can probably believe that the interaction between parts of your architecture will be only as effective as the interactions between the team members representing those parts of the architecture. As you frame out the form of your system, make sure that the stakeholders for the parts—at the level of the coders—negotiate the interfaces through which they will interact. Much of this negotiation will of course involve domain experts.

Developers and testers should be friends. While every serious system needs some acceptance or system testers who write double-blind tests, you should have ongoing testing support during development. You even need that at the system level. For such testing, the testers need to know requirements at least as well as the developers, so they’re likely to be invited to a lot of the same meetings as developers.

Even though developers and testers should be friends, at least some of them should play a game of “hide and seek.” The developer and tester can have a friendly meeting with the business people to agree on the requirements, and then they go their separate ways. The tester codes up tests for the new feature while the developer implements the feature in the current architecture. The developers works hard to implement the feature exactly as they come to understand it, going back to the business if necessary. The testers work hard to test the feature as they understand it, also asking for clarification when they need it. After one or two days of work, they come together to see if their perspectives meet up. My grandfather used to say that two heads are better than one. Start by having two sharp thinkers develop their interpretation independently; this doubles the opportunity to discover latent misunderstandings. That’s the “hiding” part of hide-and-seek. Working independently avoids groupthink, and avoids one personality overpowering the other with arguments that things must be thus-and-so [Jan1971]. Then, having “found” each other, reason together (again, with the business if necessary) to clarify mismatches. Not only does this approach increase the chance of uncovering requirements problems, but it is a weak form of pipelining, or parallelism, that shortens feedback cycles.

As for testers, there’s an old saw that architecture defines your test points. Testers have a stake in the architecture that it be testable. Hardware designers have become good at something called DFT, or “design for testability.” Software people haven’t come quite that far, but some testers have more insight and instinct in this area than others. Use testers’ insight to make key APIs available to support your test program.

Last but certainly not least are usability testers. Usability testing comes rather early in development, after Use Cases have been firmed up and before coding begins. You can do user experience testing with mock-ups, prototypes, or with simple hard-copy mock-ups of the anticipated screen designs. Usability testing can validate whether the team has captured the end user mental models properly: a crucial test of the architecture.

Your longstanding development process is in place, and you’re wondering what an Agile process should look like. Traditional development processes often organize into one (or both) of two patterns: one that exhaustively covers the stages in sequence, and one that exhaustively covers the tasks in roles. A total ordering of tasks can over-constrain self-organization, and a task organization alone can become arbitrary if it’s not grounded in knowledge or resources that support the value chain. A generic framework like RUP that tries to delineate all the roles in general has difficulty mapping onto domain-specific roles and processes, and it’s difficult to map these roles onto the value stream.

In this book, we discuss only the most basic notions of software process as they relate to Lean architecture and Agile production. This is a book about architecture and implementation. You might ask: Where does architecture start and end? To answer that, we need to answer: What is architecture? Architecture isn’t a sub-process, but a product of a process—a process called design. Design is the act of solving a problem. Viewed broadly, and a bit tongue-in-cheek, we might say that there are only three processes in software development: analysis (understanding the need); design (solving the problem); and delivery (which may include engineering, installation, shipping and deployment). This means that the time scope of architecture is broad.

A pithy but adjustable problem statement makes a great project compass. To do a good job of analysis on an extensive, complex market takes time. Getting everyone in the same room shortens feedback loops; if you can’t get them in the same room, minimize the number of communication hops between them (see the pattern Responsibilities Engage in [CopHar2004]. We care more about how roles connect to the value stream than to their place in the process (e.g., to have marketing people feed analysts who feed architects who feed designers who feed developers who feed testers) or to how their responsibilities fit together into a comprehensive set of tasks (as in RUP). You can find the stakeholders: the end users, the business, customers, domain experts, and developers in just about every enterprise that builds something for someone else. Some traditional software development processes translate well to an Agile and Lean world, but others merit special attention. So, with a light touch this section offers some rules of thumb on the process.

1.2.1      Getting started

The vision and problem statement come very early in development—often even before you have customers. Once the vision is in place, use your marketing people to extract knowledge from the market, and your domain experts to extract knowledge from the business world and technology sector. This knowledge can prepare both the business and the development community to shape their understanding of the domain and to start to understand the forms of the architecture.

If this is a new project, consider the organizational patterns that tie together the structures of your domain, your market, and the geographic distribution of your development team. (Here, “geographic distribution” includes separations as small as one building away or more than 50 meters distant. Don’t underestimate the power of space!) In particular, Conway’s Law [CopHar2004] and the related patterns Organization Follows Location and Organization Follows Market are major considerations in organizing your teams.

We have heard several conference talks on software architecture that start with a claim such as, “To get started on architecture, first get your requirements in hand.” It is true that we need to understand end user requirements if we are to deliver value to them; we dedicate CHAPTER 6, CHAPTER 7, and CHAPTER 8 to that topic. But the foundations of system structure lie elsewhere. The other side of the same modeling coin is domain expertise. Think of domain expertise as a broadening of the end user mental model into the realms of all stakeholders taken together. Expert developers learn over time what the fundamental building blocks of a given system should be. This is also the place where customer—as opposed to end user—concerns weigh most heavily. Some of those reflect the end user perspective, but they sometimes must honor the concerns of other stakeholders whose views are radically different from those of the end user. For example, checking Account Holders think that banking software comprises a ledger that mirrors the entries in their own checkbooks. However, an auditor chooses to view that account as a process over an audit trail in a transaction log. Both of these are real; which of these wins out as the systems foundation is a function of many Agile concerns, particularly ease of use, and frequency and type of changes to system functionality.

Those models balance enough generality to accommodate a wide variety of business scenarios with enough concreteness to provide a shared vocabulary for all stakeholders. Such care for the end user perspective on architecture, combined with its concreteness, take us from the problem definition one step closer to a delivered system. Of course, in real development we work on the problem definition, architecture, and use cases in parallel; however, from a conceptual perspective, architecture and its articulation provides a vocabulary and foundation for the later concerns with what the system does. So, guess what: everybody, all at once, early on rules again.

Also, if this is a new project, start small. Great projects grow from small projects that work. Have your developers work with analysts to explore the domain by building prototypes. Once you have a vision of where you’re headed, put together one or two teams to frame out the architecture and to demonstrate rudimentary functionality. Here, “team” means five to seven people. Aim for an early success and for a firm foundation that will support the product throughout its lifetime. The primary consideration should be on supporting change in the long term, being particularly attentive to feedback loops; if you can change, you can improve the value stream. The second consideration is to build your process around the value stream. Strong domain knowledge, and its articulation in an architectural framework, is one of the best things you can do to support change and to draw attention to the value stream.

1.2.2      Customer Engagement

Your thoughts will soon turn to delivering features. We’ll emphasize a previous point again: focus on user expectations rather than just wants or your perception of their needs. User experience people are experts in extracting (and anticipating) user expectations.

It is usually important to study end users and even customers in their native habitat. Don’t bring them into your office, but go to theirs. It doesn’t matter whether your office has all the touches of the best interior decorator in town, or whether it’s just a nerd’s paradise—it just can’t replace the client’s home base as an environment to learn domain knowledge and the context in which requirements arise.

To say this goes against the popular practice of on-site customer. Recent studies have found that on-site customers can in fact compound the requirements process by creating problems of trust [MaBiNo2004], [Mar2004]. On top of that is the more obvious problem of missing key contextual cues that arise in the environment. Our colleague Diana Velasco tells of a site visit where the client was describing the process they used but failed to mention the sticky notes posted around the border of the computer screen, and also wasn’t conscious of the importance of the “crib sheet” notebook that everyone kept as a guide to navigating the screen command structures. These are crucial components of the developer world and are crucial to system architecture.

Beyer and Holtzblatt’s book Contextual Design  [BeyHol1998] offers a wealth of techniques for exploring and capturing end-user mental models. Be selective in the tools you adopt from this and other sources. Use these tools on customer site visits to garner insight both for architecture and Use Cases.

As described in Section 3.1.1, you want to actively elicit feedback from end users using short development cycles or by using prototypes and models during analysis. A good feedback cycle has the appearance of causing problems. It will cause emergent and latent requirements to surface. That means rework: the value of prototypes is that they push this rework back into analysis, where it has more value. And most important, good end user engagement changes end user expectations. It is only by participating in a feedback loop that’s grounded in reality that customers get the opportunity they need to reflect on what they’re asking for. If your customer changes their expectations in the process, you’ve both learned something. Embracing change doesn’t just mean reacting to it: it means providing the catalysts that accelerate it.

1.3      The Network of Stakeholders: Trimming Wasted Time

Now that we have covered the roles, we come to the heart of the matter. There is nothing particularly lean or Agile about the roles themselves. What is important is how they work together. Here we come back to the Lean Secret: everybody, all at once, from the beginning.

1.3.1      Stovepipe versus Swarm

Old-style software development is patterned after the industrial assembly-line models of the Henry Ford era. In a simple, old-fashioned assembly line, workers interact directly only with the people in the adjacent station on the line. Worse yet, they may not even interact with the people, but may focus totally on the artifact and on their task of reshaping it or attaching something to it that adds value to the product. In manufacturing one can push this independence all the way back into the design process, because even designers can count on the laws of physics holding for the parts they design and how they will fit together and hold up in deployment. The development process is divided up into stovepipes: independent spheres of influence lined up side-by-side to create a product piecemeal.

Software has no equivalent to the laws of physics. Alistair Cockburn likens software construction to group poetry writing. It requires many different talents, ranging from knowledge of the business domain to good programming skills to keen insights into ergonomics and interaction design. What’s worse, these skill sets can’t easily be separated into process steps that can be done one at a time. And even worse, most of these skill sets drive some aspect of the basic system form: its architecture. If you follow that chain of dependencies, we arrive to the conclusion that we need everybody, all at once, from the beginning.

Figure 9:  Stakeholder Relationships

Look at Figure 9, which summarizes stakeholder relationships discussed earlier in this chapter. If we had drawn such a diagram for a manufacturing assembly line, we could each role might have a direct dependency only on the one immediately preceding it in the process. But in software, there are essential, ongoing dependencies that form an almost fully connected network of dependencies between roles.

Many software organizations handle these dependencies in an ad-hoc way, which is more or less one at a time. If the architect is sitting at his or her desk writing the Big Architecture Document, and if he or she needs the insight of the interaction designer before proceeding, too often the information request must go “through channels.” Such interactions usually draw many non-producer roles into the process, and that puts the architect into a wait state. If the architect is waiting, so are the GUI designers, the coders, the customers, and the end users. In the very best case, the answer will come back to the architect in days or weeks and is still a useful piece of information that hasn’t been invalidated by changes in the market, standards, technology, or development team staffing. More typically, the response raises as many questions as it provides answers (knowing the interaction designer’s recommendation, do we need to ask the coder if we can implement it?). Unfortunately, the real scenario is that the architect makes an assumption or guesses simply because it’s too wasteful of time to clarify the details. And that means that when the Big Architecture Document is unrolled to the interaction designer and coder, there will be much wailing and gnashing of teeth—and another big cycle of rework and waste.

So by trying to do the right thing in an assembly-line organization, an architect will cause delay. By failing to do the right thing but instead taking all the decisions upon himself or herself, the architect incurs even more delay. This is why architecture development takes months or years in linearly organized complex projects. (These are called “NASA-type phased program planning (PPP) systems” in [TokNon1986].) It isn’t that architecture is so much work; it’s that everybody spends so much time waiting while Emails sit languish in-boxes, while architects write architecture documents, or unread memos sit awaiting review.

A good team that develops relationships between the roles—relationships that correspond to the dependencies between stakeholders—can trim the architecture effort from months down to days or weeks.

Organize more like an insect swarm than as stovepipes. We’re writing this chapter from the middle of the Swedish Northwoods. Yesterday we took a walk in the forest and passed several anthills. The largest one was more than a meter high and more than two meters in diameter, and every millimeter of its surface was alive with scurrying ants. We couldn’t find a single project manager among them, nor a single process description. And we didn’t see anyone in an architectural wait state.

If you’re using Scrum, try to fit your architecture exercise into a single sprint. Who is the team? It’s the cross-functional Scrum team. If you’re in a multi-team Scrum project, you’ll need input from multiple teams.

Your team members should be collocated so they can respond to questions in seconds rather than hours, days, or weeks. What does “team” mean here? It comprises at least those roles described in this chapter. Too often, Agile initiatives limit Agile principles and practices to developers, perhaps with a token on-site customer thrown in. Scrutinizing roles and their take-holding relationships more carefully shows that things are more complicated than that.

1.3.2      The first thing you build

Brad Appleton is an old colleague of mine from the Pattern Community and is a long-time respected person of influence at Motorola. His E-mail byline has consistently said for years: “The first thing you build is trust.”

Jerry Weinberg tells a story of a company where a highly placed, powerful manager issued an urgent request for new computing equipment. The requirements were a little bit sketchy, but he did insist that “the cabinets had to be blue.” His subordinates, and purchasing, and others, scurried around trying to decode this supposedly mysterious message. “Blue! Does he mean to buy from ‘big Blue’ (IBM)?” “Does he mean that he wants it the same color as the other equipment? But some of it isn’t blue!” Someone finally got the nerve to ask him (whether it was actually before or after the equipment arrived, I don’t remember) and he said, “No, blue is my wife’s favorite color, and she thought that the new computers should be blue.”

One exercise in Lean is called “ask five times.” If someone makes an unjustified claim, ask them about it. More often than not you’ll get another unjustified claim. Within about five exchanges you’ll come to the core of it. Why didn’t anyone ask the executive why he wanted blue? It was perhaps out of fear of being an idiot for not knowing the answer. Or it was perhaps it was out of fear of potentially embarrassing the boss in public. There wasn’t enough trust in the organization to clarify the requirements.

Jerry talks about egoless development teams—a commonly misunderstood phrase that simply means that you put your personal stake in perspective so you can defer to the team’s stake as a whole. We’ve all heard the saw: “There are no stupid questions here,” but we are not always good at following it. We should be. An Agile team is a team of trust that can ask such questions openly.

1.3.3      Keep the Team Together

As stakeholders, team members have expectations, too. In addition to their value stream expectations, they come to expect certain abilities, reactions, and ways of working from each other. Knowledge of such expectations, like most expectations, is tacit knowledge and takes time to develop.

To support the team in the ever-ongoing learning of how to become a team and become a better team, keep the team together over time. If you re-assemble teams for every new product or on a periodic business cycle, you force each team into the well-known cycle of forming, storming, and norming, before reaching a performing stage. That’s waste. Get rid of it.

Teamwork works on the scale of milliseconds. Just watch a football team. Or, better, yet, watch a software team engaged in a design meeting. It’s exactly this kind of feedback that can reduce architecture efforts from months to days by displacing formal communication channels and forums. If a team is not co-located, you lose these feedback loops. To sustain team effectiveness, keep the team together in space. A multi-site team can work but will have difficulty sustaining the same pace as a collocated team, everything else being equal. Martin Fowler writes [Fow2006] that multi-site development requires more written documentation and, in general, more formal communication styles.

There are many variations of Conway’s Law that provide guidance for organizing teams. The primary organizing principle is that the team structure should reflect the architecture. However, even that is difficult, because architectures themselves have cross-cutting concerns. In the DCI architecture (CHAPTER 8), the structure of roles and their interactions cuts across the structure of the domain objects. And beyond this simple part of Conway’s Law, you also want the organizational structure to align with your market structure. You also want it to align with the physical distribution of people. You also want it to align with the structure of the business. Figuring out exactly how to structure a team means balancing the tradeoffs that emphasize different ones of these organizations.

For completeness, one organizational structure that we know does not work is to isolate all the architects in their own team.

No matter how you organize it’s important to keep the boundaries between the teams thin. Any work on the architecture must cut across organizational boundaries.

It’s hard. You’re Agile. We trust you. You’ll figure it out.

All material is Copyright © 2010 by Gertrud & Cope and John Wiley. All rights reserved.