Building Process Scalability into Software

We often associate scalability with system response time. There is another aspect of scalability that is often ignored until too late. Ironically, it is perhaps one of the easiest aspects of scalability to address before any code is written. I'm referring to scalability of the software to support large, complex processes that touch many people.

Here are two examples:

• Adding large numbers of something to your system at one time. For example, users, payments, or products.
• Making administrative changes across many items. For example, changing permissions for many users, adjusting process to many items, or applying payments to many customer accounts.

I've noticed most software appears to be designed for entry of one new user, changing one product, and so on. This works well most of the time. However, two things usually happen:

1. A customer grows so large that updates happen frequently enough to start impacting productivity. For example, posting a customer payment by hand may be reasonable if you get a few payments a day. What happens for a multiple location business with centralized billing and dozens or even hundreds of payments?
2. Periodic bulk changes consume very large amounts of time and effort for a short period of time. New laws and across the board price adjustments are examples of things that may occur only once or twice a year, but have to be done for many, many items in a very short period of time.

Scalability in this context means scalability of process. It may be fine that it takes 30 seconds to increase the price of one product by 5%. But what if I need to increase the price for 10,000 products? You are going to have to provide me an efficient way to do this.

Unlike hunting down slow performing algorithms, you have to apply user experience principles. Identify operations your users perform to large numbers of items. There are three questions that arise from this:

1. Who are your users and what are they trying to accomplish? From here, identify the most critical operations that are performed frequently.
2. What does “perform frequently” mean to them (not you). Something that is performed once daily, but takes 30 minutes may be frequent to your user. This is especially true if it is something that they will likely do more frequently if their business grows.
3. How big is "large numbers"? For some things, 3-5 times may be a large number if each time it take 2-3 minutes. In other cases, it may be hundreds of times, even if each time takes a few seconds.
   

Seconds add up
Saving one person 2 hours per task, when it is performed 10 times per week, is easy to understand. Be careful to not overlook an opportunity to save 30 seconds of a 2 minute operation when 10,000 people perform it 2 times a week. I've used somewhat extreme examples to make a point - not to set a standard. The core point is that saving time from seemingly small operations for many users can provide a very real net benefit to your customer over time. Don't overlook those opportunities, even if the savings seems small initially.

These lessons can be applied very early in the development cycle - well before any code has been written. The observations made through interacting with the users allow you to focus energy on making your customers efficient with your software, even if that means slightly fewer features.

Keep in mind - you are likely to be better off with fewer usable features than a lot of features that are inefficient. And let's not fool ourselves - once a feature is released we seldom go back and improve it proactively. Take the time to build in support for scalable processes along with all the other "ilities."

Hire for attitude

I was chatting with someone the other day (as opposed to chatting with myself - but that is another story...) about the state of talent in our industry. I relayed to him a quote I learned from Michelle Jackman (http://www.mjbizwiz.com/) many years ago: Hire for attitude and train for skills.

Given that I heard this from Michelle in 1988, it is among the most impressionable things I’ve learned in my career. Along the way, I’ve discovered a few things that help me assess a potential team member’s attitude. This is not an exhaustive list, and I welcome your thoughts.

Focus on principles, not tools
The best candidates (and subsequently, hires) have been the ones with whom I’ve had a conversation about software engineering fundamentals and principles. How do they view design? How do they view build management? Does the art and discipline of software development excite them? If none of this matters to them, I don’t care if they know languages and data structures.

You may be asking, “How do you know if they actually have skills, though? These sound like philosophical conversations.” You are correct - they are about philosophy. However, if you dig deeper, you’ll realize the candidate has developed their philosophy through their experience and learning. A candidate can’t talk to me intelligently about multi-branch, concurrent release management unless she has dealt with it in her career. I don’t necessarily care if she has used CVS, Subversion, Jazz, or ClearCase. What I care about is that she experienced the challenges with this and has developed principles about release management that are consistent with (or will add to) mine.

In addition to software engineering principles, a person must love to learn. I ask things like what websites and journals they subscribe to. I ask them what the last couple of books are that they’ve read. Candidates that come up blank are not asked back. If the candidate shows no interest in learning on their own, they aren’t going to be fit on my team.

Focus on customers, not process
I want to know if the candidate wants to be left alone to code or understands that we build software for people. If the candidate shows little to no interest in the customer, they don’t get on my team, regardless of talent.

I had a key senior architect who I wanted to accompany me on a customer visit. This was a key customer, with a team of highly technical staff. My guidance to my architect consisted of something like, “We are visiting customer ABC, to discuss our foo-bar capability. They are going to have several of their technical folks at the meeting. I want you to discuss our architecture with them. Take care; I’m on the road for the next couple of weeks. See you at the customer.”

He prepared customer friendly architecture drawings. He kept me posted on progress and asked my thoughts about the overall level of discussion and breadth of topics. In other words, he demonstrated the attitude that customers are important, and that we need to be prepared when meeting with them. That attitude, by the way, was infused into his designs.

His response was exactly what I expected based on our conversations during the hiring process. My faith in him was not drawn from his technical ability (although it was very high). It was drawn from knowing his attitude about software engineering and customers.

Concluding example

I once was approached by a person who wanted to join my team. He was a developer earlier in his career and then a manager for several years. He realized he was happier as a developer and saw an opening on our team. On the surface, there was no reason to hire him. He hadn’t developed in years and was pretty rusty with his technical skills.

Since this was an internal candidate, I took him to lunch as a courtesy. In the course of the conversation, I was impressed with his work ethic, empathy for the challenges of management, and his strong customer-centric approach to software development. Late in the conversation, I told him flatly that I loved his attitude, but had concerns about his ability to develop after being out of it for many years.

He assured me he would spend his own time re-learning languages and environments. This was one of those “look a person in the eyes and judge whether he is sincere or not” moments. I judged him to be sincere. He went through the rest of the interview process, and we agreed to hire him, strictly on attitude. After all, how often do you get a chance to bring someone with management experience and a customer focus onto your development team?

Results? He became one of our most trusted developers. He won company awards and frequently was asked to personally address issues with top customer accounts. His attitude of self-motivation, customer focus, and results were the keys to his success. Yes – I could have hired a person with a couple years of programming experience that on paper would have “coded circles around him.” But the results would not have come close to what this person delivered in his years working for me.

Think about how you acquire talent. Do you engage candidates in meaning dialogue about learning, customers, and principles of software engineering?

Help Sales Sell

I recently attended my company’s global sales kickoff. It afforded me an opportunity to observe our products from a different perspective. From a developer’s perspective, a product is a sense of personal accomplishment and a source of pride. From a customer’s perspective, a product is a way to increase pleasure, productivity, or revenue.

What is a product from a salesperson’s perspective?

To a sales person, a product represents their livelihood. If there is no product, there is nothing to sell. If there is no sale, there is no commission. Let me be clear here – there is absolutely nothing wrong with selling to make a living. That would be as silly as asserting that “all developers want to do is get paid to write software.” My point is that being able to sell a product is more than just pride – it is what pays their bills. And, in case you haven’t figured it out – your bills, too.

Luckily, these perspectives are not mutually exclusive. I’ve written quite a bit in this blog about why customers matter. I’d like to turn my attention to our sales colleagues. I’ll start with what should be an obvious statement: sales people pay our salary. That alone ought to be enough to look at sales in a different light.

It is a running joke in software organizations that development and sales are destined to be at odds with one another. I admit that I’ve bought into this belief to various degrees through the years. I’m starting to see things differently. Being surrounded by hundred of sales people for three days provided me a new view into their world.

Sales people don’t really want to sell. They want to win. And how do they define winning? It is amazingly simple. They win when a prospective customers chooses your product instead of someone else’s. It is that black and white. So here is my question to you.

What do you do to help sales win (i.e., make it easy for prospective customers to choose your product)?

Take a moment and write down a few things that you’ve done recently to help sales close deals. And don’t give me a line like “that is why we have marketing”.

Here are a few questions to jump start you:

• When you build a feature, do you know the business problem you are solving for your customers?
  If you don’t, why are you building the feature? If you don’t know, how can you expect a sales person to sell it? It is easy to use the excuse “product management should be telling me to build stuff that can be sold.” That is weak. I don’t accept this from my managers (or my developers, or my QA, or my designers, or my… well, you get the point). Don’t accept this from yourself or your managers, either.
  
  You have an obligation to understand why you build a product feature. You have a responsibility to develop it in a way that makes it easy to sell. Get close to your customers and find out what problem they really want solved. You then have a clear solution (and story) to equip your sales team with.

• Do you spend time with sales while they are selling?
  This is an area many development leaders shy away from at best, and at worst consider to be beneath them. As long as these folks are working for my competitors, I’m thrilled. My advantage is being on site, hearing objections faced by the sales person, and listening to problems faced by our prospects.
  
  When I return from the visits, I share with my development team so they understand the “why” of building our software. I share my responses to prospect’s questions with the rest of the sales team to make them more effective. I seek out input from my current customers to see if they are also facing these problems. This provides me greater understanding as new features are prioritized and designed. That generally leads to very high customer satisfaction with new features.

• Do you provide information that is helpful to sales?
  As I mentioned above, it is easy to fall into a trap that this is some other group’s responsibility. In general, it is true that product management or product marketing commercializes a product. However, you still can produce powerful sales aids. If you receive the same question from multiple sales people, write an FAQ about the topic and distribute it. If you hear objections from prospective customers in multiple deals, write a response to the objection for sales to use.
  
  Communicate in a manner that a sales person and prospects can understand. You might think it is great that you use Microsoft SQL Server Integration Services to extract, transform, and load data from your OLTP database to your data warehouse as a nightly job running in the background so it consumes non-peak processing resources. That means next to nothing in a sales engagement. What matters is that your product has a high-performance and flexible business intelligence solution.
  
  This isn’t about intelligence – it’s about language and experience. I work in the healthcare IT industry. As one of my nurses once quipped, no one ever called for a software engineer when they were dying.

• Do you ask sales people what you can do to help them?
  I think I will let this question stand on it’s own…

Review your relationship with sales. It's worth it.

Harsh start-ups and smooth landings

I recently participated in an end of release cycle product demonstration. It was a wonderful experience and I was proud of the team’s achievement. It’s a great feature that will make a significant impact to our customers and to our business.

During the product demonstration, I couldn’t help but reflect back on the first several sprint demonstrations. Those were much different. Whole screens were thrown to the scrap pile, basic flows were challenged, and even seemingly simple functionality caused excitement among the stakeholders.

As the sprints went along, the conversations deepened, the team solidified, and the product began to mature. With each sprint, the discussions evolved away from major issues and towards minor suggestions and tweaks. During the last product demonstration, the feedback was positive and only a few cosmetic changes were suggested.

The technical lead remarked that we were doing pretty well if that was all the feedback we received. I had to agree. On my way out of the room, I remarked that we had experienced a “harsh start-up and a smooth landing.” Walking back to my office, I started thinking more about the factors that led to this point in the project.

Here are a few ideas for you to consider:

• Trust
  If a feature isn’t working, is confusing, or just plain wrong, anyone on the team or in the stakeholder community has an obligation to say so. This may appear to be an obvious truth, but it doesn’t always happen.
  
  Someone assumes another will catch it. Someone feels that no one will listen. The list of reasons for remaining silent goes on and on. As a leader, you can create an environment where people will stand up and use their voice. Make it clear that all feedback provided (respectfully) is welcome. People provide feedback if they trust that they will be heard, and that they will not be judged.
  
  In short, this trust is what leads to honesty. You have a vital part in making this happen.
  
  
• My product, your idea
  Our sprint teams welcome input from all sources: customers, internal stakeholders, and each team member. I’ve seen QA folks make design recommendations, customers provide workflow suggestions, and internal groups offer changes to improve service and supportability. The teams absorb the input willingly. They do so because they know it ultimately leads to a better result.
  
  They aren’t interested in whose idea anything is. They are interested in producing a quality product that delights our customers and helps the business. This ego-less attitude has to come from you – their leader.
  
  Praise the team each time they incorporate a suggestion from outside the team. Help them understand their contribution to a culture that promotes others to contribute. Encourage them and reward them when they respond to input from others. Their job is to produce a great product – not pretend to know everything. That often means listening to others.
  
  
• Act on feedback
  Do you want to encourage feedback? Then act on it when you receive it. At our demonstrations, I often hear the team place suggestions (and criticisms) they receive into their sprint backlogs. They then prioritize the feedback accordingly. The person who provided the input knows immediately that they are heard, and that the team and the business are going to treat their input with respect.
  
  Make sure everyone knows that not everything raised will get into the release, or even into the backlog. On the other hand, not everything is ignored. It is critical that each person understand the rationale for why a suggestion does or does not make it.
  
  Equally important is assuring the team that all feedback is regarded equally, regardless of the source. I have been told on more than one occasion that my input will go into the backlog – not immediately addressed just because I’m the VP. And that’s OK. I should have to justify a request just as much as any other stakeholder.
  

In the end, build a culture that honors feedback, regardless of where it originates, and does something with it. You will also likely see the degree of ownership among your stakeholders skyrocket. They will be transformed from talking about “the” product to talking about “our” product.

I'm in love!

Yes, that's right. I'm in love. Oh, I've known it for some time, but I don't think I was ready to admit it. So, here I am, ready to say it to the world.

I love my customers.

OK, you may be thinking that I've lost my mind. Or, you may be wondering what the big deal is. After all, aren't we supposed to love our customers? Of course we are. But do we?

All right, let me explain.

You see, I'm privileged to work in a truly meaningful industry. As a software vendor to Home Health and Hospice providers, I have a chance to impact clinicians who are doing deeply important work. The patients my customers serve are among the sickest and most vulnerable among us. They are often faced with end of life illnesses or chronic diseases.

The choices my team and I make impact whether a clinician stumbles through a maze of screens and widgets, or seamlessly uses technology to care for a patient. My customers are real to me and to my team. When we start a project, we think of the customers that will benefit. I don't mean personas. I mean real people with real names. As we solicit input, we talk to them, share our thoughts with them, and listen to their feedback. We treat them as first-class team members - people who know what they need to get done and why.

Again, I know this isn't exactly earth-shattering. But the affection my team and I have for our customers matters. It's what drives us to spend the time and energy to get a feature from "useful" to "delightful." Affection wasn't obtained overnight. It has been cultivated through reaching out to and building relationships with our customers.

This effort pays off. I often stop by the desk of someone working late and ask them what is going on. And more often then not, the response is something like "I spoke to Sally of such and such Hospice, and she had this great feedback I wanted to mock up for the developers to get into the next sprint." You see, a User Story can wait until the next day. Helping Sally happens now.

I hope my customers know how much I care for them, and respect what they do. When things go bump in the night, sure they get frustrated and sometimes even angry. But so far, after all these years and the occasional setback, they have always trusted that I care about them and their patients. At the end of a sometimes long call, they tell me that they know my team will do whatever is necessary to allow them to do what matters: care for patients.

And when they are delighted by the software or service provided, they are equally eager to share. I guess what it comes down to is that after we look at markets and segments, personas and users, eventually you have to realize that on the other side of every decision you make is a real person.

Burden? Or Purpose?

I was driving the other day, feeling the weight of the world on my shoulders. Typical "woe is me" stuff. You know, feeling as though everything depended on me, and wondering whether this burden would ever end. Poor me.

Out of nowhere, something flashed into my mind. What if this isn't my burden?

What if it is my purpose?

What if every experience I've had, every part of my current life, every person who looks to me for assistance and guidance is here because that is the way things are supposed to be for me? I began to understand that I am here to be part of something, not to get away from everything.

This revelation was fascinating to me, since I've been in leadership positions for nearly my entire career. I never thought of this as a burden, although for years I've felt tremendous pressure to be successful at helping my teams and my companies achieve great things. Recently, I recognized that I actually was considering this a burden - something I had to endure.

The subtle shift in my frame of reference was a like a jolt of energy. I understood I'm here for a reason (or perhaps I was remembering something I once knew). That reason is to help others accomplish great things, and even many "good enough" things. I was given certain strengths - gifts, if you will - and it's up to me to use them.

That is at the core of what leaders ought to be doing.

Do you think of your team and your customers as burdens? Or are they an opportunity to explore and utilize your strengths?

Are they a source of growth for you? Or an energy drain?

They can be any one of these. It's your choice.

Building On-line Development Communities

I've been thinking about how I could use social networking to strengthen the sense of community on my product development team. As is the case with most things, this is not a new idea. Open Source Software development has practiced this and embraced community development for years. In fact, one often hears references to the "open Source Community".

My team already has a strong sense of identity and community. But there was (at least) one area I wanted to improve. We didn't really have a repository of our history. Like most software shops, our "history" resides in our e-mail and file systems. That's nice, but it is not easy to find or share. As long as you know the right person to ask for a copy of an e-mail thread, you are fine. Even then, you get whatever thread fragment that person deemed important to save.

There are two distinct technologies I have deployed that have impacted our ability to record history. The first is Discussion Forums; the second is wiki pages. Again, there is nothing new here. However, very few teams I talk to use these technologies. I'm assuming you can figure out how to get these technologies installed and available. I will focus on what might help your
team adopt these technologies.

Start with yourself and your management team

Before rolling out to your entire team, start with your leadership team. Create private discussions and Wiki pages for the leadership team. Seed the topics and encourage your leaders to provide their input using discussion forums and wiki pages. Redirect normal conversations to the new technologies. For example, if someone sends an e-mail to the leadership team, copy it into a discussion forum and reply to that.

Begin to post key information in discussion forums and wiki pages. Make your leadership team accountable for knowing the content by giving them no alternative to find out what you are asking of them. Once you've done this for a few weeks, deploy to the next level.

Deploy to functional teams

Next, require each of your managers to deploy to their teams. Here I assume you have functional managers, such as development, quality assurance, writers, etc. Resist the urge to post into the functional areas. However - and I consider this critical - monitor activity for a few months.

Your goal in "eavesdropping" is to ensure the teams are using the technologies to build communities within the functional areas. If you don't see much activity, discuss it with the functional manager - not with the team. If you do see a community building, by all means praise it publicly. Make it clear you are aware of what is going on. That is perhaps the single best way to make it obvious that this is important to you.

Deploy to project teams

Once folks have gotten used to on-line functional communities, drive adopt to project teams. These are cross-functional teams working on the same release or feature. These are smaller communities that are transient, just for the life of the release or feature.

The key here is to build a community around people trying to solve the same problem. This is also where you build a lot of product history. The discussion forums record the conversation among the team (for example, alternatives for technical design strategies); the wiki pages record the actual decision (final design decision and rationale).

This is not easy, but if your team trusts you and you show leadership, you can be on your way to building deeper on-line communities in a matter of months. I'd love to hear your experiences.