Big rewrites: Summing things up

My last several posts dealt with migrating technologies, people, and skills considerations. Certainly, a few posts can't do justice to something as complex as a system technology refresh.  On the other hand, there are a few principles that can be helpful guides as you navigate the the complexity.

• Iterate
  If possible, break the rewrite into smaller chunks.  Determine the smallest amount of business value you can deliver and start there.   This allows you to stay focused on software delivery fundamentals and to use the new technology in the real world.  Release management, branching and merging, and patching are just a few of the processes I've seen atrophy after years of rewriting software without a delivery.
• People
  A blended team of existing folks and new hires is desired, but comes with issues that require supportive, yet firm, management. The technology transition is going to happen - that is where you must remain firm.    It is not reasonable to expect your existing team to learn a new technology stack in the context of significant business implications.  You will have acquire skills and that will cause friction.  These new folks have to be integrated into a new, combined team.
• Skills
  The existing team members are a key to success due to expertise of the system, customers, and industry.  Your commitment to their education, skills, and to be blunt - employment, is the supportive aspect. Show them how they will participate in the transition itself, as well as their contribution in the new world after the change.  Provide them a picture of their role after the new technology is in place to assure them that they aren't helping the "new folks" build the system that is putting them out of a job.

These principles provide you a start at identifying the core issues you will face regardless of technology or industry.  For example, iteration is simple in theory.  In practice you have to address factors such as, "What is the least meaningful value I can ship to the market?", "How can I deliver 'less than everything' alongside existing software?", and "How long do I really have in the market with the existing product?"

The questions that follow from the principles are the real power.  Conversations surrounding the questions and their answers will do more to increase project success than all the project management during the project itself.  After all, if you don't have the right skills, your team is demoralized, and the project is a multi-year science project, your project status reports are going to be the least of your worries.

System migration and people: Part 2

Technologists often align themselves to the skills they possess.  You see it in job titles: Senior .NET developer; Java architect, and so on.  In the midst of a migration to a new technology stack, you are changing more than a person's business card.  You are changing who they perceive themselves to be.

This presents a fundamental challenge to large-scale technology migration. And one of the greatest ironies. 

Possession of the required new skills is the least important contribution from your existing team.  Yes - you read that right.  During this transition, what makes your existing team so valuable is not their technical skills.  So what does?

• Domain knowledge
  No matter your business, there is specialized domain knowledge.  Mine is healthcare.  That means deep understanding of patient registration, ordering medications, patient care documentation, and scores of other complex workflows.  These don't change with new technology.  Your team has spent years learning these and the interplay among them.
• Customer understanding
  Your team understands how your customers operate at the intersection of the current product and the domain.  You can't do a migration without some consideration for your customer base.  Your team has customer relationships and trust (hopefully) to increase the chances of the transition going smoothly.
• Organizational knowledge
  As I wrote last time, you may supplement your team with expertise in the new technology. These folks aren't going to have a clue about anything related to the items above.  Your team can help them navigate the challenges of finding information about the existing system and processes - or even who knows the answer in the first place.

Returning to the challenge - changing who a person perceives themselves to be.  Your goal is to shift that perception.  You have to state (and restate) that technology is fleeting - and that it can be learned.  The emphasis for the existing team is on their knowledge of domain, customer, and organization.  That can't be extinguished by Java 8, HTML 5, or any other technology.

System migration and people: Part 1

My last post discussed migrating to a new technology stack as part of extending the life of a software system.  Next, I'll explore people considerations.  The first topic is skills related to the technology stack.  My next post will discuss the role of domain experience.  Finally, I'll attempt to tie it all together with a summary post.

Chances are your current team possesses skills to evolve and maintain the existing system. Unfortunately, these may not be the technical skills needed by the new technology stack. This may be a source of fear among the team. While there are no easy solutions, here are some thoughts:

• Get expertise in the new technology stack
  The most effective way I've seen this done is to hire additional people to supplement the team.   Find people who know the new technology well and have demonstrated its use in bringing to market solutions that have some matching characteristics to your product.
  
  You need people with experience, otherwise you will end up with the old solution implemented in new technology - not at all what you intend.Your goal is to quickly apply best practices with the new technology.  Consultants have their place, but in this situation I find that nothing beats employees who are fully dedicated to the project.

• Commit to trainingFor those who want to, provide training in the new technology. It's OK if some want to stay with the older stack.  it will be around for a while and you will need to maintain it.  However, show people that you are willing to invest in their future and they will be more likely to stay with you through the transition.
  In addition to training, integrate new employees with existing teams to increase the pace of learning.  This will create one team, demonstrate by example how to apply the new tech stack, and allow new employees exposure to existing rationale, business rules, and architecture.

Understand that for some, they will equate their value to their existing skills.  My next post will address this directly.

Migrating technology

I've been thinking a lot about extending the life of a software system.   There are several hard problems to solve.  The first problem I want to discuss is migrating to a new technology stack.  I'm skipping the discussions that take place before the decision to transition to a different technical stack are made.  Whether that decision is good or bad, if it has been made it is your job to get it done.

Consideration #0 - Don't do it all at once

Big rewrites are very likely to fail. So don't try them.  Learn from our collective Agile experience and establish a backlog for your migration.  You will learn a lot from planning the backlog, such as business and sales pain points, architecture dependencies, and areas that don't require immediate attention.  This gives you the advantage of putting new technology into the market faster, further accelerating your learning feedback cycle.

Consideration #1 - Build new features in the new technical stack

While straightforward to understand, this is perhaps the most difficult to implement.  Your end user work flows have to be (relatively) seamless.  The existing system was designed to support a collection of end user goals and workflows.  If the new feature is presented is a jarring manner, you risk putting your users off.

A few years back, my team was challenged to move from Delphi to .NET.  We found ways to render .NET screens inside Delphi containers.  Although we used different component libraries, an end user had to be extremely astute to notice subtle control rendering variations.  A more recent experience involving more disparate technologies has required some visual redesign of the existing application to reduce the perception gap.

Regardless of your technologies, you have to make the user experience transition as seamless as possible.  

Consideration #2 - Rewrite parts of the existing system when appropriate

As your existing system requires modifications, determine if the feature to modify is a good candidate for being rewritten.  Again going back a few years, we were faced with a major overhaul to government reimbursement for Home Health services.

Instead of modifying the existing (tried and true Delphi) payment system, we left it alone.  We wrote the new billing engine in .NET.  This allowed us to guarantee no regression to existing billing and to add significant capability in the new system.

Consideration #3 - Accept that some stuff will remain around for a very long time, maybe forever

This can be hard to accept.  Engineering, and I suppose business, is about choices.  Those choices involve what you do and what you don't do.  There are parts of your system that work, are reliable, and aren't causing you lost sales.  Before you rewrite those pieces, have a compelling reason to do so.

In general, your goal is to devise a scheme to migrate to the new technology over a reasonable period of time, incrementally, with little impact to your end users.  I know this is hard.  But it is possible.  The alternative - a massive all-or-nothing rewrite - has proven again and again to lead to failure.

Understand old solutions to solve new problems

In my last post, I ended with a comment about the importance of understanding solutions in order to solve a broader range of problems.  Problems have an essence.  If you understand the essence of a problem, you can draw from solutions for that class of problems.

Consider the problem of managing vaccination history for a patient.  Take a moment to enumerate the essential characteristics of this problem. Can you think of systems you have seen or used that address problems with similar characteristics?

What would you say if I told you that tracking vaccinations is a very similar problem to your vehicle’s recommended maintenance schedule?  If you don’t believe me, next time you get your oil changed, ask to take a look at the screen the service manager uses to upsell you on a new air filter.  Replace each recommended service with a vaccination and you should see what I mean.

Experience with and exposure to varied solutions is important.  However, it is not enough.  You must internalize the relationship to the problems they solve and the essence of those problems.

Here is an approach that may help:

1. When you use tools (physical or software), identify the problem the tool solves.  Ignore the features – focus on the problems.  For example, don’t think of a hammer as something to “Drive a nail into a wall.”  Think “Deliver force to an object.”  
2. When solving a problem, tease out its essence.  Find solutions that solve problems sharing essential characteristics.  For example, replacing the lid on a paint can requires the use of directed force to seal the canister.

Go out and build an inventory of solutions and the problems they solve.  The larger your solution inventory, the broader the range of problems you will be able to solve.

Drive

I recently received an e-mail relating how certain programming constructs look elegant, but turn out to be grossly inefficient.  That got thinking about abstraction frameworks and how they increase perceived development speed, yet can have unintended consequences (performance being a big one).  That, in turn, got me thinking about why the guys on the thread could immediately spot the underlying inefficiency through visual inspection.

Are they inherently smarter? Maybe.

Did they screw up a similar project and learn? Possibly.

Are they driven to understand how things work at a fundamental level? Check.

What separates these guys from the huddled masses is an innate drive to understand their world (in this case, software engineering) down to the ones and zeros.  I’ve sat next to both of them while they code.  They have debuggers, output windows, browsers inspectors, and SQL execution plans running well after typical developers would have committed the code.

They have a drive to know their code is correct, fast, and readable.  They dissect it, improve it, and test it.  If someone is going to break their code, it will damn sure be them, not someone else.
But you protest.  Surely, they must not understand deadlines!  Clearly, they take longer!

You protest in vain.  These guys are among the most productive guys I’ve worked with.  Their code is clean before they release it.  There is rarely rework.  Because they have spent so much time understanding their tools, they avoid mistakes made by developers who focus on getting it done and checked-in.

Their breadth of understanding allows them to quickly select appropriate constructs and patterns based on the problem.  Contrast this with developers forced to solve problems based on what they know versus the nature of the problem itself.  More importantly, their depth of knowledge allows them to know why their solution will work and what its limitations are.  Most developers won’t fully understand why their solution works. Or what problems it can’t solve.

I’ll leave things here.  I plan to return to this last paragraph in a future post.

Free to be crazy

I’ve been thinking about why I am comfortable throwing out crazy ideas to my team. I don’t think it is solely my lack of concern about appearing foolish (although there is an element of that).

I’ve repeatedly told my team – everyone, not just managers – that one of their most important jobs is to keep me from making bad decisions. I need to know that I am going to get honest feedback from everyone. This gives me freedom to come up with crazy ideas without fear they will be accepted simply because they came from me.

How do you get the team to a point where everyone believes they can safely share feedback with you?

• Reinforce the need to know the truth from your staff. Encourage people to challenge you. They may not do this initially. After all, you are the boss.
• When you propose an idea, actively solicit input in public settings: “What do you think?” “Is there something I am missing?” “What would you do?” It will take time. If you keep asking, people will start to use their voice.
• Once people find their voice, listen. The ensuing dialogue is the reinforcement needed to move your team into a positive feedback cycle. Use this time to explore your idea, tear it apart, defend it (without being defensive), and find viable alternatives.

I have found that most of the time there are only a few outcomes (in descending order of occurrence):

• I realize I am crazy and I’m grateful that we didn’t act on my idea.
• A kernel of goodness is contained in my idea and the interaction results in a much better idea.
• I actually had a pretty good idea after all.

Remember, you aren’t trying to be right. You are trying to get to the best decision for the situation. Your job isn’t to be right – it is to make sure the right decisions are made by the team.

One last consideration. If you never accept the feedback, then you have two problems. Number one, your team will stop offering it. And two, you aren’t really interested in hearing it – which is why number one happens.

So go out and be crazy. Something good is sure to happen.