Selecting a Ranking Method for Your Project Portfolio

One of the most difficult parts of project portfolio management is deciding how to rank the projects — that is, determining which projects should be done now, later, and, most importantly, never. There are several ways to rank a project portfolio. Each is useful in specific situations and not so useful in others. But all share the same goal; namely, arriving at a single ranked list of projects.

Dave, a CIO, gathered his directors together. “OK, everyone, listen up. We have too much to do. Everyone’s multitasking, which means we’re not getting anything done. Our projects are still late. We need to decide what projects we’re going to do first, second, and third, and then stick with that, at least for a while.” His directors looked at him as if he had three heads.

“I’m serious. We’ll rank our projects and see which ones are most valuable, do those, finish them, and then go on to the next project.”

“How do you propose we do that?” asked Tanya, a director. “We don’t know what’s most important to you or to the rest of the senior management team. In fact, we keep getting different messages. First, my calendar integration project is first, then it’s the data integration project. Once last week, it was the performance project — but only for a day. I’m quite happy to rank the projects, but how long do we think that ranking will last, and how can we do it?”

Dave frowned. “Well, we’ve got a chance to make some decisions, show how we made them, and make them stick just for a few weeks. That will give us a chance to finish some work and rerank, if we have to.”

You probably share Dave’s problem. You have too many projects to do at the same time, not enough people, and not enough projects finishing. If that’s
the case, it’s time to consider ranking the projects your organization has. In this article, we will explore a number of approaches to ranking projects:

  • Assigning projects a point value
  • Assessing the risk of a project
  • Reviewing the project context
  • Using double or single elimination
  • Ranking a project according to your organization’s mission and values

So let’s get started. First, list all your active projects —that’s your project backlog. Now it’s time to start asking the difficult questions.

ASK THIS QUESTION FIRST

Once you start ranking the projects, make sure you ask this question first:

Should this project be done at all?[1]

If there’s no sponsor, no customer, and especially no identifiable value for the project, stop it.

Don’t be afraid to not do projects at all, and especially for now. Project portfolio management is about making choices that guide the technical staff to finishing work on just the most important projects in the organization. If you can’t make a compelling argument for finishing this project, put the project back on the project backlog for now. Or add it to a parking lot[2] where you list the projects that are not under active consideration but that you don’t want to forget about.

More likely, the answer to the “Should we do this project?” question will be “yes,” and you will need to proceed to ranking your various projects. Ranking requires discussion among the sponsor and the customer or customer surrogate, as well as some description of project value.

RANKING WITH POINTS

Project value is the value of the project to the organization. If you could rank the projects with an ordinal ranking — as in, 1, 2, 3, and so on — your work would be done. But you often need more conversation than mere ordinal ranking affords. Points help you articulate the business value and provide a visible means of showing the relative business value of each project.

When you rank with points, you assign a number of points to a project, allocating more points to more valuable   projects. Points represent business value. If you wanted, you could use dollars (or euros, or what have you), as in “How many dollars do we want to invest in this project?” Sometimes, that’s difficult to do if you aren’t sure how long the project will take or don’t know enough about the features. When you’re evaluating the portfolio, you may not have thought through all those issues yet. Using points as a way to rank projects separates the project funding question from the business value decision.

Separating project sizing from duration helps project team members make better estimates. In the same way, separating business value from funding allows managers to see what output they desire from the organization. If all the projects have a small number of points relative to each other, then no one cares much about the set of projects.

When you rank with points, you take a large number of points, much larger than the total number of projects. For example, if you have eight projects, take 10,000 points to start. Now assign a unique number of points to each project. If you have two very important projects, you cannot assign each of them 5,000 points. You can assign 5,001 points to one project, and 4,999 points to the other project. Yes, both projects are very important, more important than anything else you have in your portfolio. However, allotting 5,001 points to the first project says, “This project is the most important one.” The 4,999-point project is next in line.

By assigning a unique number to each project, you show the organization which project is most important for now. If you allocate the same number of points to more than one project, the technical staff doesn’t know which project is most important, and you don’t have a project ranking. It’s too easy for each person to make his or her own decision, which might be different from yours.

Table 1: A Ranked Listing of a Small Number of Projects
Project Points
Project 1 3000
Project 2 2500
Project 3 1250
Project 4 750
Total Points 7500

You might not use all your points. For example, in Table 1, one organization with four projects only used 7,500 of their possible 10,000 points. If you don’t use all your points, you may need to reconsider the projects you’re trying to rank. If you don’t have several projects that provide substantial business value, you may not be considering enough or the right kinds of projects.

As you proceed through the project ranking, you might realize that you want a larger point spread to show the rest of the organization how the projects rank against each other. In that case, you may need more points to clarify the ranking. See Table 2 for an example of a team that started with 10,000 points and wanted to show the rest of the organization how important Projects 1, 2, 3, and 4 were. Projects 5 and 6 were much less important.

Table 2: Ranked Listing of Projects Where the Decision-Makers Changed the Number of Points Used to Rank
Project Original Points out of 10,000 Updated P0ints of out of 20,000
Project 1 2000 5000
Project 2 1900 4000
Project 3 1800 3500
Project 4 1700 3400
Project 5 1600 1000
Project 6 1000 500
Total Points 10,000 17,400 (out of a possible 20,000)

 

 

 

 

 

 

 

Points help you describe the relative business value of each project. If you need more points to describe the relative importance, use them. When you use more points, and have a larger difference between the most important and least important projects, project staff understand when to stay with one project.

When I introduce this approach to management teams, they ask, “What do the points mean?” I’ve had good results saying, “You have $10,000 to assign to these projects. How do you want to assign the money?” There are times, though, when linking project value with dollars isn’t sufficient. In such cases, the managers will try to assess the risk of not doing the project and assign this risk a dollar value. One way or another, you need to link points to project value.

Table 3: Dave’s Ranked Listing of Projects
Calendar Integration 5000 points
Data integration 4000 points
Performance project 1000 points

 

Dave worked with his directors to use points to rank their projects. Their initial decision about their projects is shown in Table 3. But when Dave brought the ranking to his senior management team, they were concerned — wasn’t the calendar project too risky to keep as the top priority? Senior management was concerned that if the entire IT group was stuck on the calendar project, nothing would get done. Dave and his team needed to address risk.

RANK WITH RISK

I bet you’ve heard things like, “Don’t do the risky projects— do the sure things.” Well, that’s great, but how do you know what a sure thing is? How can you tell what the return on a project will be?

You can’t tell for sure, but you can learn a little about the risk and value of a given project if you do just one iteration’s worth of work on it. When you work on projects, feature-by-feature, inside a timebox, finishing features and showing demos as you proceed, you can learn more about the risk and the potential return. Once you’ve completed one iteration’s worth of work and received feedback on the demo from a customer or customer surrogate, you might have enough information to evaluate the project’s risk. Now you can ask the customer about the value of this project. Is the project high value? What would make it high value? The value discussion is how you can identify potential return. It’s still a guess, but it’s a more educated guess.

After due consideration of the risks of the calendar integration project, Dave and the rest of the IT directors decided to do just one two-week iteration on the projecand see where they were after those two weeks.

Ranking with risk is not without problems. Sometimes you need more than one timebox to know about the relative risk of the project. Sometimes the customer insists the project is high value, without considering the other projects in the portfolio. In that case, you need to consult other people, such as all the project sponsors or all the customers, and explain that you can only do one of these projects. Under these circumstances, which project has the most value?

One aspect of assessing the project risk and return is to determine for how long a particular project is risky or has a potential return. A number of years ago, an organization wanted to put their client information in a secure area of their Web site. Security was a new field, and they weren’t sure they knew everything they needed to do to keep the data confidential. Once a quarter, they tried to implement a feature in two weeks and test its security. As long as the other developers and testers could break the feature, the CIO decided the project was too risky. But about 18 months later, no one could break the features. Now the CIO moved the project up in the ranking because it was not too risky and had a much higher return.

Figure 1 shows how that organization reviewed the risks and the value.

Figure 1: Rating a Project According to Risk and Value
Project What kind of risk do we have if we do it? What kind of risk do we have if we don’t do it? What kind of value do we receive from this project? What kind of value do our customers see from this project?

When they reviewed the risks, they knew that if they didn’t keep the information secure, they would be in trouble with their clients. They had few risks if they didn’t do the project for a while. If the organization could implement the proper security, they would have a jump on their competition, but it would still be a selling job to their clients. Yet once the clients realized what they could do with their online information, the organization was sure they could change the way they worked with their clients.

 RANK WITH CONTEXT

At times, it can be quite difficult to rank with points. It’s especially difficult if you have different types of projects, which is quite common for IT departments. You might have “internal” projects to keep departments and systems up to date, as well as “external” projects that affect how you deliver products and services to your customers. If that’s the case, you will need to consider the context of each project when doing your ranking.

Table 4: Internal Projects, Ranked with Points
Internal Projects Points
HR Resume Systems 3000
Supply chain updates 2800
Year-end reports 2700
New reports for finance 2000

 

 

 

 

 

 

One IT group first organized their projects and then ranked them as internal or external (see Table 4). In Table 4, you can see that the HR résumé system offers the highest value of the internal projects, but two of the other projects are not far behind. And in Table 5, all of the projects are ranked higher than the internal projects. This IT department had enough teams to fully staff two of the external projects and one of the internal projects.

Table 5: External Projects, Ranked with Points
External Projects Points
Shopping cart additions 5000
Security for specific scenario 4500
Loyalty program, phase 1 3500

 

 

 

 

But separating the projects by type and ranking them helped the CIO explain to the different constituencies in the business why the IT group could not get to their project now. Once the senior management team saw the ranking, they could discuss the relative ranking and explain why different projects were ranked the way they were.

PAIRWISE COMPARISON AND DOUBLE ELIMINATION

There may be times when you need to look at every project in comparison to every other project; in other words, you need to do a pairwise comparison. Pairwise comparison is most useful when you have too many projects and not enough people to fully staff them. An easy way to do this is to write the name of every project on a stickie note, and place them all horizontally on a whiteboard or wall. Make sure you have all the necessary people in the room when you make the decisions. Once you’ve asked the “Should we do this project at all?” question, hold up two stickie notes and ask, “Which project is first, this one or that one?”

Figure 2: Double Elimination is a type of Pairwise Comparison

As you decide which project is of higher rank, put the stickies on the wall in vertical order, with the highest ranking sticky on top. Now, take another stickie from your horizontal collection. Take the top-ranked stickie and the new one and ask, “Which one is first, this or that?” Put the top-most stickie note on the wall, pick up the next one, and ask the same question. Repeat this process until all the horizontal stickies are in vertical order. You now have your project ranking.

Double elimination is a form of pairwise comparison. Many tennis tournaments are pairwise comparisons, because the winners all play losers. Figure 2 is a picture of double elimination.

Because the winners of each round eventually are compared to the losers of each round, you have the opportunity to compare each project against every other project. Note that while Project 1 initially beat Project 2 in the first round, Project 2 was not eliminated. As the team discussed their reasons for each project’s ranking, Project 2 won against all the others and eventually prevailed.

SINGLE ELIMINATION

Single elimination is a slightly easier approach to project ranking, as it does not compare each project against each other project (see Figure 3).

Figure 2: Single Elimination for the Project Portfolio

 

USE YOUR MISSION AND VALUES

You might find every ranking approach difficult to implement in your organization. Sometimes, that’s because you have not articulated your mission or have not defined the values of the organization. If either of those things is true, discussions about the ranking of each project might lead you in circles. In those circumstances, you will need to define the mission and values explicitly.

Once you know the organization’s mission, it’s possible to use it to say, “Yes, this project is part of our mission” or “No, this project is not part of our mission.” If you find that you have sacred cow projects, use your organization-defined values to describe why the projects are sacred cows and should be eliminated.

As Dave was working through the projects, he discovered that several people in Tanya’s group were working on a data consolidation project instead of the calendar integration project. He asked Tanya what that project was. “It’s a way to reduce the number of databases and disk drives in the organization,” she replied.

Dave thought for a minute, and asked, “When did this project start?”

“As part of our green initiative,” Tanya replied.

Dave considered this and said, “OK; we have a problem. One of our corporate values is to reduce waste. But this project is a lower priority, because our green value is a lower priority than enabling the business to make money.”

Tanya protested, “But when we save the company money, we make it easier for the money we make to go further.”

Dave explained, “Yes, you’re right. However, right now it’s more important to make money than to save money. That’s just for right now. I’m not saying we shouldn’t do this project at all. I am saying that for right now, we should not staff this project. That decision won’t last forever, but for these next few iterations put the data consolidation project on the portfolio backlog, and we’ll reevaluate it the next time we review the whole portfolio.”

If your organization does not have a mission, or if you have not defined and articulated your organization’s values, your efforts to rank the portfolio will make the lack of mission or value articulation visible to you. If you have defined a mission and defined your corporate values, it will be easier to make the portfolio decisions. It still may not be easy, but the decisions will be clearer.

RANKING ISN’T FOREVER

Once you rank the portfolio, you only have to wait for one iteration to finish until you can rerank the portfolio. Having that flexibility, it’s easier to make the portfolio decisions, because you know the decision doesn’t have to last any longer than one iteration. Say your organization uses four-week iterations. If you need to make decisions faster, you can ask every project to move to two-week iterations. Now you have an opportunity every two weeks to reevaluate the portfolio and determine whether you want a team working on a particular project or not.

REMEMBER TO KILL PROJECTS

Not every project deserves to stay on your backlog. Sometimes, products have had several releases (projects), and you’ve put another release on the backlog. Eventually you move it to the parking lot, because other projects are more valuable.

Yet while it’s helpful to move projects out of the way, projects that have completed some number of iterations but are not providing value should be canceled outright. Sure, you can take the “leave-our-options-open” route and move a project to the backlog or the parking lot. But if you have a number of higher-value projects and this one looks like it’s never going to deliver that value, kill the project. You can always transform it in some other way if you want to start it up again.

CONCLUSION

Every leader and manager in the organization who has responsibility for staffing projects needs to rank the projects in his or her portfolio. You might be able to staff all those projects, but more likely, you will need to make some decisions and then reevaluate the portfolio periodically. Use your organization’s mission and values to help guide your decisions. The easiest way to make portfolio decisions with sufficient frequency is to work in relatively short timeboxes, implementing by feature and finishing work in that timebox. Now you have a product you can see and can reduce your portfolio decision risks.

ACKNOWLEDGMENTS

I thank Sanjiv Augustine, Esther Derby, and Don Gray for their review of this article.

ENDNOTES

  1. Rothman, Johanna. Manage Your Project Portfolio: Increase Your Capacity and Finish Your Projects. Pragmatic Bookshelf, forthcoming 2009.
  2. Rothman. See 1.

©2008 Johanna Rothman. This article was originally published in Cutter IT Journal, January 2009, pp 13-18.


Update for 2012: Some people have asked me about using points to estimate schedule: DO NOT DO THIS. You will be wrong. This is because of the Cone of Uncertainty which I described in more detail in the scheduling chapter of Manage It! Your Guide to Modern, Pragmatic Project Management. The longer the project, the more your schedule estimate will be off. Use points to estimate business value only. Let me know if you need more blogging on this; I’ll put it on my backlog. — Johanna

Another update for 2012: Manage Your Project Portfolio is available wherever fine books are sold.

One Reply to “Selecting a Ranking Method for Your Project Portfolio”

Leave a Reply