This project planning tool will help you to identify appropriate indicators to measure how near you are to what you planned to achieve. Indicators are quantitative or qualitative measures that enable you to assess the degree to which project outputs and impact have been achieved.They also help you to focus on what exactly you're trying to achieve, and why.They reflect those areas of your product/service that you consider to be most important.They may be quantitative (e.g., how many people listened to a particular radio programme?) or qualitative (e.g., what was their opinion about the content of that programme?)
Some things are fairly easy to measure (e.g., number of people using your product/service). Others are less so (e.g., how did these users benefit from the product/service?). Because of the imperfect 'fit' between indicators and what they are meant to capture, it's often better to use several indicators to capture different dimensions.
To evaluate the performance of your information product/service, you first identify the product/service, and then describe to whom and how it is provided.This will give you indicators of the efficiency or effectiveness of this provision. Over time, repeated measurements will show if changes are in the direction you intended - and if not, you can take action to make further changes.
To evaluate the impact of your product/service, the procedure is similar. Here, indicators will relate to the output, the outcome for the primary stakeholders and the impact in the wider social and economic environment. For example, the impact of weekly radio broadcasts might be seen in the farmers' increased knowledge about market prices; an indicator here could be the change in farmgate prices at the local market compared with those in a bigger city market.This improved knowledge might lead to the farmers growing higher-priced crops; an indicator here could be the percentage change in the production of certain crops over a specific period.
Indicators are a way of making the issues that are important to you measurable.They allow you to answer questions such as:
So you need to be clear about why you want to evaluate specific issues, and what dimensions are important to you.The choice of indicators will determine how you collect data, and then how you analyse, interpret and report these data.
First, you need to be clear about what you want to know.Then you need to think about how best to find out this information and which dimensions of this information can be measured. For example, if you're interested in 'listener satisfaction', you could conduct an audience survey that includes the question 'Did you like programme X?' You then measure the proportion of listeners who replied 'yes'. But you might also want to know about what parts of the programme they enjoyed most; if this is the case, then you may be able to more easily get this information by interviewing people separately, rather than using a questionnaire.
There are five steps involved in creating indicators:
1: Make the project issues (objectives) measurable
2: Select the indicators
3: Specify the type of indicators
4: Determine the reference points
5: Assess the quality of the indicators
Step 1: Make the project objectives measurable
Some issues are obvious indicators (e.g., number of users of an information service). Others are more abstract (e.g., terms such as the ‘quality’ of the information service, and the ‘learning’ created by providing it). So you need to define the issue and the activities related to it in concrete terms.
Box 3.11 provides an example of making an abstract issue measurable.
Making an abstract issue measurable
|To measure the ‘quality of the information service’, you might need to look at:|
In order to assess the abstract concept, you look for specific items to measure. Each of them could become an indicator. How many you choose will depend on the time and resources you have available.
The items to measure should reflect the views of stakeholders.Whether an information product is really useful depends on the values of the community served. In order to arrive at an appropriate representation of the issues, seeking the views of the stakeholders can be very helpful. Using a participatory approach may also save time in the long run.
Step 2: Select the indicators
Step 1 will produce many possibilities for defining the issues at stake.The second step is to select the set of indicators to be used.You are trying to provide as many reference points as necessary to capture the essential issues to be evaluated.The evaluation of the performance of a product/service should include indicators on the services provided, their intended benefits and the target group. The evaluation of the impact of a product/service should include indicators on the project outputs, the
immediate beneficiaries and the changes in the wider community. The number of indicators to choose depends on the resources available for the evaluation. About 10–15 indicators should be the maximum, except for very large studies. Generally, it’s better to have a small number of significant indicators.
The indicators should also be selected on the basis of a set of methodological rules (see Box 3.12). To determine which of the issues identified in Step 1 should be reflected as indicators, you should check this list. Ensure that there is no overlap or duplication in the final set of indicators chosen.
Methodological rules for indicators
The commonly used criteria defining good indicators are:
VALIDITY Does the indicator measure the condition/result?
Step 3: Specify the type of indicators
Step 3 involves deciding whether an indicator should be qualitative or quantitative:
Both qualitative and quantitative indicators are refined by adding the unit of analysis in terms of the level of the social system (who – individuals, communities, organisations, networks etc.), the period of measurement (when) and, if relevant, geographical coverage (where).This could be done as shownin Box 3.13.
Whereas quantitative indicators are more precise, qualitative indicators provide a richer picture of reality and a better understanding of the reasons for change. Both types are ‘objective’ in their own way. Quantitative data are less easily challenged, but qualitative data help to show the relevance of the ‘hard facts’. In the context of an open and dynamic learning process, qualitative data are often more useful for convincing people about the value of a project. A mix of qualitative and quantitative indicators can be used; often, this is the most appropriate approach.
Examples of qualitative and quantitative indicators
|Quantitative indicators for the ‘accessibility’ of an information service’:|
It’s worth noting that it’s often not possible to get the detailed data that, ideally, you might like. In the case of the example in Box 3.11 for instance, instead of looking at the different items determining the ‘quality of the information service’ (accessibility, usability, relevance, efficiency), it
might be more realistic to use a qualitative indicator that simply seeks users’ overall perception and
level of satisfaction with the information service.
Step 4: Determine the reference points
Whether an indicator is qualitative or quantitative, in order to interpret its value it is necessary to set a point of reference against which the measurement can be compared.The ideal reference point would be baseline information that refers to the exact items specified in the indicator. For quantitative baseline data, apart from some simple measurements (e.g., the circulation figures for a newsletter) such information is usually not available. Qualitative baseline data are much easier to
find. Usually, development projects have documents concerning the problems they are supposed to address and supporting baseline data. Apart from baseline data, there are alternative reference points against which measurements can be
compared, as shown in Box 3.14.
Reference points for comparison
Step 5: Assess the quality of the indicators
The next step is to assess the quality of the indicators selected. A commonly used formula for doing this is to apply the SMART and SPICED criteria to the indicators (see Part 2, page 50).These criteria are used for a final check on the indicator. It is not easy to satisfy all criteria at once. For a cost-effective evaluation, it’s better to drop indicators that are too ambitious or costly to measure. Creating indicators is an iterative process. From Step 5, you might find it necessary to return to some of the earlier steps. Often, establishing indicators is, in itself, already part of the evaluation, especially if you do it in collaboration with stakeholders.
The SMART and SPICED criteria for measuring the quality of indicators
Check that your selected indicators are:
An example of a SMART indicator:
Check that your selected indicators are
An example of a SPICED indicator:
Applying the indicators
Once you have created the indicators, you use the data obtained to judge the performance of your product/service.The data provided by the indicators will gain meaning only through your understanding of their significance. Other data obtained during the evaluation process, apart from those derived from the indicators, could turn out to be extremely important in understanding the results. Qualitative indicators offer more possibilities in this regard.
Sources of more information on indicators
|Professional agencies and associations have compiled lists of key performance indicators and codes of conduct.|
Examples include the Library Performance Measurement and Quality Management System for electronic libraries
(http://equinox.dcu.ie), and the work on performance measures by the International Federation of Library Associations and
Institutions (IFLA) (http://www.ifla.org/VII/s22/annual/sp22.htm). Another example is the ethic standard of the
International Federation of Journalists (http://ifj.org/).
It is also useful to look through specific evaluation guidelines, such as those provided by the International Development
Research Centre (IDRC) on evaluating telecentres (Whyte 1999a, b).
Some key points in creating and applying indicators
The strengths and weaknesses of using indicators
|Measuring change without understanding the context in which the data were collected can lead to inappropriate conclusions|