APLA 14th Biennial Conference
Thursday 15th July 1999

Measuring Performance: Monitoring Productivity - The use of statistics by management for effective oversight of operations

Nick Bannenberg
Queensland Parliamentary Library

Why measure performance?

It is a standard requirement of any business or non-commercial undertaking to be able to gauge the success of its operations. In the commercial sector there is often a clear-cut measurement - the so-called bottom line; the business either makes a profit or it is bankrupted. It is of course much more useful to be able to estimate in advance of the latter fate the financial health of any venture, so as to be able to rectify potential negative trends before disaster strikes. This holds equally true of undertakings which are not framed in the context of financial gain, such as government agencies, which provide a service. Here the motive is to be able to deliver a desired outcome as economically and effectively as possible, making the best use of available resources, whether it be personnel or tax-payers' funds.

Delivery of efficient and effective service is more likely if the operations of the organisation are regularly monitored and weekly or monthly outputs measured against projected targets. This is equally as true for libraries as it is for any other organisation, and equally as necessary where the inputs and outputs are for example, actual data instead of statistics of print material processed. It is also worth bearing in mind the observation that "what gets measured, gets done". Taken further, this measurement and monitoring can, and should, underpin goals and strategic plans. In many jurisdictions it plays a vital part in the formal integrated web of Strategic Planning, Annual Reporting and Accrual Output Budgeting. The object of these measures is to try to more accurately ascertain the true costs of services and to assess the relative effectiveness of an organisation's operations.

The issue has been concisely addressed by those involved in the EQLIPSE project. "Performance measurement is concerned with the evaluation of existing operations and services. The data generated is also used to inform planning and decision-making. Performance measurement is the quantification of a library's performance and provides a basis for evaluation and comparison, for example, to show how a library is achieving its objectives."

This paper is not going to address the macro-issues involved in such a system, but will concentrate more on the types of essential data necessary for a library manager to be able to ascertain if desired outcomes are being achieved, and output targets met. It assumes that the management has already arrived at a viable strategic plan to meet the requirements of clients. While canvassing generalities, it will concentrate on specific examples drawn from the presenter's own experience to illustrate them.

What should be measured?

It is of course essential that outputs be directly linked with outcomes. Once the organisation has arrived at the core reason for its existence and the key services it needs to deliver to its clients, it is imperative to identify the outputs which will make this possible. In Library terms, at a basic level one would assume that the outcome would be the delivery of relevant information to clients, within the time frame of their need. This involves identifying data already available internally or acquiring new relevant data from external sources, processing it for use, and forwarding it to the inquirer. Hence useful outputs could include the number of inquiries satisfied in a set period; the number of inquiries requiring inputs of externally acquired data; and the number of inquiries unable to be satisfied within the specified deadline. A further significant output would be the number of clients who expressed dissatisfaction with the information provided, but that could be a little more difficult to ascertain. It also introduces the difference between qualitative and quantitative measurement. Each has a part to play in determining the success of operations.

Quantitative and qualitative measures

Quantitative measures are easily definable sets of figures which will quickly delineate change. Their positive attributes are that they are rigorous, precise, trustworthy and capable of repetition. Detractors point out that they are also artificial, mechanical, boring, and not always illuminating in terms of outcomes. They basically involve counting how many times something occurs, and at worst may merely be an exercise in measuring popularity, without delivering data which reveals anything more. Qualitative measures on the other hand are seen by their supporters as being much more deep, meaningful and holistic, while opponents see them as rough, crude, sloppy and non-meaningful. Perhaps the most commonly recognised qualitative measure is the survey or opinion poll. However there are qualitative measures short of user surveys and polling which can provide measurable and useful data. Such concepts as Accuracy; Completeness; Accessibility; Availability or Continuity; Risk Coverage; and Compliance with legal standards. Qualitative measures provide an indication of how clients "feel" and "perceive", and are useful in ascertaining client interests and needs. In fact there is an excellent case to have measures which employ both types in order to really understand how successfully the organisation is performing.

While it is important that managers choose key outputs which will provide meaningful information on whether goals and outcomes are being achieved, it is equally important that the staff are also involved in identifying these activities and realise, as major stake-holders, the importance to the organisation of recording this data. It is not productive to set staff unrealistic performance levels with the resultant loss of morale and negative impact on productivity. Nor should officers and managers fall into the trap of thinking that the figures themselves are more important than the actual activities they record. It goes almost without saying that the performance measures must be addressing client needs, otherwise they will be a complete waste of time.

How to identify good Performance Measures

First some generalisations. Ideally performance measures should be:

* Informative - meaningful in content;
* Reliable - capable of returning the same result each time;
* Valid - actually measuring what they set out to;
* Appropriate - relevant to the organisation and its operations;
* Practical - should not involve excessive time and/or cost;
* Comparable - relate to similar measures in like institutions (difficult to achieve).

A useful objective test is "Can one make a reasonable decision from them?"

Recent projects have attempted to identify data sets for measuring library performance, with particular regard to the "electronic library". Some of the more interesting ones EQLIPSE (Evaluation and Quality in Library Performance: System for Europe) URL http://www.mmu.ac.uk/h-ss/cerlim/reports/; which was previously mentioned, EQUINOX - the European Library Performance and Quality Management System, URL http://equinox.dcu.ie, and "The People's Network" - a study undertaken by the UK Library and Information Commission. The first was carried out under the aegis of the Dublin City University Library from 1996-98, and the second is an ongoing project due to finish in November 2000, being undertaken by a consortium which includes the University of Manchester Library. It builds on a number of earlier projects funded by the European Commission.

More details of these projects form Appendices to this paper. They go into the subject at considerable detail but as they are predicated for large public and academic libraries, not all of the suggested measures are totally relevant to Parliamentary Libraries. Despite this, for an institution seeking to establish sets of performance measures they are both interesting and useful as background information. It has particular significance to us all as we operate in the "hybrid" conditions where electronic and paper-based information are about equally important in our day to day operations.

One of the great benefits of having an integrated Information Management System, (which the Queensland Parliamentary Library has been fortunate to operate since the early 1980's), is its capability of producing data on which relevant measures can be based. It is simple, but meaningless to know how many keystrokes are made by an operator, especially since vast quantities of information are now down-loaded directly. However it is just as easy to record who visits a web-site, how often, and which databases are interrogated, which is significant to the manager. Similarly it is vital to know which clients are asking questions of the Research and Reference Section, whether the topics covered are reflected in the material acquired by the Information Management Section, and whether the answers were provided by the agreed deadline.

Some years ago when we started to seriously address the issue of an Operational Plan and began to compile a list of appropriate measures, we issued a very detailed document for internal use by the Library Managers, addressing specific areas of responsibilities. Here is an example of what was produced for the Research and Reference Section:

QUEENSLAND PARLIAMENTARY LIBRARY
Research & Reference Section

Goal:

To provide a research and reference service which responds promptly and professionally to specific information requests by Members of the Queensland Parliament and by other clients.

Click here to view table indicating key elements, operational targets, strategies, responsibilities, time frames and performance indicators (against operational targets).

In ensuing years Queensland has retained most of these performance measures for internal monitoring, but for Treasury and other public purposes, a much less detailed listing was maintained to provide an over-all assessment of the degree to which the Library as an entity was fulfilling stipulated goals and strategies. They operate in conjunction with appropriate service standards, a summary of which follows.

Key Output Targets to be achieved

Key Client Service Standards to be maintained

* Prompt, courteous attention to customers;
* Guaranteed confidentiality of transactions;
* Adherence to quality standards within deadline requirements;
* Non partisan apolitical advice and assistance.

Research and Reference:

* At least an 80% Client Approval Rating for quality and content of response.
* Telephones at Client Service Desk to be answered promptly.

Publications and Resources:

* Publications ordered by Members to be despatched within 1 working day.
* Quarterly Bulletin of Queensland Statistics to be circulated to Members by the last day of each quarter.
* Legislation Bulletins and Notes to be issued at least 24 hours before debate on the topic.

Information Management:

* Selection, acquisition and processing of data to be carried out to ensure databases maintain currency, relevance and optimum access by clients.
* Bulletin of recent acquisitions to be issued by first week of month.
* Requests to purchase urgently required material processed immediately.

Information Systems:

* Research databases, including full text data, to be accessible on 24 hour basis by clients via the Local Area Network or dial-up modems.
* Response to Clients' calls for assistance to be within ten minutes.
* Timely access to customised training to improve Clients' access to databases.

An example of another approach to this, covering research and information services, is provided by the Victorian Parliamentary Library:

Information Provision:

Research:

Assessment form

The Queensland Parliamentary Library have also used some interesting ad hoc measures from time to time to help gauge performance. These have included the number of times Members record in Hansard opinions on the use of material supplied by the library, and the number and nature of complaints about aspects of the service received from clients, and steps taken to address them.

Monitoring Productivity

To this point the discussion has concentrated on identifying and implementing actual Performance Measures - Stage 1. Once satisfactory measures have been identified and put in place, the emphasis swings towards monitoring the resulting data-sets as well as general productivity levels. Stage 2 is the collection and reporting process and Stage 3 management assessment and feedback. In the Queensland Parliamentary Library there is the option of on-line interrogation for much management information on an on-demand basis, as well as the more formal presentation of data for weekly Senior Staff meetings.

As a manager, it is important to not only monitor financial commitments in order to keep within projected monthly or fortnightly predicted expenditure levels, but also to keep an eye on productivity levels. There are at least two obvious reasons for this. Staff salaries usually represent by far the greatest proportion of recurrent expenditure, and hence are a key focus of value-for-money. Secondly, there is an obligation to ensure that staff members doing similar tasks are performing at satisfactory levels with as little inequality in workloads as possible. In many cases staff may also be required to perform at an agreed level to qualify for performance linked payments, with the concomitant need to be able to assess as easily as possible if the requisite levels have been reached.

The Concord Information Management System continues to play a major role in both measuring outputs and monitoring performance. Over the past 15 years Research and advanced Reference requests have been entered from input forms filled out by duty officers on a daily basis, enabling each significant enquiry to be tracked in real time to check it's progress, who is working on it, and whether the deadline is past. Nearly 4,000 discrete jobs were entered in 1998-99. Simple reference queries are not treated. This database is not accessible by anybody without the necessary security clearance, assuring confidentiality of inquirers and inquiries.

The Research Enquiries database is valuable to library management, because it offers the facility of checking on key data. These include the progress of individual jobs being processed; on-line information about the number of inquiries being completed by each officer, (aiding in the distribution of new work); which clients use the service most, while also indicating usage by political party. Complementing the inquiry side is the database containing Research Reports and Added Value Reports, which provide the full text responses to the more important enquiries. This provides a fund of material for utilisation in answering future requests on the same issue. At the same time it allows the Director and Management to assess, with minimum effort, the general quality of the product being provided to clients, as well as monitoring individual staff skills. The two complementary databases are a good example of performance measures which are both quantitative and qualitative.

In areas outside Research and Reference Concord plays an equally significant part in supplying vital information. Details of which officers have entered or indexed data on an ongoing basis, and various other processing outputs can be swiftly assessed. Similarly, with audio-visual output it is relatively easy to check which programs have been edited ready for access (and replay) by inquirers on the Concord Research Databases, and also which files have been transferred to CD for the in-house video streaming network. In this way backlogs can easily be identified by the relevant manager, and can be addressed before they grow to difficult proportions. Another valuable attribute is the facility to match by broad subject area Enquiries and Acquisitions, whether the latter is by way of purchased monographs, or electronic documents downloaded from the 'Net. Ideally the majority of new material should be acquired in those areas with the heaviest user demand, while not of course ignoring other subject areas.

To facilitate tracking of key data Senior Staff at weekly meetings rely on a compilation of statistics incorporated in an Excel spreadsheet. The amount of detail this contains is rather overwhelming for the uninitiated to take in at a glance, so it will not be reproduced at this point.

Many of these features will be demonstrated during the accompanying presentation to illustrate the advantages of being able to easily monitor key outputs in order to ensure that major outcomes are on target. Performance measures and Indicators are vital tools in the Library Manager's toolkit of. In the hybrid libraries which we currently oversee, some degree of automated monitoring of such outputs is essential. Queensland is extremely fortunate to have such a flexible and innovative system as Concord in it's varied versions to enable this to be achieved so meticulously and simply.

Definitions - from Qld treasury's "Managing for Outcomes":