Judgement and Evidence: Audience Q&A

Using Judgement and Evidence in Modern Balanced Scorecards: the use of subjective and qualitative measures.
Audience questions and responses

I presented a BetterManagement webinar on this subject, on 14th October, and it received a lot of interest. You can listen again to the webcast on the better management site (See link at the end)

Here are all the questions that those listening asked at the time, but not all were answered in the time available. I have published them here so you too can get the insight these questions raise.

Atena asked:

Q: If staff are not trusted on qualitative measures, wouldn’t this be the same for quantitative measures?

A: The premise in your question is not that the data is wrong, but that people are giving you false information. There are documented examples in UK hospitals where staff simply reported better information to central government than was actually the case to hide problems and avoid punishments. Of course you will need to sort out this trust questions, before you address judgement. But I would ask you some questions:
a) Is it the data you don’t trust, or the people? And why?
b) Why don’t you trust them? Do you trust them for other things? How do you know you can’t trust them on this? And why this?
c) Is it possible that they are using their judgement? The information you are asking for may not reflect the real situation: perhaps they are using their judgement. I am trying to expand your thinking here – give it some thought.
d) What is it about the way these people have been managed or are being managed that is encouraging them not to tell you the truth or the whole picture?

Fadi asked:

Q: I do a lot of business intelligent software, and the success of these application implementation depends a lot on making the collection of data as automatic and with minimal effort as possible, and that is very hard when collecting judgemental data, how can I deal with that?

A: Absolutely right. Business intelligence is about collecting large volumes of data and analysing it. It provides the evidence. Judgement is about how you are interpreting the data and what you are doing about it. I am sure you know that the balanced scorecard is about decisions and performance management – not just measurement. Let me explain with an example.

First do not be afraid of volume: in the case study each service may have been evaluated on up to 100 items once a year. So on that basis there were 3,000 (100×30) potential items of data, for each of the 14 regions. So first, remember you do not need a judgement for every single piece of evidence. We have an overall assessment and judgement score for all the statutory reports for a region across their 30 services. The assessment and judgement came from the regional director looking across all these and asking, “How well have we done and what do I need to pay attention to across all my services?” A manager of an individual service could do the same for the 100 items within their control. Judgement operates at a higher level and so there is less to capture.

Secondly we are looking at change, over time. That statutory analysis was done once a year, but the Regional director was working through the year to address any issues. So you are looking also at how well the actions that RD has put into place are working and their judgement of whether these are fixing things, before the next assessment. So the judgement could be about, “Is our response to the evidence, effective”

Just on example, but all this is easy to record manually, is relatively low volumes of written assessments and should be being assessed and discussed each month anyway.

Valerie asked:

Q: Is data available to mine (Data Mining)?

A: Typically the data is not available to mine because it is not in computer systems, it is in people minds. However their minds are available to mine. How do you do this? You can use the sort of techniques I explained (eg score out of 10 followed by questions) to understand what people are basing their judgement on.

If you collected a large number of people’s assessments, over a period of time, you would have some information on which you might be able to mine using textual analysis. But I would not recommend this above simply talking to people.

Se also my answer to the BI question from Fadi above.

Barbara asked:

Q: How do you address subjective measures in a culture that primarily values what you can measure quantitatively (financial, output, etc.)?

A: There is nothing wrong with valuing what you can measure quantitively. Remember it is judgement AND evidence. Two thoughts though:

First, sometimes, a strength overdone, can become a weakness. I did work for an organisation with a background from pharmaceuticals that was extremely rigorous in its analysis of information and markets. However they sometimes over-analysed things and missed opportunities because it was over-lengthy and over-rigorous. They didn’t trust their judgement to go with what they had, test it and learn. They wanted it “Right”.

Secondly I would say, go with what you have and exploit it. You say ”they primarily value…” but I suspect they “they also value…” judgement, and are unlikely to dismiss it altogether. Your staff will be making judgements based on the evidence all the time, and they will naturally be thinking the about capabilities, culture, skills, knowledge and behaviours of people that drive performance and outcomes. It may be that management are focusing attention of outcomes, quite rightly and for good reason. The reporting systems are not looking (explicitly) at the culture, skills and behaviours (Leading indicoto4rs or drivers of change). Evidence dominates judgement in discussions and decisions. So, go with what you have and, build on it. Perhaps start by capturing commentaries and assessments of the information they have looked at. This way you start to expose judgement, based on evidence. Remember someone is making a judgement about what is good evidence. It is always there.

Valerie asked:

Q: How long has the process of Judgement analysis been available?

A: Well people have always been using judgement and evidence. So it is not new at all.

It is not new to explicitly look for judgement statements and elicit them to create a conversation, but it is less common in many organisations. Remember in this case study we were pushing against an open door – they trusted their staff to use their judgement, and like any organisation recruited, developed and rewarded good judgement.

It is certainly far less common to find any use of it in the performance management, management information or software solution. In fact let us go back a step. Since 1994 balanced scorecards have been designed with objectives, before measures. Yet so many systems are incapable of expressing an objective and its measures. Yet the approach for measure design has always been, objective first, then characteristics, then, what to measure and how to measure it. But the systems concentrate on the answer – and record just the measures.
So I believe the thinking and approach has always been there. The software providers and designers of performance measurement and management approaches, in general, tend to ignore or dismiss it. Good managers naturally encourage it and develop it.

Paul asked:
Q: Doesn’t the use of subjective measures require the alignment of judgments?

Do you mean before or afterwards? It is absolutely not a pre-requisite. In fact the advantage of the approach is that it exposes differences and helps people share and align their judgement. Think of it as encouraging a conversation where people can express their judgement, learn from other’s views, discover why they think like they do or believe what they do, in a safe environment.

The Regional Directors here (and managers in every workshop on strategy and balanced scorecard development I have ever run, all say that the exposure of the deeper thinking of people, the sharing of ideas and the discussion was the most important part, because we came out aligned and understanding where and why differences occurred.

I have done this in an organisation where the people we brought together were from quite different departments and had never met, yet all tried to support the same outcomes. Obviously we were staring further back there but the conversation still helped understanding and eventual collaboration and progress.

If there was alignment of judgement everywhere you would have an organisation of clones. Just think about the credit crisis – Hardly anyone one questioned the situation with excessive exposure to hedge funds and potentially defaulting mortgages. You had a form of social acceptance of the norm, without question.

You want to encourage debate, and this approach does that.

Emre Asked

Q: Even if there is conversation between people, they may not be in the same opinion about the subjective judgements. In this case the disagreement may result in reactive behaviour of the people who was evaluated negatively. What is the correct approach to ensure smoothness of operations after the evaluation?

This question follows on nicely because it raises the topic of the culture that surrounds the conversation, the context, and what people might believe about the motives of people and consequences.

Isn’t it a shame that people feel they cannot speak openly because they will be criticised! In the presentation I talked about the difference between a context where failure was punished, (which will cause people to not say anything) vs a context where a failure to cooperate and learn, was punished (which is designed to encourage conversation and learning).

I go back to the two thoughts of “Explicitly gibing people permission” and “persistently signalling a difference in the culture”. Remember what you see is a learnt behaviour, and therefore it can be un-learnt and re-learnt. But that requires work, explicit change, repetition, practice and time.

Jean-Pierre asked:

Q: What relationship do you see with installing “culture of performance”?

I see what we are discussing as one of the ways to encourage the development of a culture of performance. But we should be clear what we mean by that phrase. I have a longer definition of what I mean by a culture of performance.