Risk Insights Logo
The Assurance Show

Episode 11 | Evaluating and using customer feedback and complaints data for audits

May 4, 2020
Episode Summary


In this episode we discuss the evaluation of customer feedback/complaints and the use of customer feedback data, within internal audit and performance audit.

This includes:

  1. Auditing the customer feedback process
  2. Using the customer feedback content (data/information)
  3. Using customer feedback data for deeper, more effective, audit analytics.

Link: Blog: why are auditors increasingly using complaints and customer feedback data?

Episode Transcript

Yusuf:   0:26
Today's discussion is going to be complaints and customer and citizen feedback. Why internal auditors and performance auditors need to use it more.

Conor:   0:38
Yeah, it's something that we've seen being used more and more as part of those projects.

Yusuf:   0:43
in terms of using complaints and customer feedback for internal audit purposes and performance audit purposes. So for assurance purposes over all, there's three areas that we have been using it for successfully and we'll cover all three. So the first is around the complaints and feedback process. The second is the complaints and feedback content. The third is enabling deeper audits. Why don't we start with number one? So that's understanding how we manage customer feedback. So why is it important to understand the feedback process? What is it that auditors can do with information like that,

Conor:   1:17
the role of internal audit or performance audit is in making sure that organisations are delivering on their charter or the objective or helping management to deliver the business strategy. So why looking at customer feedback or complaints feedback is important for internal auditors or performance auditors to understand is they can contribute to delivering on that objective. Are we doing everything as an organisation to meet the expectations of our Customers or clients or the general public?

Yusuf:   1:46
Okay, So providing audit feedback on how it is that we deal with communications from our customers or from our citizens.

Conor:   1:53
So being able to provide some assurance over whether or not we're actually doing something with that information because one would suspect that if we're not doing something with that critical information that's being provided back into us, then are we really effective as an organisation? And the second thing is, once we get that information, how Well, are we using that both from a a reactive perspective, either trying to address an issue or a complaint or some critical feedback to how can This also inform how we position ourselves or get better.

Yusuf:   2:23
So we've written an article on this previously so we won't go into great level of detail in this process element it is important. So if we are doing an audit of how feedback is collected and managed and responded to, then there's five key components that we talk about. The first is Channel. So how feedback is captured, the second is record keeping, are we actually documenting and recording the actual feedback, then decision making. So how are we triaging and then passing that through internally to different teams. What is our response to the feedback? What is our actual response to the ah individual that's providing the feedback? And are we making any sort of formal commitment around how we're going to rectify and then the last is reporting? So how we deal with and report on feedback?

Conor:   3:07
The other fundamental issue, while not A process issue in itself is that trust relationship that we, as an organisation, have with our external stakeholders. Are we doing enough to maintain that trusted relationship?

Yusuf:   3:19
If we are doing an audit of the overall feedback process, what are the potential outcomes? What have we seen as potential outcomes from an audit like that?

Conor:   3:28
one of the recurring issues that we've seen is the timeliness, or effectiveness of, how we deal with complaints and that can either be on a case by case basis where there's an individual who is one of our customers or a member of the public that has had a negative interaction or transaction with our organisation, where they have raised concerns specifically about that. But perhaps more significant is where the Internal audit or Performance Audit has identified themes of under performance. In response, to say several or a number or a group of stakeholders interactions with us because if we're not doing enough to address that as an organisation. Then we're really creating a reputational risk for ourselves.

Yusuf:   4:10
You sometimes have a separate complaints process, and sometimes it's just you have an overall feedback process where both complaints and compliments and other sorts of questions come into Contact Centre, for example. And what we saw there was that the way in which complaints was defined, varied between different entities and between teams within entities as well. And in one of those cases, the definition of complaint was quite interesting. The definition of complaint was when a customer says, I want to complain, so it's not just any sort of negative feedback that we getting or what might appear to be a complaint. The only way that they, you know, measure complaints is when the customer actually says or when they record something and you know, categorise it as a complaint is when the customer says, I want to make a complaint and that's quite interesting because what we looked at, there was what, are, the different types of ways in which complaints come in and we found this one and then we said What else would be a potential complaint? So how can we look at what people are saying and what people are talking about when they provide feedback that ultimately makes it into a complaint? And then let's look at all other feedback and see whether any of those should have been complaints as well. And it was significantly surprising overlap between them, so they should have bean ah, whole range of other matters that were really complaints. But weren't categorised as such because simply because the customer didn't say I want to complain

Conor:   5:30
if we can get those definitions right and standard understanding of that across the organisation, I think that's a fairly significant outcome that performance audit or internal audit can deliver

Yusuf:   5:40
you know the reason we saying that is not because you just want to categorise your feedback in different ways. It's because the way in which you categorise it changes the decision making and the responses that you provide, how you deal with the customer will change. The nature of the the dealing will change if it becomes a complaint. So complaints quite often are, dealt with quicker and with more management oversight than any other sort of feedback. And so, if you're not, you know, recognising something as a complaint, then you may not be giving it the attention that it deserves.

Conor:   6:09
And more and more, you're hearing phrases such as this being used by organisations. There's no wrong door or you can contact us regardless of platform or channel, and it will be treated consistently so. And that's certainly something we're seeing more and more rhetoric around. And at least in the public sector, I think performance auditors they're starting to pick up on that. And Starting into, look, if if governments are saying there's no wrong door, for example, if you're contacting them about a service delivery issue, then are we carrying through on that particular promise

Yusuf:   6:40
that feedback process important important to understand and its points to understand, regardless of the type of audit that you're doing. So either you're doing an audit of the feedback process itself, so you know, direct audit or you're doing and auditvin a related area, marketing or product launch or the operation of a product, etcetera, and understanding that overall, just as an audit team, is important. So even if you're not doing a direct audit, it's useful to understand how it is that we capture customer feedback and how that translates. And that leads into the second item that we're talking about today, and that's content within our feedback. Information that we have in feedback from customers is really a rich source of intelligence to understand what our customers are saying about what we're doing well and what we're not doing well, it helps to identify both risks and opportunities, and those risks and opportunities will be directly within processes that we're auditing, but also in terms of the specific products that we selling or the specific services that we are delivering. So we may not through a normal process, type audit, identify some of that. However, we look at feedback that's provided by customers. It can help us better understand the way in which customers are experiencing the products and services that we're delivering and what that might translate to in terms of the way in which a process is designed to deliver

Conor:   7:53
I totally agree with that. And I think part of the problem or a traditional barrier. Let's call it that may have existed, to analysing the content of some of this feedback or this complaint is that there are lots of negative connotations around complaints and customer feedback, and it's only been in more recent times. The true value of the intelligence that sort of locked up within that information is really useful for the business to try and understand how it's performing.

Yusuf:   8:18
The other thing is that the data that we need to analyse to understand what customers are saying has not always been that easy to analyse. A lot of it in various situations will be in free text when you get customers calling in or customer sending e mails to us or providing some sort of feedback, via forms, et cetera, and that information is semi structured, so free text generally, customers verbatim saying what it is that they have a problem with, and that free text has not always been that easy to analyse, other than going through a sample of it manually. The technologies that we have nowadays, open source included, allow us to better understand what's in their free text and better analyse what's in there to get to the real nuggets that we need to understand what customers are saying

Conor:   9:04
Yeah and the ability to do that to sort of mine that information in more depth is becoming stronger with the digitisation of services Now, rather than a few years ago, people might have had an issue they would have perhaps written a letter. We're probably seeing less and less of that these days, which obviously assists with analysing the underlying data

Yusuf:   9:22
There's content. That's the 2nd one and then the 3rd one is understanding and evaluating feedback enables us to perform much deeper audits. When you spoke about process, we said that it's useful to understand the process regardless of whether we're doing an audit of that process or not. If we know what it is that is coming in via customer complaints and via customer feedback we can then use that information and triangulate it with other data that we have to enable deeper, wider audits. What we mean by that is in the number off audits that we do the existing data that we have doesn't enable us to get to the exact answers that we're looking for. And sometimes using complaints data to triangulate with some of the other data that we have, in both internal and external, can be useful both when we have a direct source of data for that topic or when we can't get hold of that data.

Conor:   10:10
So if we're doing a performance audit of a government programme, for example, we want to look at whether that programme is actually delivered on its outcomes. But we can also look at members of the public, their complaints, data and, like you said, Yusuf, to triangulate that as another source of evidence so that we can arrive at a conclusion about the effectiveness of that programme

Yusuf:   10:31
and with internal audit it might be a little bit different to that. So there may be some situations where we're looking at evaluating how a particular product is delivered or how well it's received, but Also, there's some strange use cases we've had outside off what you would normally expect to see. Quite often the third parties that we deal with will directly interface with our clients and where we have complaints or customer feedback on those interactions. It enables us to better understand how those third parties, First of all, are treating our clients, but also whether they are involved in our clients at all. So one example that has come up over the last couple of years, we've seen a lot of is where we're looking at determining whether third parties are either paying us the level of the referral fees that they need to pay us or where we are paying the right level of referral fees to those parties. And because those third parties are directly interfacing with our customers we can use customer feedback to understand exactly who it is that they're dealing with and how are they dealing with them and that we then used to determine whether there is an obligation for us to be paying any money to them as a referral fee or there's an obligation on them to be paying us any money as a referral fee. Can't go into a lot more on that because it's reasonably confidential. But I think that gives enough flavour on what it is that the data is being used for. And it's it's quite a different use case, but there's a few others that we've seen like that where we use complaints to triangulate with other data and it's not necessarily directly product related when it is product related it becomes even more important

Conor:   12:00
using that complains and  feedback data provides an opportunity to look at upside risk as well as downside risks. So the opportunities that abound for us to do things better or to increase our particular revenue streams, or indeed are we exposed by identifying where we have under paid some of our external partners? One of the other areas in which that information is useful for is that you can really find things in complaints and customer data that may assist your strategic planning for your forward audit work programme. You might actually get some insights and information while not relevant to that particular audit you're conducting but would actually inform your planning for the next 1 to 3 year cycle and identify potential risk areas or further opportunities.

Yusuf:   12:46
So does that mean that we're saying overall, that its important, firstly, to understand the customer feedback process, But then also to understand what data we getting as part of our complaints and customer feedback process, so that we can use that for planning, but then also feeding that into individual audits so feeding you know, the content into into individual audits, but then also feeding some of the data into individual audits so that we can go deeper

Conor:   13:09
if you're already using it for that sort of data For some of you are to deliver or execute some of your audits. That's fantastic. But if you're also not putting your more strategic lens on it thinking, well how can what I'm seeing in here actually inform our future programme. I think you're you're missing a really valuable opportunity there

Yusuf:   13:26
as an audit team. If there is some downtime or we going into a planning cycle and we've delivered a number of audits, this might be a really good opportunity to get some of that data, get some of that information and start working with it and seeing what it is that we can do with it so that we are prepared for both our planning cycle but also execution cycles.

Conor:   13:45
Okay, I just want to pick up just briefly on reporting Yusuf and reporting here is I'm referring to not not our assurance reports as such, but how complaints or customer feedback gets reported upwards through the hierarchy within an organisation. So one of the risks we're we're thinking about there is where there are, for example, significant matters that have been raised by external parties or stakeholders, and for some reason or another, they haven't. Once they've entered our organisation, they haven't been recognised as significant, and they haven't made it to the proper authority within the organisation. And perhaps a decision has been made at a lower level to actually try and deal with it in a certain way. Or probably. The greater concern is that those charged with governance don't actually have visibility over those matters. When we're conducting a performance audit or an internal audit, and we're looking at process. We're not just looking at how things come into the organisation, how they get triaged but also how they get elevated to make sure that they're going to the level and the right seniority of decision makers so that we're not actually exposing ourselves and that those charged with governance have the visibility, particularly over those significant matters that may have, for example, a regulatory outcome for us potentially or something that would adversely impact our overall reputation.

Yusuf:   15:01
Yeah and a lot of that has been coming to the fore with the way in which , globally we treating customers fairly in various sort of different ways in which that terminology is used, depending on the jurisdiction that you're in. But treating our customers fairly is quite an important matter. And that's just the base expectation we should be. We should be wowing our customers, of course, but the base expectation is that at a minimum, we should be treating them fairly.

Conor:   15:23
Yeah, that that's certainly a pattern that's come to the fore more recently and something that's definitely in the minds of boards and those charged with governance. It's that's really something that is that is front of mind at the minute.

Yusuf:   15:36
I know we said we're gonna talk about three different areas, but I've got five takeaways from the discussion. We just had the the first is understand the feedback process and understand how it is that we are managing customer feedback. The second is getting into the detail of the content that we getting in the feedback that is being provided. So what a customer is saying about what we're doing well, not doing well. Where are the risks and opportunities. The third is that we can use complaints and customer feedback to conduct much deeper audits and triangulate with other data for a whole range of topics that we may not have thought about before. The fourth is that as we understand how the feedback process works, we need to understand how complaints are raised. What's the way in which a complaint is categorised as such? And then how are those complaints and other feedback being reported up to the relevant levels of management or the Exco, or board et cetera, so that this the correct level of visibility over it and treating customers fairly? The last thing and I think this is probably the most important is as part of any annual planning cycle or even at the time when you may have some downtime, it's important to understand. Firstly, what the customer feedback process is, and then what data we have from that customer feedback process, so that we can both inform our planning but also inform the execution of our audits.

Conor:   16:52
On reflection, based on what you've just described, there are so many benefits to actually getting across that customer feedback and complaints data, both organizationally and for the efficiency and effectiveness of our internal audit and performance audit teams.

Listen to More Episodes Like This

Conor McGarrity
Podcast host

Conor McGarrity

An authority on data-focused audits, Conor is an author, podcaster, and senior risk consultant with two decades experience, including leadership positions in several statutory bodies. He’s driven to help auditors uncover new insights from their data that help them to improve organisational performance.
Yusuf Moolla
Podcast host

Yusuf Moolla

Fellow podcaster, author, and senior risk consultant, Yusuf helps performance auditors and internal auditors confidently use data for more effective, better quality audits. A global leader in data-focused auditing and assurance, Yusuf is passionate about demystifying the use of data and communicating insights in plain language.