In this episode we explore Principle 2 - Quality - for the use of data within Internal Audit and Performance Audit.
2a: Understand the audience and meet their needs.
2b: Focus on the audit objective.
2c: Ensure quality (a.k.a. quality assurance).
Yusuf: [00:00:30] Today we're going to talk about thought three things to consider in ensuring quality in the use of data and analytics as part of audits.
Conor: [00:00:34] Let's get straight into it.
Yusuf: [00:00:36] The use of data and analytics is pretty common nowadays, that includes things like visualizations, both static and dynamic, and we call that auditviz. And then also obviously the selection of data and the use of analytics through the work that we do.
Conor: [00:00:53] Three things we are focusing on today and we need to understand, and the first one is our audience.
Yusuf: [00:00:58] So there'll be various members of an audience for audit outputs. Of course, you've got the audit committee or board. In most cases, the chief internal auditor, chief audit executive, head of internal audit would be reporting to functionally anyway. The key audience for an internal audit function would be the audit committee that are looking for particularly assurance on the way in which activities are being managed and projects are being managed. The secondary audience, some would say the primary, but most highly recognized as the secondary audience is management who need to do something about the individual items that are identified in internal audit. And then you may have regulators and others. that are interested in the outputs of, audit reports, but they would be a tertiary audience.
Conor: [00:01:41] Primarily you're targeting, your auditviz towards your audit and risk management committee.
Yusuf: [00:01:46] That's right? Yep. So that would be , the first set of people that, need to know, and of course management need to know maybe to a great level of detail. If you think about where to pitch your audit there's, then there's audience. It will be reasonably high level to the audit committee, , and a, but more detailed to management.
The audit committee. sometimes have a need or desire to get into the detail, but generally they want reasonably high level and then management wants something below that.
Conor: [00:02:07] would it be reasonable land to spend a little bit of time at the start of your audit program every year, trying to understand what the audit committee would like to see and how they like to see things to help you design your audit visit as you go through your program,
Yusuf: [00:02:19] that would be a prudent approach.
So knowing your audience by asking them directly would be important. Yes. And then the other thing you need to do is as you work through the audit plan, think about what it is that you want the audience to see. Based on what their needs are. And then what is the outcome that you're looking to achieve?
So that's number two, the specific outcome that you're looking for, sort of split into two areas. It's the objective of the audit. So what is it that the audit needs to achieve and then the outcome? So what do you want to happen with the results that you've created?
Conor: [00:02:52] the audit objective that's fairly easily understood and developed at the outset. are there any circumstances when the actual audit outcomes?
So the change you want to happen as a result of your audit may change as you go through the audit based on your audit is.
Yusuf: [00:03:07] Yeah, absolutely, your objectives are reasonably straightforward, particularly a high level objective because you define that. And because it's high level, it doesn't need to change significantly through the audit.
There may be situations where it does, because you'll find that, you know, there's nothing happening , , in a particular area and you need to take a slightly different approach, however, the results and what it is that you find. Well then dictate differences in the outcomes that you look to achieve. depending on the level of, issues , that have been identified where the issues , have been identified and what it is that you want people to do with it, or that you need people to do with it, that will then change and your orders will change.
So that's. two. And then the third is quality assurance. Now this is probably the easiest, which is why it's number three. And this is about various levels of QA, you know, self QA, reviewing your own work. Peer QA, so somebody else looking at particularly the technical details of the work that you've done. And you've got functional QA, so a level above or depending on the nature of the team that you're in, maybe another peer that does functional QA and then overall audit QA. So that's within the audit team. And then you'd want to do validation with the business team to ensure that what you've looked at, what you found actually makes sense. It aligns with what the business person is thinking as well.
Conor: [00:04:20] just for clarity here, when we're talking QA and disrespect, we're talking QA specifically related to audit fees. And what's been displayed or shown as opposed to broader QA across an audit itself. Is that
Yusuf: [00:04:31] yes, that's right. So this would be, making sure that the data that was used, the rules that were followed, the exceptions that have been identified actually make sense.
And then that the visualization, is correct. So it shows what it needs to show and it is technically correct. So it actually has the right technical backing to it, but also is functionally correct. So it's not overemphasizing one area over another, just to get an outcome.
Conor: [00:04:57] As we conduct the audit and we go through and we make our auditvizzes and invariably we see lots of really interesting things, while in and of themselves may not go to the objective of the audit. What are some of the ways that we can park some of those
Yusuf: [00:05:11] issues or
Conor: [00:05:12] how should we treat them that they can be useful later on?
Yusuf: [00:05:16] So it depends on whether the issues that we're finding directly relate to the objective.
So sometimes the hypotheses that are selected. Appear to cover the objective, but you find other things as you go through exploratory analysis that you can then tie back , quite clearly to the objective. And , that's the easy one. when it's something else outside of the stated audit objective the second layer then is, , is this actually something that we want the audit committee to know what we need the audit committee to know.
And then you can't hide behind the fact that the audit objective doesn't cover what you found because you found it. The third is where you find things that are relevant to management. They aren't necessarily relevant to the audit committee, but they are useful. And then you need to ask yourself, do I actually go down the path of finding out what's going on in more detail or do I pass it on to management to investigate further?
Conor: [00:06:03] are you talking there about the whole issue of going down rabbit holes and knowing how far to go and when to stop?
Yusuf: [00:06:08] Yeah, that's right. So it depends on the relationship that you have with various stakeholders and what the expectation is.
It also depends on the level of maturity of your. management teams. So sometimes, those rabbit holes just not make sense, efficiency, effectiveness, sense to go down because you really need to be focusing on the objective of the audit sometimes , so if you've got a situation and there's lots of audit teams that are in the situation where often you are able to change or enable change by showing what answers to questions that have been asked, what sort of value that can provide, then you may want to do it once and then pass it onto management for them to continue to do it.
Conor: [00:06:44] So it turns a little bit to me, like not going down the rabbit hole too far, it's a bit more of an art than a science and correlates with an individual's personal experiences. Is that fair to say?
Yusuf: [00:06:54] it's similar to what you would have with most audit work is that you can actually go down and dig really deep into details or broadly across and outside of the audit objective. But it all depends on what it is that you need to be able to provide, whether you can provide it, how much of time you have to provide that.
and whether there's. A benefit either to the audit committee management
Conor: [00:07:11] , and people are naturally curious, I guess when it's always good to have, , a peer have a look at your work and give you an almost independent view about , are you going too far into that rabbit hole?
Or, or maybe it's a supervisor or somebody in a more senior position, a up more experience to say, I think we've dug enough off here. let's pass that on.
That's exactly right. So, I mean, we've been talking about internal audit. For this whole conversation now, so we probably need to switch it around and maybe ask you the questions around performance audit.
So an and this is useful both to internal auditors and performance auditors, because there are lots of lessons to be learned from the conduct of performance audit. So what are these three things that we've been talking about? So audience, objective/outcome and quality assurance, mean in conducting performance audit.
The audience for performance audits, are generally outside of the organization. So it's sort of the auditor General's or controller General's or the Supreme audit institution. And for the most part, they are the parliament, and/or members of the public, and generally both. But most Supreme Audit Institutions actually report directly to the parliament. Just as you described with internal audit that they're, you know, as a primary audience, the second drill at ANZ, and sometimes at tertiary holdings in new Westminster democracy, certainly the primary audience would be the parliament and the secondary audience. So the. No less important are members of the public.
When you're bearing those audiences in mind, you really have to ensure that the way in which you are explaining what you've done in the audit, what you find, and the outcomes are really tailored to those particular audiences. So that takes us into, , number two and that's really having proper consideration of the outcomes we're looking for. So the objective of performance audits and bearing in mind, the performance audits can go for a significant period , could be up to and including perhaps a year, sometimes over a year, because they're usually quite in depth and extensive.
The outcome for the most part that you're trying to achieve is firstly, to give reasonable assurance over the activity that you're reviewing. But also you want to highlight where there are some opportunities for improvements, and that's generally communicated through the issuance of recommendations audit visit is particularly useful where you're trying to explain some analysis that may be quite complex, but the auditviz can translate that into quite simple chart or a simple diagram that is understandable by those audiences we just described.
Yusuf: [00:09:37] So knowing your audience, internal audit versus performance audit.
Internal audit work or generally stay within the organization. So up to the audit committee, to management and to others, but generally given where it does go outside. So two regulators, external audit sets under they're quite strict confidentiality requirements. , what you mentioned there was that your performance audit.
Audiences would be the general public as well. So in terms of thinking about that audience and ensuring quality, you probably have an extra layer of ensuring confidentiality and, making sure that you're not revealing any sensitive information.
Conor: [00:10:18] Yeah, that's absolutely right. And of course, every auditor out there will understand that sometimes taking.
Something very complex and explaining it in simple terms. It's not actually easy to do in and of itself. The risk being that if sometimes if you try and oversimplify something, you can slightly change the message and that's quite a significant issue. So for example, if you've got a particular performance audit finding, and you've done a lot of complex.
And I'll assist to arrive at that finding, and you try to simplify it for the ease of, for example, a member of the public to understand what you've done. You have to make sure that you've got the quality over not just what you've done with what you're conveying so that you're not actually misleading members of the public.
Yusuf: [00:11:05] Okay. So that means there's a heightened focus on quality assurance.
Conor: [00:11:09] and sometimes that can be quite difficult. And one of the barometers of trying to understand what external parties interpret from performance audits. It's quite often as through the prism of media. So what's been reported in the media based on a particular performance audit. If there are things being reported that do not align with the findings or the key messages contained within non-report, then perhaps we need to consider were we clear in how we reported? And if we used auditviz, for example to simplify some issues, did we use them correctly? Did we do enough to minimize the risk of misinterpretation of those auditviz products?
Yusuf: Sounds a bit easier for internal audit and performance audit is I have to say, because you don't have that, external expectation and the need to get to for a wide variety of audiences. However there are some differences, but based on what we've been talking about, just then, yes, there are differences, but there's just a lot of similarity in the way in which we need to be thinking about it. They may just need to be a few tweaks here and there to get it for your different audiences. So that's, so the areas that highlights the importance of knowing who the audience is and catering for what they need and what you're able to provide to them and what the outcome is that you try to achieve.
Conor: Absolutely. So the, the principals all align us between internal audit and performance audit. And we just have to ask the same questions, perhaps, maybe just one or two extra questions in terms of what we're displaying through performance audit is.
Yusuf: [00:12:44] in terms of the discussion we just had, there's obviously a blog article that we wrote , about this and we call it principle number two for data and analytics governance within internal audit, very similar principle will apply to performance or Adobe that some of the details will be a little bit different and we'll put a link to the article in the show notes.
Conor: [00:13:02] three main principles. When we're thinking about all, this is what we need to think about. First one is the audience. Do we know who our audience or audiences are? Do we understand what their needs are? Second one is, is it clear what the objective of the audit is and what the outcome we are seeking to achieve for our particular audiences?
And then thirdly, which is very important, of course, is the quality assurance. How do we make sure that we've ticked all the boxes in terms of understanding the audit visit that we are putting together, trying to display our findings and our results attracts the right level of quality.