In this episode we discuss going beyond the audit report.
That is, the audit report is not necessarily the end point or final deliverable.
There are a few things that we can do to go beyond that and increase the value that we provide.
- Remedial actions – going beyond the initial remedial action (related blog article here).
- Using the report for maximum impact.
- Following up on reported items.
Today our topic is going beyond the audit report. Sounds very exciting. I think there are three things we’re gonna touch on today.
There are three things. So the first is remedial action that we identify in the report and going beyond that initial remedial action. The second thing is going to be about using the report for maximum impact. And the third thing we’ll talk about is following up. So follow-up audits and other follow-up activity.
I think it is useful, though, to spend some time just talking about remedial action. So, in terms of an internal audit then, what are we talking about when we say remedial action?
So remedial actions are those actions that we have worked with management or they have worked on independently to address any issues that we’ve identified through the audit. When we talk about going beyond that remedial election we’re talking about present, past and future. Quite often we think about the present, and we think about the specific items that have been identified where the issue was found. In the old days we used to do sample selection. Nowadays we do full population recalculations using data, right. Maybe a couple of years ago, when we were still doing sample based auditing – we select the sample; we look for anomalies based on what we know of the process or the risk and where we identify any anomalies we raise those.
Quite often those anomalies are taken as the items that need to be fixed. So as opposed to fixing the whole issue. So when we talk about the present, we talk about fixing the whole issue, not just those specific samples that have been found, but any other such items that we can identify as well, either within or outside of the specific process that’s been audited. We then want to go and look at the past is well and determine what a reasonable period in the past to look at is for any actions that we may need to address. Let’s say we charge customers fees on a banking account. If we identify that there are some fees that have been charged incorrectly. If it’s an underpayment by the customer, we may decide to only fix it going forward. If it’s an overpayment by the customer. Typically, we want to go back about seven years, so that we’re treating our customer fairly. In some cases, we want to go back to underpayments is well, but that’s the topic of a separate discussion. The future is about – what can we do now to make sure, to the extent that we can, we don’t have that situation come up again in the future.
Okay, so those three chronological aspects you’ve just mentioned there – so remedial actions that deal with present, deal with the past and deal with the future. Are they in order of importance?
No, they’re not in any particular order. We may decide that one is more important than the other, and it will depend on the circumstance. In some circumstances, the past will be more important. In some circumstances, the future will be more important. The point is to find all three scenarios – so past, present and future, and then prioritising them for implementation.
And I’m guessing that in big organisations in particular, the business stakeholders most interested in the past issues or the present issues and the future issues may in fact be different people.
In fact, sometimes the people that were involved in the past, are no longer there. We see that quite often. Usually the easiest situation. There’s no personal connection with any errors that have existed in the past or processes that haven’t worked well. And the new person coming in just wants to see it get fixed.
Issues that may impact the future, particularly where there are customers concerned – that may fall within the remit of someone trying to maintain the customer base or to grow the customer base, to grow the business, that type of thing, to ensure that problems are eradicated.
You have situations where there are projects that are on the go and knowing about these issues, and, you know, folding those into business cases for those projects will be important. So if you identify these sorts of things and go to not just the BAU people, but any project people that are involved in related activities, you might get better take up because you’re helping build business cases for the projects they involved in.
And does that include the people most in most interested in the present issues – how we’re travelling currently, or could that be a spread of people around the organisation?
Again it will depend on the individual circumstances. Often what we see is that the future is something that people want to fix because there’s budget for it, etc. We do need to ensure that we go a bit beyond what’s just been identified so that we make as much impact as we can with the audit work that’s being conducted. Identifying that scenario is not always an easy thing. Once it’s been identified, if you can look across various processes or sub processes, a lot of the hard work would have been done up front, just thinking about it, just thinking about what could go wrong and then use that and make sure that that’s maximised.
Going beyond the initial remedial action is the first thing that’ll help us maximise the impact of our reporting. I’m gonna go to the third one and then come back.
The third one is following up on matters that have been identified in the report.
Linked to the first one. So the first one was about remedial action, right, and going beyond that. Following up is about looking at the processes as they’ve been as they are being designed, or, as they have been redesigned. Controls that have been fixed as a result of actions that have been agreed on. And following up on those to determine whether that now works and also works in the context of the current business. Sometimes you’ll do a follow up after 12 months or 18 months in BAU world. In sort of project active audit you would do it as you go. But as an audit team, we need to be helping management work through whether the actions continue to be relevant, because your actions are actions are usually appropriate at a point in time at the time that the audit was conducted. But businesses change. Business processes change. Expectations change. Objectives change. The strategy will change. So we need to, in conducting follow up work, ensure that actions that have been identified continue to be relevant and appropriate.
Slightly different in the performance audit world. Every report or pretty much every report results in recommendations that need to be implemented by those public sector entities that have been subject to the performance audit. Typical practice is for those entities to provide their response about to those recommendations that actually gets included in the report that goes to the Parliament. So if we’ve got entity A, entity B and entity C, and we’ve made several recommendations, they will each respond to us saying yes we accept, or we decline the recommendations or we disagree and this is why. Then, that gets folded into the report that’s ultimately prepared and that then is combined and goes to the Parliament. Usually, any response by those entities where they have agreed, provides a timetable for implementation. So they may say, we’ll do this this in 6 months or 12 months. That then gives a window of opportunity for the auditor general that’s issued that report to actually then go back and revisit the issue and say – look, as an organisation, you committed to doing this. Please provide us a written response as to how you’ve implemented these recommendations. So that is one way that’s regularly used to follow up particular entities on the implementation of particular recommendations. Now one of the other methods that’s commonly used by auditors general is they take a subset of their reports that are done every year. So for the sake of argument, let’s say we do 10 performance audit reports a year. In two or three years time, we might say, as part of this year’s work programme, we’re going to actually follow up on three of those reports. So these entities told us this is what they were doing to implement our previous recommendations from a couple of years ago. We’re actually going to do a mini performance audit and determine. Firstly, if they have implemented them. And secondly, if their implementation has actually solved the problem or solved the risk that was identified in the first instance. So there are two aspects to follow-up: follow-up individual recommendations made to organisations within reports. But then a secondary measure is to select a sample of performance audit recommendations and to do a mini performance audit of their implementation.
So that’s both – have you implemented it and the implementation has resulted in the risk being addressed?
What then happens if the action that was determined two years ago is no longer relevant and won’t address the risk. If they haven’t taken that action, but have taken an alternate action. Is that okay?
So I think you’ve asked two separate questions there in one. If the issue is redundant, as it was originally identified, and either doesn’t require action to address it or if a different action is required. Then that’s perfectly fine, if that can be rationalized and proved by the entity that was responsible for it.
But then do we look at what the alternate action was and whether it’s been implemented?
Yes,we do. So, firstly, what have they done and why in the first two questions. So we told you to do this. You didn’t do it this way because the action didn’t address the matter as you saw it. Why didn’t it address it? What have you done to address it? And then we need to then think we need to make an assessment whether or not what they have done has been effective or not.
In terms of addressing the initial effectiveness or efficiency or economy criteria?
If that issue is still live. If the issue has morphed, and, their response has been well, we have changed what we’ve done based on the changing issue, then we need to determine does it still address the root cause problem?
So some internal audit teams are involved in closing off actions. Audit will identify an issue. Management will identify the action to address that issue. And then, quite often, the auditors will come in to determine whether the action has been taken as per the due dates and report that back to the audit committee.
And is there any set timeframe for follow up by internal auditors on the implementation of matters, or is that on a case by case basis?
It’s a case by case basis. The follow up of actions that have been taken would depend on the way in which the first, second and third lines work together. Sometimes what we see is that the second line will be involved – if there’s a second line, so if there is a risk team. Second line will be involved in helping management to address the issue, and then doing a first pass of any remedial action that’s been taken. As in, once the action has been taken, checking that it makes sense. And then audit will come in to actually close the finding at the end. Validate. So that’s ongoing. So it’s not a specific follow up audit that’s been undertaken. In other cases, there will be specific follow up audits and that may be, you know, six months or 12 months down the line, depending on what the dates that have been identified are. If the audit committee sees that there’s a particularly high risk area, or very high risk area, where they’ve been failures, then they might request that audit actually conduct a full follow up audit on that area. It does depend.
The second thing is using the report for maximum impact. What are you thinking there in terms of impact?
This will be different for internal audit and performance audit. Internal auditors will usually use the work that they’ve done internally. They’d usually be reasonably stand alone sets of work. There is obviously a lot of integration between audits, but usually the audits will stand alone. Report will be created and those issues need to be addressed. But following on from the themes you mentioned under going beyond the initial action and following up, what is it that the audit report can be used for more broadly across the business in the case of internal audit, more broadly, across various government departments in the case of performance audit, to maximise the work that was done and the outcomes of those reports, the work, the recommendations and the potential for improving practices.
When you talk about Mike’s maximising impact there, the first thing that immediately comes to mind to me in terms of performance audit is visibility. How do we get the visibility of the report we’ve produced either across the sector or in front of the people that need to see it so that they can actually take some action?
What would that look like?
In terms of visibility one of the very important mechanisms that can be used by performance audit in particular is the media. And that’s primarily because they have an extensive reach to a very broad group of people. And this is a subject we touched on in one of our previous episodes, when we spoke about audience and how to position in front of your audience. The media while it has and we’re talking there, you know, print media, radio, media, social media, and other sort of traditional and emerging media sources. It’s important to, firstly, make them aware of your report on what you’ve done and how you’ve done it. But potentially most importantly, what are the findings? What are the messages? What does it mean to the people within that community or who use that service or who deal with the public sector on a day to day basis? So firstly, making sure that they understand the meaning so that they can convey that properly.
Okay, so the first thing is to get the report distributed through the media?
After it’s been provided to the Parliament, quite obviously, because that’s our primary audience. But I think in this day and age it’s more and more important for the media to have a good understanding of: firstly, what a performance audit report is. What’s its purpose? Who does it speak to and what does it try to convey? So that’s an education of the media firstly, and secondly, to get them to use your report to spread the message to a wider audience.
Okay, so we have delivered our report to Parliament. Before that, we would have issued a draft or close to final report to the individual agencies that were involved in the performance audit so that they know what the issues were that had been identified and for them to respond to those. That response is folded into the report. The report is issued to Parliament, And then we put it – as performance auditors we’d put that onto our website, for example. Is that right?
And then the media will pick it up. Or do we actually engage the media?
I think that differs from audit office to audit office. I know there’s recently there’s been more of engagement, proactive engagement from auditors general with the media in trying to get on the front foot and say: Look, this is our work programme for the year. This is when we intend to produce our reports. You guys should get ready for it so that they are forewarned as well and they can kick into gear their processes and reporting on our reports, so to speak, at the relevant times. In the past, dare I say there was probably more of a passive arrangement whereby there wasn’t as much engagement with the media from auditors general around messaging. It was more of a clinical relationship whereby auditors general took the view: Look, I report to the parliament. That’s the end of my reporting obligations. Anything else that happens thereafter with that report will not be instigated by my organisation.
So there’s been a shift away from that because auditors general have recognised the benefit to be obtained by working through the media?
Coming to the realisation of the benefit to be gained through working with the media to spread your message is the result of community stakeholders actually saying, for example, to people that work in an audit offices: We find out about you guys not through your office or through the parliament, but through more informal means or through the media. So perhaps then auditors general have thought: Well, that’s how people consume the product, which is my report, performance audit report. And perhaps it’s in everybody’s interest – and going back to the thing we’re talking about – trying to maximise impact of the great work that we have done, if we have more of a proactive engagement with the media directly.
Ok, so that’s traditional media, then. What about the use of social media?
Certainly that’s been amplified in at least the past year or two across many of the audit offices nationally and internationally on. What we’ve seen, for example, is a lot more traffic on blog’s, on social media feeds, modern ways to communicate in smaller parcels of information rather than traditionally you say, Come to my website. I’ve got a 100 page report. Here’s what it is.
One of the types of things that we seeing that could help create impact. Its traditional media. It’s social media. So Facebook, LinkedIn, Instagram Twitter?
Yeah, all of the above.
Haven’t seen too many podcasts in the performance audit sphere, I have to say.
Blog articles, certainly becoming more and more popular. So I think another important thing is and trying not to veer off topic too much here, Yusuf, is if we’re talking about maximising impact, individual reports themselves need to be written in a way that’s – not familiar to the reader but presents information in a digestible way. When we talked earlier about the media, traditional, non traditional, so forth, that’s the reach and the spread of your audience, how do I get the most people possible to see what I’ve done and what we’re saying. The second thing is, once they know it’s available, how do we make it so that they can actually do something with it, understand what we’ve done?
That’s the report itself or is that all the different artefacts that can help you understand the report?
It’s the report itself. And then the report should obviously signpost the artefacts that are most relevant.
When I say the different artefacts. I’m thinking you have: YouTube video overviews of a report. You have summaries of report. You have the conclusion at the beginning of a report. So let’s face it, right? Not everybody’s gonna read 100 page report or 80 page report. It’s too long. So what are those digestible snippets that can enable you to get people to actually read and understand the essence?
Okay, that’s an excellent question. I can tell you that we’ve dealt with several audit offices, their performance audit teams, where upon consultation with parliamentarians have said: Look, just like you said, I just don’t have time to read all these 100 page reports that your audit office and all these other great organisations provide me. What can you do for me, that’s more digestible. Audit offices have listened to that and gone away and done exactly as you’ve just described. They have said: Okay, here’s a three minute audio of our report, here are the key findings, key messages, the conclusion and these are the things we’re asking agencies to do. And feedback generally from those MPs and chief executives of departments has been: This is fantastic. It’s really useful. I can get to the main messages very quickly. And if there’s a particular thing that I want to follow up on, I can speak to the relevant person within my organisation.
So Conor, what about the language that’s used to create the report? Just going back to internal audit for a sec, what we have been seeing quite a bit over the last few years is a focus on the use of diagrams, so as opposed to just text. But also importantly, there’s a very big focus on the use of plain English language to write those reports so that we go away from the way in which we were taught to write findings and issues and conclusions and write it in a way that is more digestible, not just in terms of the the format, but in terms of the actual words.
So there certainly has been a push we’ve observed in our workings with various audit offices around the use of active language. Say things simply in an active voice as opposed to making a finding that is passive, which was the more traditional approach to saying things. That’s not as easily done as people may understand. And it takes a little bit of time for the authors of some of these performance audit reports to get their head around that; it’s time and effort well spent because it makes it much easier for the reader to actually understand what we’re trying to say as auditors.
And part of that perfect in the way in which we speak as well, right? So we often speak in passive language as auditors. It’s just the way. I often speak in passive language as an auditor. Just the way that I learned over the years and some of that habit has to be broken. But I have come across a few good resources for plain English language, one of them based out of the UK, and we’ll put a link to that in the show notes. Very easy to follow. Of course, as you would expect. What else are we seeing in terms of having impact?
So I just want to talk about your point there around the growing well, certainly in performance audit, the growing use of diagrams on charts.
A picture paints a thousand words. It’s a cliche because it’s true and as long as you make that picture. – so in terms of an audit reported – you know, if it’s a chart or a diagram or something that explains a process or can situate a risk for the reader, quite simply.
Or just an icon.
Or just an icon. Yeah, absolutely. It’s a more inviting report to read. But the difficulty with that is it’s not always easy to simplify complex things into diagrams. So we have to understand that the readers of our reports, which is what we’re talking about ultimately, may be more comfortable with descriptions than words.
Sounds like something that we need to explore – in detail – in a separate episode, and we’ll do that in a couple of weeks time hopefully. In terms of making impact: so we’ve spoken about the use of the media, use of extended media – so various social media channels and blog’s etc., plain English language, the way in which reports are actually produced – so more digestible, shorter snippets that are more digestible than just 100 page reports. What else can performance audit teams do to maximise impact?
In my mind, one of the most important things that they can do to maximise impact is refer to similar reports that have been done, maybe in other jurisdictions or even other countries, and draw on some of the key issues identified there. So while we, for example, might be doing a performance audit on a health related topic. Another country may have done a very similar performance audit, slightly different scope. But there’s an opportunity in our report to reflect on some of those issues because readers are quite often interested in understanding not just what’s happening in my backyard, but I can I situate ate this against the performance of a similar function, maybe in a like country or another country. So the answer to your question, which is a long winded answer is one of the things we can do to maximise impact is to say, this is how we’re travelling in our own backyard. But look, this other country or this other area is dealing with similar problems and this is how they performed, or these are things we can learn from them.
I think the last thing we wanted to talk about is the use of presentations and other types of events, discussions, etc., to spread the message across government entities. So usually when we do a performance audit we’ll focus on a subset of entities, they will be heavily involved in the audit. So they’ll usually be committed to addressing the issues that have been identified. What are the those mechanisms? So those informal and formal discussions that we can have to help maximise or to help extend the reach of what we’ve seen and what needs to be done across various government entities that fall into our jurisdiction.
The delivery of presentations about performance audit reports are very effective mechanisms for a couple of reasons. The first one being you’ve got a captured audience and they’re there by virtue of the fact they’re interested in the subject matter, and they’re probably interested because either they’re responsible for that in their own organisation, or they are going to be responsible for implementing what the auditor general has said in a performance audit. So you’ve got those people who have a genuine interest in being there. But the second thing, which is really important, what you don’t get from a written audit report is: during a presentation, people have the opportunity to ask questions. And some of those questions may go to matters that you may not have been able to explain within the report for various reasons. Or they may just have dropped off through the editorial process. There is real value and having the interaction and like minded people in a room talking about what they see and things that arise in their mind as a result of your report. So I’m a strong, strong backer of presentations about audit reports.
And that might be something that the internal audit community could work with as well. Often our reports would be provided to management that were involved, it then goes to the audit committee and then actions are tracked. That particular business unit and the risk team, the audit committee and the audit team will be very aware of what’s actually in there. But often we see that other entities in the organisation don’t get the opportunity to ask questions about what was identified, what the potential is for them to use better practises identified to change their own practises as well.
There’s a stereotypical view of auditors, as you know, locked in dark rooms, looking at numbers all day and all night. Giving presentations across organisations, whether it’s internal audit or performance audit actually demonstrates to stakeholders that auditors are real people. And maybe the stereotype doesn’t hold true so much and they can interact. And in fact, the more progressive audit functions these days are actually really willing to engage and get early feedback and use that to actually drive their own performance. That will be my observation.
That’s great. You were talking about the presentations of results of performance audits or results of an individual performance audit. Is that done for all performance audits?
We should. As to much, how many resources we devote to each of those presentations would depend on things we’ve mentioned before, such as the relative impact, the reach of stakeholders involved, the genuine public interest in it. So we may do five presentations on a particular performance audit topic because it hits all those high notes. But in any event, I think, regardless of the number of performance audit we do. It is a genuine benefit to do a presentation for each and every one of those.
So it sounds like the presentation on a performance audit topic can provide very high levels of impact across entities that the audit office deals with. Are we seeing today that every audit office is providing a presentation on every performance audit report that is produced?
Absolutely not. Not every audit office is presenting on the results of every performance audit. However, there are certain audit offices that do accompany their reports with presentations in every circumstance.
So what is it that’s holding those audit offices back?
I would suggest those audit offices that are not doing that – the reason is the auditor general of those offices does not consider that that’s an appropriate way to fulfil their mandate. So perhaps they have got more of a traditional perspective as we talked about earlier where they feel that: I report to the Parliament. Once I’ve done that, I’ve fulfilled my duty. Whatever happens in relation to that particular report beyond that is extraneous.
What are they then missing out on? What is the impact that they’re not able to provide and maybe this goes to – can we actually measure the impact of a presentation that’s provided about an audit report? Or at a minimum, can we measure the level of engagement that we have in those presentations which will then go to helping understand that impact?
So I’ll take your first question first. I think that would be a great little project for us to do Yusuf, to actually measure the benefit those audit offices get from having presentations accompany the report versus the risk attached to those auditors general not taking that approach to have presentations for the report.
It sounds like the risk is actually higher if you don’t do the presentation.
Yeah, which is what I would concur with. Yeah.
Ok, so what you’re saying is that some audit offices are not conducting that because they don’t see that as part of their mandate. Are we seeing any audit offices where some audit reports result in presentations and others don’t.
There are some audit offices that selectively do presentations on some of their performance audits and on others they don’t. And I suspect the basis of that decision making is they need to focus their resources on where they think they’re going to get the biggest bang for their buck. So they may say, for example, we’ve done ten audit reports this year. Which are the three that we want to do presentations on and they may say, ok, these three have the highest impact or greatest likelihood to impact the public interest, so we’ll devote resources to doing a bit of a campaign around these three, and the other seven we just can’t devote any resources to do.
So, that’s really interesting, right, because in our experience, the cost of conducting an individual performance audit often goes into hundreds of thousands of dollars. So if you’re spending half a million dollars – Australian dollars that is, to conduct a performance audit, and I’m sure in the US, Canada, the UK there’ll be similar numbers. If you’re spending half a million dollars let’s say to conduct a performance audit. What would a presentation cost in real terms? 10k, 15k, 20k, 25,000 bucks?
Well, I don’t think it’s actually the financial cost that’s the biggest determinant would be my observation. It’s more around freeing up the time for the individuals who had been involved and have the greatest insights to be able to go and deliver that work,
But that then comes back to financials. Right. Because if you, have a bigger team then you can actually free that up. So it then becomes do I get one extra FTE in my organisation to be able to free that time up.
So taking it back to financials, as was your question. If there’s quite a substantial amount of investment in each performance audit in terms of public monies, as you’ve just described, then there’s a fairly small component of that that’s actually required to deliver the presentation attached to that performance audit.
And so back to can that presentation provide high levels of impact, particularly if a small sample was selected and we can actually get the message out?
Absolutely. Particularly if you’ve got a captured audience that can can make a difference and can actually go away and do something with that presentation.
So I wonder if, because often we see the cost of producing an audit report is X, and that’s public reported by many audit offices. I wonder if capturing the cost of all of those ancillary activities, presentations and the like can be captured there as well. And then overall, there’s a consideration as to what are our performance audits costing us, because that is, from the sounds of it, that is an important aspect of the performance audit process. That is, getting the message out to the right people so that the work that was done can be used as widely as possible.
Exactly and if we come back to the original proposition about how do we or the original question, how do we maximise our impact if we’re able to say we did these presentations on our performance audits this year at a cost of X, and we’re able to distribute the message to so many more people, which would influence our impact. Then there’s a really simple equation, and you may for example, be able to develop a business case on that basis. And when you come to your appropriation from the Parliament, perhaps the next year, you can say that we just need a slight adjustment to our appropriation. But look at the impact we will have on the basis of that.
Assuming that there are still some internal auditors listening, because we have been going on about performance audit for a while, but assuming that they are and hopefully you have because I think there’s a lot to be learned from what performance auditors are doing in terms of the way in which they’re using the work that they are doing to provide impact across the range of entities that they focus on: presentation, answering questions that other people might have, the use of the various media – you’re obviously not going to use social media to explore the results of your audit , but when you have internal communications group, so Intranet portals, lots of people are using internal social media channels, Teams and a range of other mechanisms for communicating with individuals within the organisation. Then we should be looking at how we can use that to spread the message of, what it is that we’re doing and what the impact of our reports is and where other entities within our organisation, other business units, other people can benefit from that. So this is obviously going to be an area that grows a lot more. But if we can, we do spend a lot on conducting our audits. I’m not saying we spend more than the value that we produce, but it’s a lot of money that is being spent, and if we can communicate that and reach as many people as possible, we would have better return on the investment in ourselves as internal auditors or performance auditors.
So the three things we spoke about today in no order of importance: remedial action, go beyond it so past, present and future, using reports for maximum impact through presentations, through the use of infographics, through better use of English language, through ensuring that we’re able to get the reach that we need and then following up and ensuring that when we do follow up we consider emerging needs and the potential for changes in what the expectations are around those actions.
If you really want to maximise the impact of your report. Firstly, grow your reach, whether that be through new audiences or using other mechanisms such as social media to tap into those audiences. The second thing is make your report digestible and understandable to your readers.