Episode 33 | Ten common areas to focus on when using data for audits

The Assurance Show
The Assurance Show
Episode 33 | Ten common areas to focus on when using data for audits
/

 

Show Notes

In this episode, we discuss 10 of the more common areas of focus in using data for audits.

  1. Get over the fear of using data
  2. Focus on the objective and develop the hypotheses
  3. Check for previous work, to avoid duplication
  4. Source the data – internal and open
  5. Prepare the data and check quality
  6. Perform the analysis, prove/disprove the hypotheses
  7. Conduct QA and business validation
  8. Report and visualize
  9. Share for potential reuse (audit team, management, other agencies)
  10. Lessons learnt: governance, quality of data, timing, process, etc.

 

Transcript

Narrator: 

You’re listening to The Assurance Show. The podcast for performance auditors and internal auditors that focuses on data and risk. Your hosts are Conor McGarrity and Yusuf Moolla.

Yusuf: 

Today we are talking about 10 of the most common areas to focus on when using data for audits. This is not about data in audit more generally. We’ve found and what we’ve come to believe is that the best way to learn and explore is on an audit. General topics are quite important, more broadly in terms of how data is used across the audit function. But delivering on an audit is where most of the focus should go.

Conor: 

Absolutely. And you’re able to work in something that’s tangible, as opposed to just conceptual stuff that you might learn in the classroom.

Yusuf: 

In our experience, the 10 items in the list today are going to be the most common areas that you need to get across and to focus on when using data as part of your audits.

Conor: 

Number one, this is common where auditors are starting to use data. And that’s related to the fear that exists in using data.

Yusuf: 

The first thing is to get over that fear, and be okay with making a mistake, be okay with not necessarily delivering an outcome. And know that you can do most of this work yourself. Every internal auditor, every performance auditor, should be using data directly. And so getting over that fear is important.

Conor: 

What’s the second thing then we should be thinking of.

Yusuf: 

So we’ve got over the fear. We happy that we can do this ourselves. Before we jump into data or tools or any of those sorts of things, we need to focus on the objective for the specific audit. And then develop the hypotheses based on that objective. We’ve spoken about the hypothesis based approach, and we think that that is the best way. So the audit objective, and the hypothesis that you’re going to be creating on the back of that audit objective, that’s the first thing that you need to focus on. The first thing you need to think about after you’ve got over the fear.

Conor: 

You’ve settled on the hypotheses you want to test. What do you need to think about thirdly?

Yusuf: 

The third thing is, has any other work been done in this area? Has any other work been done either within the audit team or by a first-line team? We want to avoid duplication, and so we look both within the audit team, which is the easiest place to start, but then also across other teams. So data warehouse, teams, management teams, And then work out what is it that has been done before that I don’t necessarily have to repeat. Now sometimes you want to repeat the work that’s been done to confirm that that work is correct. And you can either do that through a controls type approach where you talk to them about where they got the data from, how they went through their QA, etc., or you’re going to redo some of that work.

Conor: 

So the key thing there is firstly, asking what’s been done before, secondly, trying to leverage that where possible. Number four. What should we think about next?

Yusuf: 

Now we want to source the data that we need. This is a common challenge, getting hold of the data, particularly when it’s internal data that you need. You want to do this well in advance of the audit. There’s a bit of a balance there. You want to have your audit objective and your hypotheses first, to a large extent, but the audit objective doesn’t necessarily have to be settled. What we’re seeing a lot more of nowadays is, refining the objective and refining the hypothesis as you go. A bit more agile approach where you come up with something that’s 70 or 80% of the way there initially, and then you refine that as you go. When you starting to think about what an audit might be, and you’re working through initial planning, it’s at that point that yo need to at least have the discussions and figured out where the data actually sits. At a high level, you need to source the data, find out where the data is both internally, and then what open data you can use that, that exists, that might be able to help.

Conor: 

And you want to do that as soon as possible in your audit.

Yusuf: 

You want to do that as soon as possible within the planning phase of your audits.

Conor: 

Number five. What should we think about next, once we figured out where the data may reside and we may even have obtained an early snapshot of it?

Yusuf: 

This is where you start to prepare that data and check the quality of the data. We will have a separate episode on this, but this is importing the data, checking the quality of the data, making sure that you have the data that you need, there’s no missing fields, that you’re able to join all the data up. Sometimes it’s difficult to actually join the data because you don’t have the keys to join the data. So you don’t have something that links datasets together. In your preparation and checking quality, that’s one of the more important things to check is what data am I going to have to join together? And can I join that data together? Or am I going to have to come up with something else, get more data or come up with some sort of way in which to merge them. If anybody else has used this data before you do want to see whether they’ve documented anything, are there any known issues with the dataset that we’re getting, talk to the person that’s providing the data to you as well and ask you about any known issues that might exist. In a data project, upwards of 60% of the effort is actually preparing that data. In an audit context, we’re talking between 40% and 60%, because there’s other things that we need to do that do take up a little bit more focus like QA and business validation. But preparation of data is an important step. And you could easily see this as being around about half of the effort that you need to put in. You do need to make sure that you don’t put too much of effort into preparing the data either. Don’t try to fix everything at this point. If you know what the analysis you’re going to be doing, you focus the preparation of data on that analysis. So don’t try to prepare the data in a way that every field is a hundred percent correct. Just get what you need initially. And you can refine that as you go. So if you prepare the data now, go through the analysis and you find a particular field that I need to fix. It’s better to come back to that field and fix it or come back to that field and do something with it than to try to fix the hundred fields that you found in the first place, because that will take a lot more time.

Conor: 

So you’ve got a good handle on the data. You’re reasonably satisfied with its quality. What comes next?

Yusuf: 

This is where you then perform the analysis that you need to do in order to either prove or disprove hypotheses that you’ve set out. You’ve cleansed the data, joined the data. Now it’s about, I’ve got a hypothesis, I’m going to try to prove it. Conducting the specific tests, look for exceptions, , or disprove it. And this is not a decision that you make upfront. The data will determine whether the hypothesis is proved or disproved. But here you want to answer that question by looking at the data that you have at hand, and then looking at exceptions that might arise. Looking for exceptions is quite an important part of the analysis process for an audit. A little bit different maybe to the way in which analysis is done for first line more broadly. And so with audit analysis, you are looking for any exceptions that might exist that then may either disprove the hypothesis, or that may just highlight a few anomalies or discrepancies that may need to be investigated further.

Conor: 

You’ve got an initial view now of whether your hypotheses have been proved or disproved. Where do you go next?

Yusuf: 

So next thing is to conduct QA. So there’s various levels of QA that you need to conduct. QA your own work, so self QA. Get a peer to do some QA for you, ask questions, maybe do a bit of technical QA as well for you, check the formulae that you’ve used, etc. And then you go on to audit manager QA, so whoever’s leading the audit or managing the audit would do a bit of high level QA. And then you want to do business validation as well. So it’s, at this point, you want to start engaging with the business to understand whether what you’ve come up with and what you’re seeing is actually making sense. DOes it align with what they would have expected to see? ,Is there anything that you’ve done that they may be able to explain the reasons for, so any exceptions that you’re finding, can they explain why that might be either technically because the data needs to be looked at in a particular way or there’s something else that you haven’t done or something that you haven’t considered?

Conor: 

After your QA, you’re comfortable with the accuracy and validation of your results. Where do we take that next.

Yusuf: 

JUst to note that each of the ones that we’ve spoken about, sourcing the data, preparing the data, performing the analysis, conducting QA, they would all be iterative. So it’s not a linear process. You don’t just source prepare, analyze, and QA. There would be some sort of feedback loop that you working through at each step. So sometimes you do the QA, the QA points to something in the data that you haven’t checked properly or haven’t got. So you got to go back to those steps. So those steps are iterative. And then once you’re comfortable with all of that, this is when you start to put together your reporting and visualization. A lot of this will be explanatory reporting. So where you want to explain things for a report that is being put together, sometimes you want to exploratory particularly where somebody else in the audit team, they’re going to be using the data. You may need to have some sort of exploratory analysis as well, but mostly what we see with audit visualization, with auditviz, is explanatory visualization. The difference there is exploratory is where you create some sort of dashboard where you can play with the different variables and see what the results might be and filter your data, etc. Explanatory is where you creating something that is reasonably static. It’s not something that is meant to be played with a lot. But it’s more about creating some sort of explanation that can be used for your audit report and for your discussions with key stakeholders.

Conor: 

We followed all these steps so far, noting that there is a lot of iteration as we go along. We’ve had a rigorous process. Hopefully we’ve got some really good outcomes for our audit through the use of data. How do we maximize those outcomes?

Yusuf: 

The next two off the 10 items we want to talk about are exactly what you said. Maximizing the first of those is sharing for potential reuse. And that could be within the audit team or outside of the audit team. Let’s focus first on the audit team itself. So you’ve done a whole bunch of data cleansing work. You’ve sourced data. You’ve understood how the data works. You’ve gone through a range of business validation activities. For both internal data and open data, the data that you use and the way in which you use it, and what you learned from there can be useful for future audits . And this is very often, right? and We’ve used this example a few times now, but complaints data and feedback data from customers where we’ve used that once. And then we were able to reuse that data across three or four audits afterwards, to enhance our audits in different ways, So that’s an example of where you’ve done some work on some data you learnt a lot about it. And so you want to then share that with them broader team, verbally, but then also document what you’ve done so that somebody can pick it up later on. That’s internal to the audit team. The other thing most of the work that we do as audit does is going to be once off. Most forward thinking audit teams don’t repeat their work year on year. If you’ve done some work that might be useful to the business, and quite often we see this, right. You go to a first-line stakeholder and they say, Oh, I really love what you’ve done here, can you share it with me? WE should be sharing that, we should be sharing the results of our data work. The only exception to consider there is anything of a forensic nature where there might be some sort of fraud that’s been identified, then you’re probably not going to be sharing anything. And the team that’s receiving the data or the workflows or the analysis steps that you’ve put together, they need to be aware that they can’t rely on your work and they need to check it and make sure that it’s workable for what they use. Otherwise you could be creating something and then putting it into the business and it becomes a control that you then have to come in later and look at. With those little fail safes in place, there should be no reason for you not to share. And then the last one is any lessons that we’ve learnt that would be useful for the audit team more broadly. And so this is around how we have governed access to the data, so who had access, who didn’t have access, how that all worked, where we got data from what the quality of the data was overall, how long it took to get that data, how many cycles we had to go through when we were iterating. Anything that will be relevant to the broader team, that would be useful for them to think about in future audits. We learn every day, right? That’s just what it is to be an auditor. And we learn every day when we’re using data in audit. These key lessons that we’ve learnt are useful to share, because it just means that we can be more efficient, more effective and produce higher quality the next time we get to use data on an audit.

Conor: 

One of the key things there is to document and note don’t these lessons learnt as they arise, and then do a review at the end of your audit as well. Because quite often people say: oh that was a good thing, I’ll get to that at some stage, and then they forget about it because they get busy on their next audit or the next project.

Yusuf: 

Yeah. So most of what we spoke about today is relevant for both internal auditors and performance auditors obviously, but I’ve been thinking about it a bit more from an internal audit perspective when discussing it. Is there anything that might be a little bit different for performance auditors or they might need to think about some of these things a little bit differently.

Conor: 

Nothing that would be very different per se. But I’d probably just highlight the importance of sharing the lessons learned and really maximizing what you’ve observed across your performance audit team. And even with other jurisdictions that are probably going through the same new challenges with using data in their work as you are. So, not sharing the pain, but sharing what you’ve learnt.

Yusuf: 

We spoke about 10 things there.

Conor: 

The 1st one we touched on is getting over the fear of using data. The 2nd one was having a good, solid focus on your objective and your hypothesis from the outset. Number 3 was checking before you even start to grab any data, whether any work has been done before and how and if you can leverage that for your particular audit project. Number 4, then actually go about sourcing your data, maybe internal, maybe external. Number 5, once you’ve got the data, you need to spend a fair bit of time preparing it. Number 6 , this is where you actually start doing your analysis and start testing your hypothesis, using the data, Number 7, doing your QA and business validation. Number 8, where it all comes together, reporting and your visualization and making sure that all the good analysis you have done is actually hitting the mark. Number 9, what have you learnt on your project? How can you share those lessons. And lastly, touching on data governance, how many cycles you had to go through to prepare it and so forth, things that you can take forward and apply broadly across your future projects. So they are the 10 things we touched on today. Some of them we’ll pick up and deal with in a bit more detail in future episodes.

Yusuf: 

Fantastic. Thanks Conor.

Conor: 

Thanks Yusuf.

Narrator: 

If you enjoyed this podcast, please share with a friend and rate us in your podcast app. For immediate notification of new episodes, you can subscribe at assuranceshow.com. The link is in the show notes.