Part two

The group which considered this section of AIME believed that most of section 2 was clear and the questions were not difficult. However, they also had the following observations:

• What is an AI policy? Just like with the response above regarding AI systems record, further guidance is required to help SMEs understand exactly what an AI Policy is. Recognition also needs to be given to the fact that an AI Policy may be comprised of several documents not just one singular document called “AI Policy”.
• Who is the policy aimed at? The group’s overall impression on the section on AI Policy is that it is primarily inward-looking in nature i.e. towards employees or consultants of an organisation using AI. For example AIME Q2.2 asks whether an AI Policy is available and accessible to all employees. The group suggests an AI Policy should not just be accessible to employees and may be relevant to other parties who are external e.g. customer and suppliers. It may be the case that the AIME tool’s reference to AI policy is too narrow.
• Most difficult question in section 2: The group felt that AIME Q2.3 (“Does your AI policy help users evaluate whether the use of an AI is appropriate for a given function or task?”) was the hardest to answer because it could be widely interpreted. Guidance is needed on how to determine whether use of AI for a task is appropriate. Some case examples would be helpful.

The group focussing on fairness spent a lot of time debating how this is likely to be applied in practice. It is clearly one of the more complex areas of AIME because it is easily misunderstood.

• The problem of direct impact. AIME Q3.1 is difficult because directness is a flexible concept which permits a high rate of error based entirely on the judgement of the user. The examples given are also quite widely drafted and the group were of the view that most AI use types would be caught in this description. AIME Q3.1 would actually be easier if the word direct was removed because most users will equate directness and impact as equal terms. In other words, for the average user an indirect impact is no impact at all. Have said that, we question whether it is right to delineate between direct and indirect impact. We think it may be better if the user is entitled to choose between different types of impact. For example, if part of the impact of the AI is to remove jobs from the workforce, this could (arguably) be an indirect impact within the scope of the AIME questions as posed but perhaps should still be considered as part of good baseline governance. An alternative would be to rephrase AIMEQ3.1 as, ‘Have you identified who will be impacted by the AI you are using, and how they will be impacted?’ To cover the full scope of potential unfairness, the list of examples should be broadened to encourage users to think about distribution of benefits arising from the AI use, and less direct impacts of the AI. The example of employees being affected by reduction in jobs, and the demographics of those affected, would prompt users to think about the wider impacts of AI use.
• Being clear about what fairness means. In the context of AI, our group considered that fairness relates to how AI impacts a particular user or group of users. It is mostly about how the organisation decides to implement the AI and what that means for those affected by the decision. In this sense it is very different to bias, which is a broader, more objective, concept relating to the input or training data and the effect that has on the AI model. To help clarify our point, we consider that a totally unbiased AI system can be used in an unfair manner by an organisation if it deliberately or unintentionally uses it to harm or discriminate against certain types of people; this use would be unfair and responsibility for it would rest only with the organisation. Conversely, an organisation may seek to deploy AI systems in a fair manner and have in place lots of checks and balances to ensure that it does so, but use an AI system trained on biased data. In this latter example a user or group of users is harmed, but not on account of unfair use. Instead, it is on account of failure to mitigate bias, responsibility for which may rest with many organisations in the AI ecosystem including AI developers and providers. The guidance to AIME Q3.2 says "We differentiate unfairness from bias, where bias is statistical phenomenon that is characteristic of a process such as decision-making, and unfairness is an outcome of a biased process being implemented in the real world." This is unnecessarily wordy and, whilst the intention may be similar to what we have described above, it falls short.
• Fairness continued: The group analysing fairness concluded that what matters most is that users know, before they start to use an AI system, what that system is designed to do, what sort of outcome it can give, and where they can access support if needed. If this sort of information is provided up front, the likelihood is that the AI system’s use will be fair. To achieve this, organisations should be encouraged to be transparent about the use of AI by providing notices or policies to users. The group acknowledged that there is some crossover here with AIME Q4.1 but would observe that this topic (the topic of impact on individuals) sits more neatly in AIME section 3.
• Monitoring fairness is tricky: The group considered that AIME Q3.4 will be difficult for organisations to implement. To successfully monitor fairness requires a clear understanding of what it is they are being asked to monitor. Many of the participants in the group reviewing this section of AIME thought that SMEs would just conflate monitoring fairness with complaint monitoring generally which may be the only method by which unfairness in AI systems’ use is captured. It is likely to be reactive unless tested voluntarily and pro-actively. It would be appropriate for the DSIT guidance to recommend that this task is designated to an appointed representative in each organisation.

The group reviewing section 6 identified the following questions as difficult to answer:

• Separate users and developers: AIME Q6.1 this question is most likely targeted at AI developers and deployers. However, if that is the case, it is not obvious from the context and in fact most organisations will be involved in assisting in model development by continued use of the system. This group thought that sort of model training was not the target of AIME Q6.1 but, if that is the case, the question should be clearer.
• Unrealistic for SMEs: AIME Q6.6 (“Do you document details about the data preparation activities undertaken to develop your AI systems?”) is problematic for smaller organisations. We appreciate that the legal standard requires what is posed, but the question does not recognise the reality of life for small businesses. They generally struggle without legal departments to document contracts with all third parties who process personal data. The majority of staff would not be able to answer this question, as they are often unsure of who deals with contracts. In addition, AIME Q6.6 would be better suited to AIME section 8 (data protection).

Bias easier to grapple with than fairness: the group reviewing section 7 considered the issue of bias mitigation in detail. This group considered that the topic of bias mitigation will be easier for most organisations to understand than fairness (a connected topic, at AIME section 3) but conversely is harder for most organisations to manage. This is because most organisations completing the AIME tool will not have access to or be able to influence training data sets for large AI models at a scale that could impact bias. The AIME tool does not appear to take this into account but it should do so. This is one of many good examples where separation between users and developers would be advantageous.

In particular the group picked out AIME Q7.4 “Do you have processes to ensure compliance with relevant bias mitigation measures stipulated by international or domestic regulation” which they believed would be too difficult for most AI user organisations to answer. This is because, based on the experience of this group, many providers of AI systems will not provide information relating to any bias in their systems. It is extremely difficult to eradicate bias and so it can be argued that these questions, whilst not inherently unnecessary or superfluous, are potentially inefficacious.

Too wide: This group was concerned that the scope of the bias topic is very large and the nature of the questions would yield imprecise answers on account of unfamiliarity with how to tackle the answer. For example, in answering AIME Q7.1 (“Do you take action to mitigate against foreseeable harmful or unfair bias related to the training data of your AI systems?”), organisation Y might take a single action of mitigation in relation to the limited AI systems it uses enabling it to tick yes to Q7.1 (a). In contrast, organisation Z might have a suite of mitigation measures it implements against unfair bias for a majority of its AI systems but will only be able to tick Q7.2 (b). Organisation Z has the more comprehensive AI management system but their answer will generate a lower score than organisation Y (on the presumption that’s how answers will feed into the rating). This means if an organisation genuinely wants to use the tool to understand the state of their AI management systems and practices, their score could be inaccurate unless they think deeply about their answers. If they don’t, AIME could generate a score which gives them unfounded confidence in their AI management systems and practices.

Clean bill of health: The group reviewing section 8 broadly felt that the questions were not difficult to answer, with the exception of AIME Q8.5. The term “interference from third parties” seems far too broad and could create confusion with users.

What does good look like? The group reviewing section 9 was of the view that generally AIME Q9 is not difficult to answer, but the questions could be widely interpreted. The challenge when reviewing AIME Q9 is understanding what is intended. The tool would benefit from guidance and examples, as is it not clear what ‘good’ and ‘better’ would look like and it is not clear what a reportable ‘issue’ may be.

For example:

• Who is an external third party for reporting purposes? AIME Q9.1 requires reporting mechanisms for all employees, users and external third parties. It is not clear in this context who an ‘external third party’ would be. Even in large organisation there is not likely to be a method for external third parties to conduct reporting of system failures.
• The poor DPO. Regarding AIME Q9.3 (“Have you identified who in your organisation will be responsible for addressing concerns when they are escalated?”) it would be helpful to provide guidance on who the responsible person should be and in particular what skills they should hold and how they would be expected to fulfil the role. In a small organisation there may not be sufficient resources or risk to justify a dedicated role, whereas a larger developer which provides tools with a more significant risk profile may require a dedicated role. One participant said: “I could be wrong but – as the person who is already DPO and other things besides – the business is just going to make me the point of escalation for AI issues. Is that what you think DSIT is getting at here? If not me then who?”
• Can one be unintentionally transparent? Whilst ‘transparency’ in AIME Q9.4 is defined, it is not clear what ‘meaningfully’ adds to that definition, and perhaps would benefit from widening the definition to include what is intended by ‘meaningfully’, with examples. As with answers to other questions, particularly those regarding fairness or AI policy, we think transparency is an important thread that should run through much of the AIME tool when encouraging good compliance.
• Time scales: When considering the time scale in responding to concerns (AIME Q9.5), the group was of the view that an appropriate timely response will very much depend on the severity of the issue being reported. The current requirement for all issues to be responded to within 72 hours does not address the fact that some issues will require a response than others. We appreciate the rationale for using 72 hours and, subject to what we say above, it provides a useful benchmark.
• What goes in the response? AIME Q9.5 is unclear about what a ‘response’ would entail. Would an acknowledgement be sufficient or is it intended that the issue will be resolved within 72 hours? Some concerns may just require an automated response, acknowledging receipt, but others may require a fuller response, and a more timely approach. Over reporting may lead to reputational damage, and so the required response should take into account the level of control the business has over the tool, such as whether a deployer or developer.
• Take action? Once a concern is reported, AIME Q9.6 refers to documenting the report, but perhaps this would be better to include a process to follow in addition to documenting the report. Merely recording the issue may be good practice, but considering what action may be needed following the report would nudge the user to ensure any mitigation etc was put in place.

The group which considered this section of AIME had the following responses to question 6:

• Most of the questions seem appropriate but AIME Q2.2 could be redundant. Accessibility is the main concern that could be addressed as part of broader governance practices rather than requiring a dedicated question. It could perhaps be rephrased as “Is your policy accessible to those who need to access it in a clear, easy-to-use format?”

The group which considered this section of AIME continued to hold the view that fairness and bias mitigation are conflated by AIME. None of the questions were obviously redundant but, continuing the theme, they observed that:

• Fairness is still tricky: The questions in AIME Q3 are thematically closer to questions of bias rather than fairness.
• What is fairness in AI? AIME Q3.2 requires a better explanation of fairness (this may form part of the guidance) which was tackled in the response to question 5 above. In our view fairness is more about how the AI’s use impacts the individual exposed to the AI’s operations. For example, bias (as AIME points out) is more closely aligned to a statistical phenomenon in the data sets on which the AI system operates. What matters most is that those using the AI system are aware they are doing so and what the most likely outcomes of that use are, so that they can make informed decisions about what happens next. The most obvious method of transparency is something like a published AI policy, but other more innovative options should be encouraged too. The means should not determine the ends; what matters most is that users know, before they start to use an AI system, what that system is designed to do, what sort of outcome it can give, and where they can access support if needed.

Merge AIME Q4 and Q5: The group which considered these sections of AIME did not think any of the questions were obviously redundant but, continuing with previous themes, they observed that it is very easy to confuse section 4 (impact assessment) and section 5 (risk assessment). While the questions in AIME section 4 are important, this group thinks it makes sense to think about the impacts as associated with risks (e.g. see the repeated use of the word "potential" in this section). They suggest a framework tying these together in a more signposted way or, if they are intended to be different, AIME should make it very clear how they are different.

Quality and utility: The group that reviewed section 1 of AIME did not feel that this section was overly burdensome but did query whether it was realistic to expect organisations to have an AI systems record in place. There is the potential for great disparity between AI system records for all organisations, ranging from none at all through to complex system records. The group suggested that DSIT should work on the assumption that most SME organisations would not have such a record unless they were directly involved AI development or deployment. As we have said elsewhere, quality and utility matter are very important for good governance frameworks.

Clean bill of health: The group that reviewed section 2 of AIME did not feel that this section was overly burdensome or unrealistic for the target audience and had nothing further to add beyond previous comments about section 2 (a few of which overlap).

The group that reviewed section 3 of AIME thought that some questions may be overly burdensome for start-ups and SMEs deploying widely-available AI tools, although the accompanying DSIT guidance may provide clarity over this.

The group that reviewed sections 4, 5 and 6 of AIME thought that some questions may be overly burdensome and commented as follows:

• Separate users and developers: the AIME tool may be overly burdensome simply because it asks questions in each of these sections that should probably just be posed to developers. It is unduly burdensome on SMEs to be expected to have in place a AI impact assessment or to understand what AI training involves. This group repeated earlier calls for AIME to segment users and developers.
• Missing guidance: Answering this question is also difficult because the group observed that they are looking at the tool without the associated guidance. It may be the case that, with the guidance, some of the concerns above for non-developer users may be allayed.

The group that reviewed section 7 of AIME had the following observations:

• Bias a burden for some SMEs: The AIME tool is relatively straightforward and not inherently burdensome or unrealistic, it does have the potential to be more burdensome to start-ups and SMEs. This is in part because resource limitation will play a factor in the scope of due diligence that organisations are able to carry out, as referred to in AIME Q7.1 and Q7.3. Additional difficulties may be encountered by small organisations who may not have established, or have the ability to maintain, the relevant processes and measures.
• SME resource issues: A lack of technology and/or legal literacy may also increase the burden on small organisations, particularly where they are simply AIaaS users. This is noticeable in relation to AIME Q7.3 and Q7.4, where recognising what constitutes appropriate due diligence, or understanding which international or domestic regulations should be consulted on the topic of bias mitigation, is likely to be prove difficult.
• AIME Q7.2 and 7.3 are unrealistic for the stated target audience of AIME, namely, start-ups and SMEs. Consequently, the inclusion of these questions is unlikely to provide any helpful insight into whether an organisation adopts responsible AI management practices.

The group that reviewed section 8 of AIME had the following observations:

• the AIME tool in this section is not burdensome or unrealistic per se.
• a lack of technology and/or legal literacy may also increase the burden on small organisations, particularly where they are simply AIaaS users. This is noticeable in relation to AIME Q8.3 and Q8.4 that presuppose some form of prior data protection knowledge and they noted that many of the questions in AIME section 8 (8.2 – 8.4) do not specifically relate to AI, which may confuse some users.

The group that reviewed section 9 of AIME had the following observations:

• The questions are generally not overly burdensome for organisations to answer but the lack of guidance may make it difficult to all organisations (regardless of size) to understand what is required in terms of having reporting mechanisms for external third parties to report concerns and system failures (per AIME Q9.1).
• The obligation to provide timely responses within 72 hours (per AIME Q9.5) may also be unduly burdensome depending on the nature of the issues reported and depending on what is required by way of response.