Summary

We really like AIME and think it is a good first step. It will help those organisations and business which need help figuring out what good AI governance looks like. There are some gaps and a few areas where we recommend improvements. All of that is summarised below and set out in more detail in this response document.

1. What are your overall impressions of the AIME tool?

Our overall impression is positive and we commend the work of DSIT and the AI Assurance team. However, we have identified several areas which we think could be better. Our response sets out in greater detail those areas which we thought were good and those sections we think require improvement.

The reason for our overall positivity arises in part from the AIME tool itself (it is a great initiative) and in part from our collective awareness of the national appetite for such a tool in the context of increasing demand for AI. The UK is at the dawn of a new technology revolution and the businesses operating here need a useful benchmark of good practice to help deliver certainty and confidence when using AI. As a result of the rapid growth of AI systems and the UK government’s intention as set out in the AI Action Plan to “unleash AI” into the UK business community, guidance on the use of AI systems is desperately needed and the intentions behind the creation of the AIME tool are laudable.

Things that are good about AIME.

The positives identified by this group can be summarised as follows:

  • Simplicity is its strength: The structure and simplicity of the tool will make it very useful for SMEs or organisations which are less sophisticated with their use of AI. It is a good hygiene test for the many businesses arriving to AI for the first time.
  • The decision tree is helpful: The use of a visual decision tree which does not involve lots of text was well-received by our group, especially those who are SMEs.

Things that need improvement with AIME.

The criticisms identified by this group can be summarised as follows:

  • Separate users and developers to get best results: The way that users of AI engage with AIME will be very different to organisations involved in the AI supply chain, particularly developers or sophisticated deployers of AI systems. We think there should be one version of AIME for the general business population (which we call “users”) and another for developers. We hold this view because our consultation members who are SMEs specialising in AI found the questions too simple and sometimes unhelpful. Furthermore it was felt that a number of the questions would not be able to be answered by users and appeared more intended for developers. Most participants also thought that the AIME tool could benefit from focussing on the nature of the AI tool being assessed by a user. Some AI tools are inherently more complex or risky than others and it was felt that AIME would benefit from being able to make this distinguishment. The views in this paragraph were almost unanimously held across all group participants and we think this one is of critical importance to ensure success of the project. We expand on this in our answers and point out sections where the distinction is most apparent.
  • Outcome of AIME: Many contributors struggled to identify what the overall benefit was to AIME on account of the lack of certificate, score or similar outcome. We were uncertain about whether AIME serves as a self-assessment mechanism, a compliance reporting tool, or a means to benchmark against best practices. This needs to be clarified. This issue becomes increasingly pressing if successful completion of AIME is a factor in government or private procurement processes.
  • Guidance: Without sight of the detailed guidance many participants felt our review of AIME was incomplete. If the tool has clear guidance and support (in the form of examples) clearly communicating what good looks like we felt that the tool would be significantly improved.

Things we are unsure of about AIME.

Areas of unresolved debate about AIME by this group can be summarised as follows:

  • Binary questions: The view of our group about the binary nature of answers was mixed. Many participants were of the view that the binary nature of the questions had as many drawbacks as it did positives. For the most part, the group’s members felt that the binary answers were too simple and overlooked the complexity in the answers (we expand on this in our answers to other questions). This view was prevalent among those who had greater specialism in this topic or who represented larger organisations. However, a lot of the group’s members liked the simplicity offered by the binary questions, particularly those who represented SMEs or were less familiar with the topic. It may also be possible to use AI text processing for answers in the form of a chat bot rather than binary questions.
  • Missing topics: Many times our group identified topics which we felt were important to good AI use by any organisation but not covered in the AIME tool. We were unsure whether their inclusion would improve AIME (some felt it would) or whether it would over-complicate a tool which is designed to be simple (some held this view). Missing topics that we identified include a reference to use of AI in a sustainable way, and analysis of risks of IP infringement.