Do you agree with the analysis presented by the ICO, in this document?

What we agree with

We agree with the general theme of the consultation.

We think it is right that the accuracy principle is central to the reliability of GenAI, whether used in business or consumer settings. The consultation makes it clear that not all data needs to be accurate. The level of accuracy is dependent upon the nature of the information and the way the model is going to be used.

Here are some of the things we agree with:

• A question of degree: If the data is relied upon by users as a valid source of information which has a material impact on the lives of data subjects, it is more important that the information is correct and developers/deployers should be held to account with greater rigour in these scenarios. The extent of the accuracy principle’s application is a question of degree and can change based on context.
• Purpose limitation is key: There has to be a clear and present connection between the purpose limitation and the accuracy principle. One cannot exist without the other.
• Accuracy assessment: There should be more of an onus on developers to make it clear to deployers the accuracy (statistical or otherwise) of the training data and whether it is sufficient for how it is going to be used. Deployers must consider how this data can impact an individual and make it much clearer if the data is statistically accurate so that this can be taken into account by the users of the data.
• Accuracy disclaimer: We agree that statements regarding accuracy, in particular where the developer has concerns about (statistical or UK GDPR) accuracy would be a good thing and would help with compliance with the transparency principle. However, it could give rise to negative behaviours among developers and deployers. For example, it could lead to them making little effort to ensure accuracy and instead simply deploying their inaccurate GenAI models with lengthy disclaimers and usage limitations. We challenge the ICO to clarify how it will prevent this negativity bias in a model where it appears possible to disclaim accuracy at the expense of data subject rights.

What we disagree with

We disagree with the following points in the consultation:

• Terminology: The explanation of the differences between statistical accuracy and the accuracy principle in the consultation is not sufficiently clear. We understand statistical accuracy to be closer to mathematical accuracy, meaning that whether accuracy is capable of being reduced to a mathematical measurement based on a probability score when measured against its true value. This is clearly different to the accuracy principle in UK GDPR which is significantly (and deliberately) less precise, being more concerned with concepts regarding the nature and state of the data rather than a system and its output. This distinction would have benefitted from more examples and practical scenarios in the consultation.
• Responsibility: It is important for GenAI developers and deployers to not only put a warning or disclaimer on the information but to make it clear to whom a user can complain about the output of GenAI and to have some level of human intervention/monitoring.
• Creative AI does not have a lower accuracy risk: The consultation says that where GenAI models have a purely creative purpose, the requirement for accuracy is unlikely to be a first priority. However, as can be seen from the research by The Bulimia Project even AI which is intended for the sole purpose of creating artificial imagines can have significant negative impacts on body image and mental health, particularly in relation to young people. We think it is unhelpful of the ICO to suggest that “the more a generative AI model is used to make decisions about people, or is relied on by its users as a source of information rather than inspiration, the more that accuracy should be a central principle in the design and testing of the model”. The modality of GenAI use cannot be easily dissected between information and inspiration; often both may be a feature in a user’s activity.
• Excessive accuracy: Excessive accuracy in limited data sets can bring its own problems, e.g. have biases. Certain illnesses come with specific race or certain jobs more associated with a specific gender – generalisations. The consultation does not address data bias or accuracy in this context.