On the 4th of April 2021 the European Commission proposed a draft for the regulation of artificial intelligence systems that are to be used within the EU. Maybe you have even heard about it and have been losing sleep over the question "Does my AI model need to be compliant?", but haven't had the time to go through 130 pages of legal jargon in order to find out. Luckily, you have come to the right place. I will break down which applications are affected, what the requirements are and what scary things await companies in case of non-compliance.
The EU has several ways of influencing the governance of its member states so we need to get one distinction straight: the difference between directives and regulations. A directive is a rule that each member state has to then convert into national law until some fixed due date. A regulation on the other hand is legally binding directly in each member state as soon as it comes into effect at the EU level. The legislation we are talking about is a regulation. This means that, in principle, as soon as it passes, compliance could become necessary. In practice, it is likely that the law gets passed some time in 2023 and that it will come with some grace period to allow companies to get ready. At the time of writing however, the exact date is not known.
Of course, we first need to know what the lawmakers consider to be "AI". To quote the definition in the text: machine learning, logic- and knowledge-based approaches and statistical approaches. It's fair to say that this is as broad as one can make it.
However, rather than putting blanket restrictions on every type of AI system under the sun, the regulation classifies all applications of AI systems into three broad categories, based on the risk they pose: unacceptable risk (completely banned), high risk (need to satisfy stringent requirements) and low or minimal risk (no or next to no special requirements).
This category basically tries to capture most applications of AI that would clearly be unethical with respect to the values of the European Union. As such, all applications that fall under this umbrella will be completely prohibited within the EU. These unethical uses include the subliminal manipulation of individuals, the exploitation of vulnerable groups (like children, elderly etc.) or social scoring systems (like the infamous Chinese social credit system) that assign a measure of trustworthiness to individuals using data that has been aggregated for completely unrelated purposes or that unjustly discriminate against certain groups. The precise boundaries will likely have to be drawn by the courts. For example, will personalized feeds on social media platforms be deemed to be subliminal manipulation? Or, according to this law, when is the unfavourable treatment of some group justified and proportionate and when is it not?
Additionally, there are restrictions on law enforcement using real-time biometric identification (such as facial recognition) in public spaces. Essentially, the police are only allowed to use such systems when they can justify it for a specific and limited purpose, rather than for large-scale surveillance.
Now we get to the most interesting (and most complicated) category in the proposal. There are two lists of applications that are classified as high-risk. If your application is on either of them then you will need to satisfy a laundry list of requirements. However, there are exceptions and those apply if your application is already covered by legislation in yet another list. Let's try to break it down.
The first list consists of a host of existing legislations , and if your AI system is a product (or safety component of a product) that already needs to pass conformity assessments under the existing legislation, then it will soon additionally need to comply with the ``high-risk AI'' requirements. Think industrial machines, safety equipment and devices for medical diagnoses. For the brave readers, I have included the entire list below with links to the relevant laws. The important caveat is that the application is exempt if it already falls under the scope of any of the eight exceptions below, which cover many obvious applications such as automotive and aerospace. For these cases the European Commission intends to amend the existing acts in order to incorporate requirements for AI systems for the individual sectors.
The second list of high-risk applications consists of potentially harmful uses of AI that are not already regulated via third-party conformity assessments. This includes uses like tools for recruitment, university admissions, credit scoring or criminal justice (full details in Annex III).
So if after all these complicated rules you have finally figured out that your application actually is considered high-risk, what exactly does the law ask of you? There are quite a few different requirements so it is, yet again, time for a list!
You need to establish and document a process for identifying and mitigating the risks that your AI system can pose. This process needs to start at the beginning of the project and, even after deploying the model, you need to keep up this risk management process for as long as your AI system is in use. When defining the risks that your system might pose, you have to consider both the proper use of your system as well as ``reasonably foreseeable misuse''. You should also take into account that the users of your system may not know as much about how to properly use it as you do. Make sure that, after you reduce each risk as much as possible, the remaining risk can be ``judged acceptable''. Finally, you also need to show that you tested your system against all ``high-risk'' requirements (including the ones below) before you shipped it.
So unless you are hoping 90s-style ``Good Old-Fashioned AI'' is making a comeback anytime soon, your system is probably using data. According to the legislative proposal, you will need to explicitly document how and why you collected the data that you did, as well as how it was processed and labeled. The final dataset is then supposed to be ``relevant, representative, free of errors and complete''.
Importantly, you will have to demonstrate that the data is not in some way biased. The law specifically states that you will be allowed to process personal data in order to assess this, given that high standards of security and privacy protection are kept. Finally, you will have to address ways in which your data might nonetheless be incomplete and how this could be improved.
In order to prove that you really did implement the risk management process, data management and all the other required steps towards your compliant AI, you need to document everything in a central place that makes it easy for a potential auditor to assess. If your application is already regulated then this should be integrated into the documentation that is already required for the existing legislation.
The regulation makes it quite explicit what you need to include in your documentation, but, just to give you a flavor, here is a summary:
Essentially, you have to log the AI's operation in a way that's detailed enough so that one can use the information to see if the AI is operating as expected or if any changes are needed. If harmonised standards for the keeping of logs already exist for the application then you need to satisfy these. Plus, there are also some additional requirements for all systems that biometrically identify individuals.
Basically, you will need to provide a user with an instruction manual that explains how to safely operate the system and how to properly interpret the outputs. This manual should, for example, tell them what kind of accuracy and robustness the user can expect from their system, under which conditions the system might fail and how to properly keep the software updated.
It's important that at any point in time a human could override or reverse the decisions made by a high-risk AI system. Depending on the application, this could mean having flags for anomalous behavior or simply a good old red stop-button. The law also explicitly says that one should guard against automation bias (i.e. an operator's tendency to overly trust the AI's decisions), but it's not entirely clear what this will mean in practice.
You have to make sure that your AI system has ``appropriate level of accuracy'' to do the job that it's intended to do. Additionally, you need to ensure that your system is robust with respect to the issues that might occur in practice (new lighting conditions, broken sensors etc.). Lastly, if cybersecurity is any concern at all, you are required to defend against attacks on the AI. Famous examples of such attacks are adversarial attacks or data poisoning attacks.
As you can see, conformity with these requirements actually requires some quite elaborate measures and it certainly requires planning ahead. While the European Commission estimates the costs of compliance to lie on the order of a few thousand Euros, industry experts are highly sceptical of these numbers. My personal guess is that just the requirements on data quality alone are likely more costly than the 6000-7000€ that the Commission has estimated.
So that begs the question, what happens if you slip up? The fines can be quite serious. Providing wrong or misleading information about your AI system to the authorities can already cost up to either 10 million euros or 2% of annual revenue - whichever one is higher! Outright non-compliance with the requirements can even cost twice as much and, when violating the requirements on data management, even three times as much.
Finally, we have all other applications of AI. For these there are basically no requirements at all. The only exception applies to AI systems that interact with humans (like chatbots), use emotion recognition methods or deep fakes. In each of these cases, the human has to be informed that they are interacting with an AI or AI-generated content.
In short, if you suspect that your AI application may fall under the umbrella of ``high-risk'', the time is now to start preparing for compliance. The legislative proposal is of course full of many more interesting details, like how exactly the member states of the EU are to implement the measures or the special provisions given to small-scale providers and start-ups. But, since this article is complicated enough as it is, I will cover these topics in future posts. If you are considering an AI project in high-risk applications and have questions about the current state of affairs, feel free to reach out!