In an ever-increasing digital, algorithmic and automated world, the evolution and ultimate utilisation of AI has become a prime topic of debate. While AI has emerged across many strands of society – whether production lines, playlists, healthcare, hospitality or transport – so too have concerns, including about the lack of global standards for the legislation and regulation of such cutting-edge technology.

As such, the news that the European Union has issued a draft Artificial Intelligence Act, the first such piece of legislation to address AI issues, is a welcome advance.

The European Commission published its Proposal for a Regulation on Artificial Intelligence on April 21, 2021. The proposal addresses the risks of AI systems and positions Europe to play a leading role in setting the standard for the regulation of AI globally.

The Regulation follows the well-known risk-based approach the European legislator already implemented for a range of product categories, such as those implemented in the Medical Devices Regulation, including differentiating between high-risk AI, limited-risk AI, and minimal-risk AI. Depending on the risk level involved, providers of AI systems must employ different safety measures to ensure the trustworthiness of their AI system.

Race against the machine

According to the proposal, AI systems that create a high risk to the health and safety or fundamental rights of persons will be permitted on the European market subject to an ex-ante conformity assessment. Such high-risk AI systems are:

• those intended to be used as safety components of products that are subject to a third party ex-ante conformity assessment, such as medical devices, machinery, motor vehicles, radio equipment and personal protecting equipment (AI components);
• other stand-alone AI systems with mainly fundamental rights implications, such as remote biometric identification, educational and vocational training, employment and law enforcement.

High-risk AI systems will be subject to strict obligations before they can be put on the market:

• Adequate risk assessment and mitigation systems;
• High quality datasets feeding the system to minimise risks and discriminatory outcomes;
• Logging of activity to ensure traceability of results;
• Detailed documentation explaining the system’s purpose and providing all other information necessary for authorities to assess its compliance;
• Clear and adequate information provided to the user;
• Appropriate human oversight measures to minimise risk;
• High level of robustness, security and accuracy.

Obligation stations

Providers of such high-risk AI systems will have to evaluate whether the AI system complies with these obligations prior to placing it on the market. In the case of AI components, conformity assessment of the AI system will be part of the conformity assessment of the overall product. The conformity assessment must be renewed, wherever AI-systems are substantially modified. For AI-systems that continue to learn after being placed on the market, changes to the high-risk AI and its performance that have been predetermined by the provider at the moment of the initial conformity assessment and are part of the information contained in the technical documentation do not require recertification.

Also, providers of high-risk AI systems must have a post-market monitoring system in place, where providers actively collect, document and analyse relevant data throughout their lifetime. Providers and users will also have to report serious incidents and malfunctioning. Stand-alone AI systems must be registered in an EU database.

Certain high-risk AI systems considered to be a clear threat to the safety, livelihoods and rights of people will be banned entirely. The prohibitions cover practices that have a significant potential to manipulate persons through subliminal techniques beyond their consciousness or exploit vulnerabilities of specific vulnerable groups such as children or persons with disabilities in order to materially distort their behavior in a manner that is likely to cause them or another person psychological or physical harm. The proposal also prohibits AI-based social scoring for general purposes done by public authorities. Finally, the use of “real time” remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement is also prohibited unless certain limited exceptions apply.

Manipulation stipulations

Under the proposed Artificial Intelligence Act, AI systems with risk of manipulation will have to comply with specific transparency obligations. Such transparency obligations will apply for systems that:

• interact with humans,
• are used to detect emotions or determine association with (social) categories based on biometric data, or
• generate or manipulate content (‘deep fakes’).

When using such AI systems, generally people must be informed in advance. This allows them to make informed choices or step back from a given situation.

The proposed Artificial Intelligence Act also allows the free use of all other applications such as AI-enabled video games or spam filters. Such minimal-risk AI systems can be developed and used subject to the existing legislation and without additional legal obligations.  Voluntarily, providers of such minimal-risk AI systems may choose to apply the requirements for trustworthy AI and adhere to voluntary codes of conduct.

In the further legislative procedure, the European Parliament and the Member States will need to adopt the Commission’s proposal.

The implementation of the Artificial Intelligence Act will have a major impact on the market for AI systems. Companies will no longer be free in the assessment of safety of their products using AI software, but will have to observe the strict requirements of the new regulatory framework. Now is the time to get familiar with the proposed legislation, raise potential concerns to the attention of the lawmaking bodies and prepare for compliance with the new rules and obligations.

AI has been a mainstay of fiction for decades, especially dystopian fiction. In issuing the proposal, the EU has taken a positive step forward in turning that fiction into a measured, controlled and regulated reality.

Elisabeth Kohoutek is a senior associate advising companies in the life sciences sectors at King & Spalding. Ulf Grundmann is a partner at the same firm and head of its German life sciences practice; both are based in Frankfurt.