Decoding the EU AI Act Part 6 – Obligations for GPAI Providers with Systemic Risk

The EU’s Artificial Intelligence Act (Regulation (EU) 2024/1689) imposes comprehensive transparency and safety duties on providers of general-purpose AI (GPAI) models.

A special focus hereby is on those AI models with systemic risk, which have the potential for large-scale harm. In practice, this means that either leading edge models or any model that the EU Commission declares to have systemic risks capabilities, come within the scope of additional rules under Articles 51–55 of the Act.

This article explores how the AI Act defines systemic risk, when a GPAI model meets that threshold, and what model providers must do in addition to the normal obligations for all GPAI models once its model is considered to be within the systemic risk ambit.

What is a General-Purpose AI Model?

A general-purpose AI model is broadly defined under Article 3(63) as a highly capable AI model ‘trained with a large amount of data using self-supervision at scale that displays significant generality and is capable of competently performing a wide range of distinct tasks’.

In common parlance, this covers today’s large language models, multi-modal and foundation models that can be applied to many different applications. The form of release (open-source versus API access, etc.) does not change this definition. By contrast, a narrowly trained model for a single task would likely not come within this definition.

Providers of GPAI models face a set of  provider specific requirements. All GPAI providers, even those whose models do not pose systemic risks, must comply with Article 53 of the AI Act.

 In brief, Article 53 requires:

1. Technical model documentation, including the training and testing (Annex XI).
2. Documentation about model capabilities (Annex XII) for downstream deployers
3. A copyright-compliance policy to ensure compliance with Art.4(3) CDSM
4. A publicly available detailed summary of content used for training.

Open-source models are largely exempt from the documentation and disclosure duties in Article 53, unless they are identified as systemic risk models.

What is “Systemic Risk” Under the AI Act?

The term ‘systemic risk’ is defined in Article 3(65) of the AI Act as a risk tied to a model’s high-impact capabilities, which have ‘a significant impact on the Union market either due to:

1. reach, or
2. due to actual or reasonably foreseeable negative effects on critical areas such as:
   
   
   1. public health,
   2. safety,
   3. public security,
   4. fundamental rights, or
   5. society as a whole
      
      that can propagate at scale’.
      

In plain language, systemic risk means ‘large-scale’ harms that could emerge from GPAI models. These harms are not limited to any one sector. For example, Recital 110 of the Act mentions risks like:

1. aiding the development of chemical or biological weapons,
2. catastrophic accidents, and
3. loss of control over autonomous systems.
4. serious impacts on democratic institutions
5. public and economic security
6. social cohesion via disinformation or manipulation

In short, systemic risks are the worst-case hazards, ranging from physical disasters to security threats to major violations of rights, that could arise if an extremely capable AI model were to be misused or went astray at scale.

When Does a GPAI Model Present Systemic Risk?

A general-purpose model is formally classified as systemic-risk under Article 51 if it meets either of the two conditions :

1. High-impact capabilities: The model has capabilities that, by objective evaluation (benchmarks, indicators, etc.), match or exceed the most advanced GPAI models on the market.
   
   Even if a model falls below the above threshold, the Commission can designate it as systemic-risk, ex officio or after an alert by its scientific panel, if, based on criteria in Annex XIII, the model is deemed equivalent in capability or risk to the high-impact category.
   
   The Annex XIII criteria help flesh out what “equivalent impact” could mean. They include technical measures like the model’s size and compute and practical measures like market reach.
   
   For example: parameter count and dataset size, the total compute (or cost/energy) used in training, supported modalities (text, image, audio, multimodal, etc.), benchmark performance, adaptability and autonomy, and even user statistics (e.g. being used by 10,000+ EU businesses). In other words, if a model is very big, trained on vast data, highly capable across tasks, and/or widely deployed, it is very likely to qualify as systemic risk.
   
   
   A word of caution: There is maybe the general assumption that only the most capable AI models like GPT 4/5 come within the ambit of being systemic risk models. However, these days there are already many AI models available in a variety of distribution options, such as ‘open-weight‘ or ‘open-source‘, that can be freely accessed from sharing services that might already come within the ambit of being considered systemic risk, simply by way of business users or being already powerful enough to cause certain harm.
   
   
2. The Ai Act presumes any model whose training compute exceeds 10^25 floating-point operations (FLOPs) has such high-impact capabilities. Note that the EU Commission can adjust this threshold over time via delegated acts to keep pace with technology. Providers must notify the Commission when their model crosses the 10^25 FLOP threshold.

Under Article 52, once a GPAI provider knows its model exceeds or will exceed that threshold, it has two weeks to notify the EU Commission. The Commission will then consider whether to designate the model as a systemic risk model. The provider can argue against this finding if they believe the model’s specific nature avoids systemic risk, but failing to persuade the Commission means the model stays “systemic risk”. The Commission will also maintain a public list of all GPAI models designated as systemic risk.

Additional Obligations for Systemic-Risk GPAI Models under Article 55

If a GPAI model is classified as systemic risk, additional requirements apply under Article 55. These are in addition to the Article 53 duties, and they address the more severe risks that such models pose.

In brief, a systemic-risk model provider must perform:

1. Model Evaluations: including adversarial testing. Providers must actively probe and stress test the model to discover vulnerabilities or unsafe behaviors. They must document these tests and use them to guide improvements.
   
   
2. Risk Assessment and Mitigation: continuously assess and mitigate systemic risks possible at Union level. This means identifying all ways the model’s development, deployment, or use could generate large-scale harms and taking active steps to reduce those risks. Providers should consider risks arising from the entire lifecycle of the model.
   
   
3. Incident Reporting: track, document, and report serious incidents and near-missesinvolving the model. A ‘serious incident’ might be an AI-driven accident, a discovered vulnerability, or any harmful outcome. Providers must notify the EU AI Office and, where relevant, national authorities promptly with information about the incident and any corrective actions taken.
   
   
4. Cybersecurity: ensure adequate cybersecurity protectionfor the model and its infrastructure. Given the risks of theft or malicious use (for instance, someone stealing a model to train weapons AI), providers must use strong security measures throughout the model’s operation and storage.

These four points are explicitly spelled out in Article 55(1)(a–d). The Article encourages use of standardized protocols (once available) and explicitly requires documenting adversarial tests. Compliance can be demonstrated by adhering to the recently published EU Codes of Practice or future EU harmonized standards, but providers who chose not do so must show they have other adequate means of compliance.

In short, systemic risk GPAI providers must build and maintain a safety and security frameworkaround their model. This aligns with best practices in AI safety such as thorough red teaming and evaluations, formal risk management processes, incident surveillance, and robust defenses. For example, the newly released non-binding Code of Practice by the EU Commission lays out how such a ‘Safety and Security Framework’  could be constructed.

Systemic Risk Domain Examples

The AI Act does not list every possible harm, as the future potential for AI harm is not entirely foreseeable. Instead it provides categories of risks and the potential effects of systemic risks span across a wide array of legal domains.

The following table illustrates a non-exhaustive list of potential systemic risks that are regulated or outright illegal under certain legal regimes in the EU in addition to the EU AI Act.

Model Providers vs AI System Providers

It’s important to distinguish who is responsible for GPAI models under the Act. Providers of GPAI models hold the Article 53/55 obligations we have described above. Downstream providers of AI systems, by contrasts,  are any companies or developers that take a GPAI model and integrate it into a larger AI system or application. Under the AI Act, these roles have different duties. Model providers must, amongst other obligations discussed above, furnish the transparency and safety documentation. AI system providers, in turn, must follow the rules for the final AI system they build. For example, if their system becomes classified as a high-risk AI system, they have to comply with the criteria in Chapter 3 Section 2 of the AI Act.

Crucially, the Act ensures knowledge flows from providers to subsequent deployers. Article 53 obliges model providers to give downstream integrators enough information about the capabilities, limitations, intended uses, etc. so that these users can comply with the law.

Practical Takeaways and Timeline

For AI model developers the key takeaway is to assess compute and capabilities now. If your model might exceed the 10^25 FLOP mark or otherwise come within the definition of a systemic risk AI model, start preparing to meet the obligations.

Remember, even if the models is below the current compute line, the EU Commission can designate any model as systemic risk if warranted by scale or impact.

Compliance steps include: setting up an internal risk-management framework for your most powerful models, documenting every design/training choice, planning adversarial testing campaigns, and establishing incident-reporting processes.

For all models, draft thorough technical docs, downstream integration information, copyright policies and detailed training data summaries.

Disclaimer:

The content of this blog is provided for general informational purposes only and does not constitute legal advice. While we strive to ensure that the information is accurate and up to date, it may not reflect the most current legal developments or the specific circumstances of your organization. Readers should not act upon any information contained in this blog without first seeking professional legal counsel. No attorney-client relationship is created through your use of or reliance on the information provided herei