C(2025) 5052 — Prohibited practices

Commission guidelines on the prohibited artificial intelligence practices established by Article 5 of the AI Act.

Authority	Commission
Reference	C(2025) 5052 final
Legal basis	Article 96(1)(b) and Article 5 AI Act
Status	In force; non-binding Commission guidelines
Date	29 July 2025
Source	Commission publication
Document	Official English PDF

Brussels, 29.7.2025 C(2025) 5052 final

COMMUNICATION FROM THE COMMISSION

Commission Guidelines on prohibited artificial intelligence practices established by Regulation (EU) 2024/1689 (AI Act)

EN

CONTENTS

1.	Background and objectives ............................................................................................................. 1	Background and objectives ............................................................................................................. 1	Background and objectives ............................................................................................................. 1
2. Overview of prohibited AI practices ...............................................................................................	2. Overview of prohibited AI practices ...............................................................................................	2. Overview of prohibited AI practices ...............................................................................................	2
2.1.	2.1.	Prohibitions listed in Article 5 AI Act .................................................................................... 2
2.2.	2.2.	Legal basis of the prohibitions ................................................................................................	3
2.3. Material scope: practices related to the 'placing on the market', 'putting into service' or 'use' of an AI system .......................................................................................................................... 4	2.3. Material scope: practices related to the 'placing on the market', 'putting into service' or 'use' of an AI system .......................................................................................................................... 4	2.3. Material scope: practices related to the 'placing on the market', 'putting into service' or 'use' of an AI system .......................................................................................................................... 4	2.3. Material scope: practices related to the 'placing on the market', 'putting into service' or 'use' of an AI system .......................................................................................................................... 4
2.4.	2.4.	Personal scope: responsible actors ..........................................................................................	5
2.5.	2.5.	Exclusion from the scope of the AI Act ..................................................................................	7
2.5.1. National security, defence and military purposes ...........................................................	2.5.1. National security, defence and military purposes ...........................................................	2.5.1. National security, defence and military purposes ...........................................................	7
......................................	......................................	......................................
2.5.2. Judicial and law enforcement cooperation with third countries	2.5.2. Judicial and law enforcement cooperation with third countries	2.5.2. Judicial and law enforcement cooperation with third countries	9
2.5.3. Research	2.5.3. Research	& Development	............................................................................................... 9
2.5.4. Personal non-professional	2.5.4. Personal non-professional	activity	................................................................................ 10
2.5.5. AI systems	2.5.5. AI systems	released under free and open	source licences ............................................. 11
2.6.	Interplay of the prohibitions with the requirements for high-risk AI systems ...................... 12	Interplay of the prohibitions with the requirements for high-risk AI systems ...................... 12	Interplay of the prohibitions with the requirements for high-risk AI systems ...................... 12
2.7.	Application of the prohibitions to general-purpose AI systems and systems with intended purposes ............................................................................................................................................ 12	Application of the prohibitions to general-purpose AI systems and systems with intended purposes ............................................................................................................................................ 12	Application of the prohibitions to general-purpose AI systems and systems with intended purposes ............................................................................................................................................ 12
2.8.	Interplay between the prohibitions and other Union law ...................................................... 14	Interplay between the prohibitions and other Union law ...................................................... 14	Interplay between the prohibitions and other Union law ...................................................... 14
2.9.	Enforcement of Article 5 AI Act ........................................................................................... 17	Enforcement of Article 5 AI Act ........................................................................................... 17	Enforcement of Article 5 AI Act ........................................................................................... 17
2.9.1. Market Surveillance Authorities ...................................................................................	2.9.1. Market Surveillance Authorities ...................................................................................	2.9.1. Market Surveillance Authorities ...................................................................................	17
2.9.2. Penalties ........................................................................................................................	2.9.2. Penalties ........................................................................................................................	2.9.2. Penalties ........................................................................................................................	17
3. Article 5(1)(a) and (b) AI Act - harmful manipulation, deception and exploitation .................... 18	3. Article 5(1)(a) and (b) AI Act - harmful manipulation, deception and exploitation .................... 18	3. Article 5(1)(a) and (b) AI Act - harmful manipulation, deception and exploitation .................... 18	3. Article 5(1)(a) and (b) AI Act - harmful manipulation, deception and exploitation .................... 18
3.1. Rationale and objectives .......................................................................................................	3.1. Rationale and objectives .......................................................................................................	3.1. Rationale and objectives .......................................................................................................	18
3.2. Main components of the prohibition in Article 5(1)(a) AI Act - harmful manipulation ...... 19	3.2. Main components of the prohibition in Article 5(1)(a) AI Act - harmful manipulation ...... 19	3.2. Main components of the prohibition in Article 5(1)(a) AI Act - harmful manipulation ...... 19	3.2. Main components of the prohibition in Article 5(1)(a) AI Act - harmful manipulation ...... 19
3.2.1. Subliminal, purposefully manipulative or deceptive techniques ..................................	3.2.1. Subliminal, purposefully manipulative or deceptive techniques ..................................	3.2.1. Subliminal, purposefully manipulative or deceptive techniques ..................................	19
3.2.2. With the objective or the effect of materially distorting the behaviour of a person or a group	3.2.2. With the objective or the effect of materially distorting the behaviour of a person or a group	of	persons ............................................................................................................................ 24
3.2.3. (Reasonably likely	3.2.3. (Reasonably likely	3.2.3. (Reasonably likely	to) cause significant harm .............................................................. 28
3.3. Main components of the prohibition in Article 5(1)(b) AI Act - harmful exploitation of vulnerabilities .................................................................................................................................... 33	3.3. Main components of the prohibition in Article 5(1)(b) AI Act - harmful exploitation of vulnerabilities .................................................................................................................................... 33	3.3. Main components of the prohibition in Article 5(1)(b) AI Act - harmful exploitation of vulnerabilities .................................................................................................................................... 33	3.3. Main components of the prohibition in Article 5(1)(b) AI Act - harmful exploitation of vulnerabilities .................................................................................................................................... 33
3.3.1. Exploitation of vulnerabilities due to age, disability, or a specific socio-economic situation 33	3.3.1. Exploitation of vulnerabilities due to age, disability, or a specific socio-economic situation 33	3.3.1. Exploitation of vulnerabilities due to age, disability, or a specific socio-economic situation 33
3.3.2.	3.3.2.	3.3.2.	With the objective or the effect of materially distorting behaviour .............................. 38
3.3.3. (Reasonably likely to) cause significant harm .............................................................. 38 3.4. Interplay between the prohibitions in Article 5(1)(a) and (b) AI Act ................................... 42	3.3.3. (Reasonably likely to) cause significant harm .............................................................. 38 3.4. Interplay between the prohibitions in Article 5(1)(a) and (b) AI Act ................................... 42	3.3.3. (Reasonably likely to) cause significant harm .............................................................. 38 3.4. Interplay between the prohibitions in Article 5(1)(a) and (b) AI Act ................................... 42

3.5.	Out of scope ..........................................................................................................................	Out of scope ..........................................................................................................................	43
		Lawful persuasion .........................................................................................................	43
3.5.2. significant	3.5.2. significant	Manipulative, deceptive and exploitative AI systems that are not likely to cause harm ............................................................................................................................	45
3.6. Interplay with other Union law .............................................................................................	3.6. Interplay with other Union law .............................................................................................	3.6. Interplay with other Union law .............................................................................................	46
4. Article 5(1)(c) AI Act - social scoring ..........................................................................................	4. Article 5(1)(c) AI Act - social scoring ..........................................................................................	4. Article 5(1)(c) AI Act - social scoring ..........................................................................................	50
4.1. Rationale and objectives .......................................................................................................	4.1. Rationale and objectives .......................................................................................................	4.1. Rationale and objectives .......................................................................................................	50
4.2. Main concepts and components of the 'social scoring' prohibition ......................................	4.2. Main concepts and components of the 'social scoring' prohibition ......................................	4.2. Main concepts and components of the 'social scoring' prohibition ......................................	51
	4.2.1. 'Social scoring': evaluation or classification based on social behaviour or personal or personality characteristics over a certain period of time ...............................................................	4.2.1. 'Social scoring': evaluation or classification based on social behaviour or personal or personality characteristics over a certain period of time ...............................................................	52
	4.2.2. The social score must lead to detrimental or unfavourable treatment in unrelated social	contexts and/or unjustified or disproportionate treatment to the gravity of the social behaviour	. 55
		Regardless of whether provided or used by public or private persons	.......................... 59
4.3. Out of scope ..........................................................................................................................	4.3. Out of scope ..........................................................................................................................	4.3. Out of scope ..........................................................................................................................	61
4.4. Interplay with other Union legal acts .................................................................................... 63 Article 5(1)(d) AI Act - individual risk assessment and prediction OF CRIMINAL OFFENCES	4.4. Interplay with other Union legal acts .................................................................................... 63 Article 5(1)(d) AI Act - individual risk assessment and prediction OF CRIMINAL OFFENCES	4.4. Interplay with other Union legal acts .................................................................................... 63 Article 5(1)(d) AI Act - individual risk assessment and prediction OF CRIMINAL OFFENCES	4.4. Interplay with other Union legal acts .................................................................................... 63 Article 5(1)(d) AI Act - individual risk assessment and prediction OF CRIMINAL OFFENCES
5. 64	5. 64	5. 64	5. 64	5. 64
5.1. Rationale and objectives .......................................................................................................	5.1. Rationale and objectives .......................................................................................................	5.1. Rationale and objectives .......................................................................................................	65
5.2. Main concepts and components of the prohibition ...............................................................	5.2. Main concepts and components of the prohibition ...............................................................	5.2. Main concepts and components of the prohibition ...............................................................	65
5.2.1. Assessing the risk or predicting the likelihood of a person committing a crime ..........	5.2.1. Assessing the risk or predicting the likelihood of a person committing a crime ..........	5.2.1. Assessing the risk or predicting the likelihood of a person committing a crime ..........	66
5.2.2. Solely based on profiling of a natural person or on assessing their personality traits and	5.2.2. Solely based on profiling of a natural person or on assessing their personality traits and	5.2.2. Solely based on profiling of a natural person or on assessing their personality traits and	5.2.2. Solely based on profiling of a natural person or on assessing their personality traits and
characteristics ................................................................................................................................ 67	characteristics ................................................................................................................................ 67	Exclusion of AI systems to support the human assessment based on objective and verifiable facts directly linked to a criminal activity	.................................................................... 69
		Extent to which private actors' activities may fall within scope	................................... 71
5.3. Out of scope ..........................................................................................................................	5.3. Out of scope ..........................................................................................................................	5.3. Out of scope ..........................................................................................................................	72
5.3.1. Location-based or geospatial predictive or place-based crime predictions ...................	5.3.1. Location-based or geospatial predictive or place-based crime predictions ...................	5.3.1. Location-based or geospatial predictive or place-based crime predictions ...................	72
		to a criminal activity ..........................................................................................................	73
		AI systems used for crime predictions and assessments in relation to legal entities	.... 75
5.3.4. AI systems used for individual predictions of administrative offences ........................	5.3.4. AI systems used for individual predictions of administrative offences ........................	5.3.4. AI systems used for individual predictions of administrative offences ........................	75
5.4. Interplay with other Union legal acts ....................................................................................	5.4. Interplay with other Union legal acts ....................................................................................	5.4. Interplay with other Union legal acts ....................................................................................	76
6.	6.	5(1)(e) AI Act - untargeted scraping of facial images	...................................................... 77
			77
	6.1. Rationale and objectives .......................................................................................................	6.1. Rationale and objectives .......................................................................................................	6.1. Rationale and objectives .......................................................................................................
6.2. Main concepts and components of the prohibition ............................................................... 6.2.1. Facial recognition databases .........................................................................................	6.2. Main concepts and components of the prohibition ............................................................... 6.2.1. Facial recognition databases .........................................................................................	6.2. Main concepts and components of the prohibition ............................................................... 6.2.1. Facial recognition databases .........................................................................................	77

6.2.2.	6.2.2.	Through untargeted scraping of facial images .............................................................. 78
6.2.3.	6.2.3.	From the Internet and CCTV footage ...........................................................................	79
6.3. Out	6.3. Out	of scope ..........................................................................................................................	79
6.4.	6.4.	Interplay with other Union legal acts ....................................................................................	80
7. Article 5(1)(f) AI Act emotion recognition ...................................................................................	7. Article 5(1)(f) AI Act emotion recognition ...................................................................................	7. Article 5(1)(f) AI Act emotion recognition ...................................................................................	80
7.1.	7.1.	Rationale and objectives .......................................................................................................	80
7.2.	7.2.	Main concepts and components of the prohibition ...............................................................	81
7.2.1.	7.2.1.	AI systems to infer emotions ........................................................................................	82
7.2.2.	7.2.2.	Limitation of the prohibition to workplace and educational institutions ......................	84
7.2.3.	7.2.3.	Exceptions for medical and safety reasons ...................................................................	87
7.3.	7.3.	More favourable Member State law ......................................................................................	88
7.4. Out	7.4. Out	of scope ..........................................................................................................................	89
8. Article 5(1)(g) AI Act: Biometric categorisation for certain 'sensitive' characteristics ...............	8. Article 5(1)(g) AI Act: Biometric categorisation for certain 'sensitive' characteristics ...............	8. Article 5(1)(g) AI Act: Biometric categorisation for certain 'sensitive' characteristics ...............	90
8.1.	8.1.	Rationale and objectives .......................................................................................................	90
8.2.	8.2.	Main concepts and components of the prohibition ...............................................................	90
8.2.1.	8.2.1.	Biometric categorisation system ...................................................................................	91
8.2.2.	8.2.2.	Persons are individually categorised based on their biometric data ..............................	93
8.2.3.	8.2.3.	To deduce or infer their race, political opinions, trade union membership, religious or philosophical beliefs, sex-life or sexual orientation ......................................................................	93
8.3. Out		of scope	.......................................................................................................................... 94
8.4.	8.4.	Interplay with other Union law .............................................................................................	95
9. Article 5(1)(h) AI Act - Real-time Remote Biometric Identification (RBI) Systems for Law Enforcement Purposes .......................................................................................................................... 95	9. Article 5(1)(h) AI Act - Real-time Remote Biometric Identification (RBI) Systems for Law Enforcement Purposes .......................................................................................................................... 95	9. Article 5(1)(h) AI Act - Real-time Remote Biometric Identification (RBI) Systems for Law Enforcement Purposes .......................................................................................................................... 95	9. Article 5(1)(h) AI Act - Real-time Remote Biometric Identification (RBI) Systems for Law Enforcement Purposes .......................................................................................................................... 95	9. Article 5(1)(h) AI Act - Real-time Remote Biometric Identification (RBI) Systems for Law Enforcement Purposes .......................................................................................................................... 95
9.1.	9.1.	Rationale and objectives .......................................................................................................	96
9.2.	9.2.	Main concepts and components of the prohibition ...............................................................	97
9.2.1.	9.2.1.	The Notion of Remote Biometric Identification ...........................................................	97
9.2.2.	9.2.2.	Real-time .....................................................................................................................	100
9.2.3.	9.2.3.	In publicly accessible spaces . .....................................................................................	101
9.2.4.	9.2.4.	For law enforcement purposes ....................................................................................	103
9.3. Exceptions to the prohibition ..............................................................................................	9.3. Exceptions to the prohibition ..............................................................................................	9.3. Exceptions to the prohibition ..............................................................................................	104
9.3.1.	Rationale and objectives .............................................................................................	Rationale and objectives .............................................................................................	105
9.3.2.		Targeted search for the victims of three serious crimes and missing persons ............	105
9.3.3.	9.3.3.	Prevention of imminent threats to life or terrorist attacks ...........................................	107
9.3.4.	9.3.4.	Localisation and identification of suspects of certain crimes	..................................... 109

10.	Safeguards and Conditions for the exceptions (Article 5(2)-(7) AI Act)	................................ 112
10.1.	Targeted individual and safeguards (Article 5(2) AI Act) ..............................................	112
10.1.1.	10.1.1.	Fundamental Rights Impact Assessment ................................................................ 114
10.1.2.	10.1.2.	Registration of the authorized RBI systems ............................................................ 118
10.2.	Need for prior authorisation ............................................................................................ 119	Need for prior authorisation ............................................................................................ 119
10.2.1.	10.2.1.	Objective ................................................................................................................. 120
10.2.2. administrative	10.2.2. administrative	The main principle: Prior authorisation by a judicial authority or an independent authority .............................................................................................................. 120
10.3. systems	Notification to the authorities of each use of 'real-time' remote biometric identification in publicly accessible spaces for law enforcement ............................................................ 126	Notification to the authorities of each use of 'real-time' remote biometric identification in publicly accessible spaces for law enforcement ............................................................ 126
10.4.	Need for national laws within the limits of the AI Act exceptions ................................. 127	Need for national laws within the limits of the AI Act exceptions ................................. 127
10.4.1. Principle: national law required to provide the legal basis for the authorisation for all or some of the exceptions............................................................................................................ 127	10.4.1. Principle: national law required to provide the legal basis for the authorisation for all or some of the exceptions............................................................................................................ 127	10.4.1. Principle: national law required to provide the legal basis for the authorisation for all or some of the exceptions............................................................................................................ 127
10.4.2.	10.4.2.	National law shall respect the limits and conditions of Article 5(1)(h) AI Act ...... 127
10.4.3.	10.4.3.	Detailed national law on the authorisation request, the issuance and the exercise . 128
10.4.4. authorisation	10.4.4. authorisation	Detailed national law on the supervision and the reporting relating to the
10.5. Annual reports by the national market surveillance authorities and the national data protection authorities of Member States ......................................................................................... 130	10.5. Annual reports by the national market surveillance authorities and the national data protection authorities of Member States ......................................................................................... 130	10.5. Annual reports by the national market surveillance authorities and the national data protection authorities of Member States ......................................................................................... 130
10.6.	Annual reports by the Commission ................................................................................. 131	Annual reports by the Commission ................................................................................. 131
10.7.	Out-of-Scope ................................................................................................................... 131	Out-of-Scope ................................................................................................................... 131
10.8.	Examples of uses ............................................................................................................. 132	Examples of uses ............................................................................................................. 132
11.	Entry into application .............................................................................................................. 135	Entry into application .............................................................................................................. 135

1. BACKGROUND AND OBJECTIVES

(1) Regulation (EU) 2024/1689 of the European Parliament and the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending certain regulations ('the AI Act') 1 entered into force on 1 August 2024. The AI Act lays down harmonised rules for the placing on the market, putting into service, and use of artificial intelligence ('AI') in the Union. 2 Its aim is to promote innovation in and the uptake of AI, while ensuring a high level of protection of health, safety and fundamental rights in the Union, including democracy and the rule of law. (2) The AI Act follows a risk-based approach, classifying AI systems into four different risk categories: 3. (i) Unacceptable risk: AI systems posing unacceptable risks to fundamental rights and Union values are prohibited under Article 5 AI Act. 4. (ii) High risk: AI systems posing high risks to health, safety and fundamental rights are subject to a set of requirements and obligations. These systems are classified as 'high-risk' in accordance with Article 6 AI Act in conjunction with Annexes I and III AI Act. 5. (iii) Transparency risk: AI systems posing limited transparency risk are subject to transparency obligations under Article 50 AI Act. 6. (iv) Minimal to no risk: AI systems posing minimal to no risk are not regulated, but providers and deployers may voluntarily adhere to voluntary codes of conduct. 3 (3) Pursuant to Article 96(1)(b) AI Act, the Commission is to adopt guidelines on the practical implementation of the practices prohibited under Article 5 AI Act. Those prohibitions apply six months after the entry into force of the AI Act, i.e. as from 2 February 2025. (4) These Guidelines aim to increase legal clarity and to provide insights into the Commission's interpretation of the prohibitions in Article 5 AI Act with a view to ensuring their consistent, effective and uniform application. They should serve as practical guidance to assist competent authorities under the AI Act in their enforcement activities, as well as providers and deployers of AI systems in ensuring compliance with their obligations under the AI Act. They strive to interpret the prohibitions in a proportionate manner that achieves the objectives of the AI Act to protect fundamental rights and safety, while promoting innovation and providing legal certainty. (5) These Guidelines are non-binding. Any authoritative interpretation of the AI Act may ultimately only be given by the Court of Justice of the European Union ('CJEU'). (6) The drafting of these Guidelines was informed by input from a variety of stakeholders, e.g. providers and deployers of AI systems, civil society organisations, academia, public

1 Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) (OJ L, 2024/1689, 12.7.2024).

2 Article 1 AI Act.

3 Article 95 AI Act.

authorities, business associations, etc., collected during a broad consultation process organised by the Commission. The Member States within the AI Board and the European Parliament were also consulted. These Guidelines will be regularly reviewed in light of the experience gained from the practical implementation of Article 5 AI Act and technological and market developments.

(7) The application of Article 5 AI Act will require a case-by-case assessment, which takes due account of the specific situation at issue in an individual case. Therefore, the examples given in these Guidelines are merely indicative and are without prejudice to the need for such an assessment in each case.

2. OVERVIEW OF PROHIBITED AI PRACTICES

(8) Article 5 AI Act prohibits the placing on the EU market, putting into service, or use of certain AI systems for manipulative, exploitative, social control or surveillance practices, which by their inherent nature violate fundamental rights and Union values. Recital 28 AI Act clarifies that such practices are particularly harmful and abusive and should be prohibited because they contradict the Union values of respect for human dignity, freedom, equality, democracy, and the rule of law, as well as fundamental rights enshrined in the Charter of Fundamental Right of the European Union ( ' the Charter'), including the right to non-discrimination (Article 21 Charter) and equality (Article 20), data protection (Article 8 Charter) and private and family life (Article 7 Charter), and the rights of the child (Article 24 Charter). The prohibitions in Article 5 AI Act also aim to uphold the right to freedom of expression and information (Article 11 Charter), freedom of assembly and of association (Article 12 Charter), freedom of thought, conscience and religion (Article 10 Charter), the right to an effective remedy and fair trial (Article 47 Charter), and the presumption of innocence and the right of defence (Article 48 Charter).

2.1. Prohibitions listed in Article 5 AI Act

(9) Overview on the Prohibitions

Provision	Prohibition	Content
Article 5(1)(a)	Harmful manipulation, and deception	AI systems that deploy subliminal techniques beyond a person's consciousness or purposefully manipulative or deceptive techniques, with the objective or with the effect of distorting behaviour, causing or reasonably likely to cause significant harm
Article 5(1)(b)	Harmful exploitation of vulnerabilities	AI systems that exploit vulnerabilities due to age, disability or a specific social or economic situation, with the objective or with the effect of distorting behaviour, causing or reasonably likely to cause significant harm
Article 5(1)(c)	Social scoring	AI systems that evaluate or classify natural persons or groups of persons based on social behaviour or personal or personality characteristics, with the

		social score leading to detrimental or unfavourable treatment when data comes from unrelated social contexts or such treatment is unjustified or disproportionate to the social behaviour
Article 5(1)(d)	Individual criminal offence risk assessment and prediction	AI systems that assess or predict the risk of people committing a criminal offence based solely on profiling or personality traits and characteristics; except to support a human assessment based on objective and verifiable facts directly linked to a criminal activity
Article 5(1)(e)	Untargeted scraping to develop facial recognition databases	AI systems that create or expand facial recognition databases through untargeted scraping of facial images from the internet or closed-circuit television ('CCTV') footage
Article 5(1)(f)	Emotion recognition	AI systems that infer emotions at the workplace or in education institutions; except for medical or safety reasons
Article 5(1)(g)	Biometric categorisation	AI systems that categorise people based on their biometric data to deduce or infer their race, political opinions, trade union membership, religious or philosophical beliefs, sex-life or sexual orientation; except for labelling or filtering of lawfully acquired biometric datasets, including in the area of law enforcement
Article 5(1)(h)	Real-time remote biometric identification ('RBI')	AI systems for real-time remote biometric identification in publicly accessible spaces for the purposes of law enforcement; except if necessary for the targeted search of specific victims, the prevention of specific threats including terrorist attacks, or the search of suspects of specific offences (further procedural requirements, including for authorisation, outlined in Article 5(2- 7) AI Act).

2.2. Legal basis of the prohibitions

(10) The AI Act is supported by two legal bases: Article 114 of the Treaty on the Functioning of the European Union ('TFEU') (the internal market legal basis) and Article 16 TFEU (the data protection legal basis). Article 16 TFEU serves as a legal basis for the specific rules on the processing of personal data in relation to the prohibition on the use of remote biometric identification ('RBI') systems for law enforcement purposes, biometric categorisation systems for law enforcement purposes, and individual risk assessments for law enforcement purposes. 4 All other prohibitions listed in Article 5 AI Act find their legal basis in Article 114 TFEU.

4 Recital 3 AI Act. Regarding the prohibitions based on Article 16 of the TFEU, there are two relevant opt outs for Ireland and Denmark. With the discretion granted to Ireland under Protocol No. 21 on the position of the United Kingdom and Ireland in the area of freedom,

2.3. Material scope: practices related to the 'placing on the market', 'putting into service' or 'use' of an AI system

(11) The practices prohibited by Article 5 AI Act relate to the placing on the market, the putting into service, or the use of specific AI systems. 5 As regards real-time remote biometric identification ('RBI') systems, the prohibition in Article 5(1)(h) AI Act only applies to their use. Article 3(1) AI Act defines what constitutes an AI system. The Guidelines on the Definition of an AI system provide the Commission's interpretation of that definition . (12) According to Article 3(9) AI Act, the placing on the market of an AI system is 'the first making available of an AI system [...] on the Union market'. 'Making available' is defined as the supply of the system 'for distribution or use on the Union market in the course of a commercial activity, whether in return for payment or free of charge.' 6 The making available of an AI system is covered regardless of the means of supply, such as access to the system and its service through an application programming interface ('API'), via cloud, direct downloads, as physical copies, or embedded in physical products.

For example, a RBI system developed outside the Union by a third-country provider is placed on the Union market for the first time when it is offered in return for payment or free of charge in one or more Member States. Such placing on the market may occur by providing access to the system online through an API or other user interface.

(13) Article 3(11) AI Act defines putting into service as 'the supply of an AI system for first use to the deployer or for own use in the Union for its intended purpose', therefore covering both supply for first use to third parties, as well as in-house development and deployment. The intended purpose of the system is the 'use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions of use, promotional or sales materials and statements, as well as in the technical documentation.' 7

For example, a provider builds a RBI system outside the Union and supplies that system to a law enforcement authority or to a private company established in a Member State to be used for the first time, thereby putting it into service.

For example, a public authority develops a scoring system in-house and deploys it to predict the risk of fraud of household allowance beneficiaries, thereby putting it into service.

security and justice (AFSJ) annexed to the TEU and TFEU, Ireland may decide not to apply the rules concerning the prohibition of real-time use of RBIs in public spaces for a law enforcement purpose as well as the procedural rules linked to that article (Article 5(2) to (6) AI Act) (see Recital 40). Denmark benefits from opt-out agreements when applying Protocol No. 22 to the TEU and TFEU and may decide not to fully apply the prohibitions based on Article 16 of the TFEU (see Recital 41).

5 See for definitions of these terms also the Commission Notice - The 'Blue Guide' on the implementation of EU product rules 2022, 2022/C 247/01, Section 2.

6 Article 3(10) AI Act.

7 Article 3(12) AI Act.

(14) While the ' use ' of an AI system is not explicitly defined in the AI Act, it should be understood in a broad manner to cover the use or deployment of the system at any moment of its lifecycle after having been placed on the market or put into service. This may also cover the integration of the AI system in the services and processes of the person(s) making use of the AI system, including as part of more complex systems, processes or infrastructure. While providers of AI systems must consider the conditions of use which may be reasonably foreseen prior to placing their AI systems on the market (intended use and reasonably foreseeable misuse 8 ), deployers remain responsible for taking the lawful conditions for the use of the system into account. 9 For the purposes of Article 5 AI Act, the reference to 'use' should be understood to include any misuse of an AI system ('reasonably foreseeable' or not) that may amount to a prohibited practice. 10

For example, an AI system used by an employer to infer emotions at the workspace is prohibited, except when used for medical or safety purposes (Article 5(1)(f) AI Act). The prohibition applies to deployers regardless of whether the provider (the supplier of the system) has excluded such use in its contractual relationships with the deployer (the employer), i.e. in the terms of use.

2.4. Personal scope: responsible actors

(15) The AI Act distinguishes between different categories of operators in relation to AI systems: providers, deployers, importers, distributors, and product manufacturers. The present Guidelines will focus only on providers and deployers given the scope of the prohibited practices in Article 5 AI Act. (16) According to Article 3(3) AI Act, providers are natural or legal persons, public authorities, agencies or other bodies, that develop AI systems or have them developed and place them on the Union market, or put them into service under their own name or trademark 11 (see section 2.3 above). Providers established or located outside the Union are subject to the provisions of the AI Act if they place those systems on the market or put them into service in the Union, 12 or if the output of the AI system is used in the Union. 13 Providers must ensure their AI systems meet all relevant requirements before placing them on the market or putting them into service.

For example, a provider of a RBI system is the manufacturer of the system that markets the system in the Union under its trademark. The provider of such a system could also

8

9

10

11

12

See Article 3(12) and (13) AI Act.

See for definitions of these terms also the Commission Notice - The 'Blue Guide' on the implementation of EU product rules 2022,

2022/C 247/01, Section 2.8.

Recital 28 AI Act.

Article 3(3), (9) and (11) AI Act. In relation to high-risk AI systems, Article 25 AI Act envisages that 1. Any distributor, importer,

deployer or other third-party shall be considered to be a provider of a high-risk AI system for the purposes of this Regulation and

shall be subject to the obligations of the provider under Article 16, in any of the following circumstances: (a) they put their name or

trademark on a high-risk AI system already placed on the market or put into service, without prejudice to contractual arrangements

stipulating that the obligations are otherwise allocated; (b) they make a substantial modification to a high-risk AI system that has

already been placed on the market or has already been put into service in such a way that it remains a high-risk AI system pursuant

to Article 6; (c) they modify the intended purpose of an AI system, including a general-purpose AI system, which has not been

classified as high-risk and has already been placed on the market or put into service in such a way that the AI system concerned

becomes a high-risk AI system in accordance with Article 6.

Article 2(1)(a) AI Act.

13 Article 2(1)(c) AI Act.

be a public authority that develops the system in-house and puts it into service for its own use.

(17) Deployers are natural or legal persons, public authorities, agencies or other bodies using AI systems under their authority, unless the use is for a personal non-professional activity. 14 'Authority' over an AI system should be understood as assuming responsibility over the decision to deploy the system and over the manner of its actual use. Deployers fall within the scope of the AI Act, if their place of establishment or location is within the Union 15 or, if they are located in a third country, the output of the AI system is used in the Union. 16 (18) Where the deployer of an AI system is a legal person under whose authority the system is used, i.e. a law enforcement authority or a private security company, the individual employees that act within the procedures and under the control of that person should not be considered to be the deployer. A legal person also remains a deployer if it involves third parties (e.g. contractors, external staff) in the operation of the system on its behalf and under its responsibility and control. (19) Operators may fulfil more than one role concurrently in relation to an AI system. For example, if an operator develops its own AI system that it uses afterwards, it will be considered both the provider and the deployer of that system, even if that system is also used by other deployers to whom the system has been provided in return for payment or free of charge. (20) Continuous compliance with the AI Act is required during all phases of the AI lifecycle. This necessitates ongoing monitoring of and updates to AI systems placed on the market or put into service in the Union to ensure that an AI system remains compliant with the AI Act throughout its lifecycle and that it does not result in a practice prohibited under Article 5 AI Act. Providers and deployers of AI systems have different responsibilities depending on their roles and control over the design, the development and the actual use of the system to avoid a prohibited practice. For each of the prohibitions, these roles and responsibilities should be interpreted in a proportionate manner, taking into account who in the value chain is best placed to adopt specific preventive and mitigating measures and ensure compliant development and use of AI systems in line with the objectives and the approach of the AI Act.

2.5. Exclusion from the scope of the AI Act

(21) Article 2 AI Act provides for a number of general exclusions from scope which are relevant for a complete understanding of the practical application of the prohibitions listed in Article 5 AI Act.

2.5.1. National security, defence and military purposes

(22) According to Article 2(3) AI Act, the AI Act does not apply to areas outside the scope of Union law, and should not, in any event, affect the competences of the Member States

14 Article 3(4) AI Act.

15 Article 2(1)(b) AI Act.

16 Article 2(1)(c) AI Act.

concerning national security, regardless of the type of entity entrusted by the Member States with carrying out tasks in relation to those competences. The AI Act expressly excludes from its scope AI systems that are 'placed on the market, put into service, or used with or without modification exclusively for military, defence or national security purposes, regardless of the type of entity carrying out those activities.' Whether that exclusion applies therefore depends on the purposes or the uses of the AI system, not the entities carrying out the activities with that system, which may also cover private operators entrusted by the Member States with carrying out tasks in relation to those competences.

(23) According to the CJEU, the term 'national security ' refers to 'the primary interest in protecting the essential functions of the State and the fundamental interests of society and encompasses the prevention and punishment of activities capable of seriously destabilising the fundamental constitutional, political, economic or social structures of a country and, in particular, of directly threatening society, the population or the State itself, such as terrorist activities.' 17 National security does not cover, for example activities relating to road safety, 18 or the organisation or administration of justice. 19 As stated by the CJEU, 'it is for the Member States to define their essential security interests and to adopt appropriate measures to ensure their internal and external security, [...] a national measure [...] taken for the purpose of protecting national security cannot render EU law inapplicable and exempt the Member States from their obligation to comply with that law.' 20 (24) For the exclusion in Article 2(3), second subparagraph, AI Act to apply, the AI system must be placed on the market, put into service or used exclusively for military, defence or national security purposes. Recital 24 AI Act further clarifies how the notion ' exclusively' should be interpreted and when an AI system used for such purposes may nevertheless fall within the scope of the AI Act.

For example, if an AI system placed on the market, put into service or used for military, defence or national security purposes is used (temporarily or permanently) for other purposes, such as for civilian or humanitarian purposes, law enforcement or public security purposes, that system will fall within the scope of the AI Act. In that case, the entity using the AI system for the other purposes should ensure compliance of the AI system with the AI Act, unless the system already complies with that act, which has to be verified before such use.

(25) Furthermore, recital 24 AI Act clarifies that AI systems placed on the market or put into service for an excluded purpose, namely military, defence or national security, and for one or more non-excluded purposes, such as civilian or law enforcement purposes (so

17 Judgment of the Court of Justice of 6 October 2020, La Quadrature du Net and Others , C-511/18, C-512/18 and C-520/18, EU:C:2020:791, paragraph 135; Judgment of the Court of Justice of 5 June 2023, Commission v Poland , C-204/21, EU:C:2023:442,

paragraph 318, referring to Case C-439/19, paragraph 67 and Case C-306/21, paragraph 40.

18 Judgment of the Court of Justice of 22 June 2021, Latvijas Republikas Saeima , C-439/19, EU:C:2021:504, paragraph 68.

19 Judgment of the Court of Justice of 5 June 2023, Commission v Poland , C-204/21, EU:C:2023:442, paragraph 319.

20 Judgement of the Court of Justice of 6 October 2020, Privacy International, C-623/17, EU:C:2020:790, paragraph 44.

called ' dual use ' systems), fall within the scope of the AI Act. Providers of those systems should ensure that they comply with the requirements in the AI Act.

For example, if a company offers a RBI system for various purposes, including law enforcement and national security, that company is the provider of that 'dual use' system and must ensure its compliance with the requirements in the AI Act.

(26) However, the fact that an AI system may fall within the scope of the AI Act should not affect the ability of entities carrying out national security, defence and military activities to use that system for national security, military and defence purposes, regardless of the type of entity carrying out those activities. 21

For example, if a national security agency or a private operator is tasked by a national intelligence agency to use real-time RBI systems for national security purposes (such as to gather intelligence), such use would be excluded from the scope of the AI Act.

(27) The clear delineation of the national security exclusion is particularly important where AI systems are placed on the market, put into service or used for law enforcement purposes that fall within the scope of the AI Act. This is relevant for the prohibitions regarding individual crime predictions and assessments and regarding the use of realtime RBI systems for law enforcement purposes laid down in Article 5(1)(d) and (h) AI Act respectively. Police and other law enforcement authorities are tasked with the prevention, detection, investigation and prosecution of criminal offences or the execution of criminal penalties, including safeguarding against and preventing threats to public security. 22 Whenever AI systems are used for such purposes, they will fall within the scope of the AI Act. (28) The activities of Europol and other Union security agencies, such as Frontex, fall within the scope of the AI Act.

2.5.2. Judicial and law enforcement cooperation with third countries

(29) According to Article 2(4) AI Act, the AI Act does not apply to public authorities in a third country or international organisations, where those authorities or organisations use AI systems in the framework of international cooperation or agreements for law enforcement and judicial cooperation with the Union or with one or more Member States, provided that such a third country or international organisation provides adequate safeguards with respect to the protection of fundamental rights and freedoms of individuals. Where relevant, this exclusion may cover the activities of private entities entrusted by the third country in question to carry out specific tasks in support of such law enforcement and judicial cooperation. 23 At the same time, for the exclusion to apply, these frameworks for cooperation or international agreements must include adequate safeguards with respect to the protection of the fundamental rights and freedoms of individuals, to be assessed by the market surveillance authorities competent

21 Recital 24 AI Act.

22 Article 3(46) AI Act.

23 See Recital 22 AI Act.

for the supervision of AI systems used in the area of law enforcement and justice. 24 Recital 22 AI Act clarifies that the recipient national authorities and Union institutions, bodies, offices and agencies making use of such AI outputs in the Union remain accountable to ensure their use complies with Union law. When those international agreements are revised or new ones are concluded in the future, the contracting parties should make utmost efforts to align those agreements with the requirements of the AI Act.

2.5.3. Research & Development

(30) According to Article 2(8) AI Act, the AI Act does not apply 'to any research, testing or development activity regarding AI systems or AI models prior to their being placed on the market or put into service'. This exclusion is in line with the market-based logic of the AI Act, which applies to AI systems once they are placed on the market or put into service.

For example, during the research and development (R&D) phase, AI developers have the freedom to experiment and test new functionalities which might involve techniques that could be seen as manipulative and covered by Article 5(1)(a) AI Act, if used in consumer-facing applications. The AI Act allows for such experimentation by recognising that early-stage R&D is essential for refining AI technologies and ensuring that they meet safety and ethical standards prior to their placing on the market.

(31) As clarified in recital 25 AI Act, the AI Act aims to support innovation and recognises the importance of scientific research in advancing AI technologies and contributing to scientific progress and innovation. Article 2(6) AI Act therefore provides an exclusion for 'AI systems or AI models, including their outputs, specifically developed and put into service for the sole purpose of scientific research and development'.

For example, research into cognitive and behavioural responses to AI-driven subliminal or deceptive stimuli can provide valuable insights into human-AI interactions, informing safer and more effective AI applications in the future. Such research is permitted, since it is excluded from the scope of the AI Act, notwithstanding the prohibition in Article 5(1)(a) AI Act.

(32) The exclusion in Article 2(8) AI Act is, however, without prejudice to the obligation to comply with the AI Act where an AI system is placed on the market or put into service as a result of such research and development activity. 25 Testing in real-world conditions within the meaning of the AI Act 26 is also not covered by that exclusion.

For example, a municipality wishing to test facial recognition software using a RBI system in the streets during carnival recruits volunteers to be identified by the system in real-world conditions. Because real-world testing does not fall within the exclusion of Article 2(8) AI Act, the planned testing must be fully compliant with the requirements for RBI systems in the AI Act, unless the system is tested in an AI regulatory sandbox or in accordance with the special regime for testing in real world conditions outside the sandbox, as provided for in Articles 60 and 61 AI Act. 27

24 See Recital 22 and Article 74(8) AI Act.

25 Recital 25 AI Act.

26 According to Article 3(57) AI Act, 'testing in real-world conditions' means the temporary testing of an AI system for its intended purpose in real-world conditions outside a laboratory or otherwise simulated environment, with a view to gathering reliable and robust data and to assessing and verifying the conformity of the AI system with the requirements of this Regulation. The AI Act provides a special regime for such testing in real-world conditions which does not qualify as placing the AI system on the market or putting it into service within the meaning of this Regulation, provided that all the conditions laid down in Articles 57 or 60 are fulfilled, including obtaining free and informed consent from the persons participating in the testing etc.; see Article 60 AI Act.

(33) In any event, any research and development activity (including when excluded from the scope of the AI Act) should be carried out in accordance with recognised ethical and professional standards for scientific research and should be conducted in accordance with applicable Union law 28 (e.g., data protection law that remains applicable).

2.5.4. Personal non-professional activity

(34) Article 2(10) AI Act provides that the AI Act 'does not apply to obligations of deployers who are natural persons using systems in the course of a purely personal nonprofessional activity'. The definition of deployer also excludes users engaged in such activities (see section 2.4. above). Any activity through which a natural person gains an economic benefit on a regular basis or is otherwise involved in a professional, business, trade, occupational or freelance activity should be considered as a 'professional' activity. The specification of 'personal' is a qualifier of non-professional, meaning that the person should act in both a personal and a non-professional capacity. The exclusion should therefore, for example, not encompass criminal activities since these should not be considered purely personal.

For example, an individual using a facial recognition system at home (e.g. to control access and to monitor for safety the entrance to the home) would fall under the exclusion of Article 2(10) AI Act and, hence, would not be subject to the obligations for deployers under the AI Act, even in cases where it is required to transmit (parts of) the footage to law enforcement authorities.

By contrast, a natural person using an AI system for professional activities such as freelancers, journalists, doctors, etc. would need to comply with the obligations for deployers of facial recognition systems under the AI Act. Any use by natural persons where they are acting on behalf or under the authority of a deployer acting in a professional capacity will also fall within the scope of the AI Act.

Furthermore, criminal activities cannot be considered purely personal activities, even if no economic benefit is sought or attained. For other unlawful activities, such as noncompliance with consumer protection or data protection law and national administrative legislation, the exclusion in the AI Act applies, but the other relevant legal frameworks continue to apply).

(35) The exclusion in Article 2(10) AI Act applies only as regards the obligations of deployers when using the system for purely personal non-professional activities. The system as such remains within the scope of the AI Act as regards the obligations of

27 The AI Act contains detailed and specific obligations for AI Regulatory Sandboxes and real-world testing. See Article 57 AI Act et seq.

28 Recital 25 AI Act.

providers placing the system on the market or putting it into service, other professional deployers, and other responsible actors, such as importers and distributors.

For example, an emotion recognition system, if intended to be used by natural persons for purely personal non-professional activities, remains a high-risk AI system as classified in Article 6 AI Act and must be fully in compliance with the AI Act. At the same time the deployer that uses it for purely personal non-professional use (e.g., an autistic person) will not be covered by specific obligations for deployers under the AI Act and the use would be out of scope.

2.5.5. AI systems released under free and open source licences

(36) According to Article 2(12) AI Act, the AI Act does not apply to AI systems released under free and open-source licences, 29 unless they are placed on the market or put into service as high-risk AI systems or as an AI system that falls under Article 5 (prohibited AI practices) or Article 50 AI Act (transparency obligations for certain AI systems). This means that providers of AI systems cannot benefit from this exclusion if the AI system they place on the market or put into service constitutes a prohibited practice under Articles 5 AI Act.

2.6. Interplay of the prohibitions with the requirements for high-risk AI systems

(37) The AI practices prohibited by Article 5 AI Act should be considered in relation to the AI systems classified as high-risk in accordance with Article 6 AI Act, in particular those listed in Annex III. 30 That is because the use of AI systems classified as high-risk may in some cases qualify as a prohibited practice in specific instances if all conditions under one or more of the prohibitions in Article 5 AI Act are fulfilled. Conversely, most AI systems that fall under an exception from a prohibition listed in Article 5 AI Act will qualify as high-risk.

For example, emotion recognition systems, where they do not fulfil the conditions for the prohibition in Article 5(1)(f) AI Act, classify as high-risk AI systems according to Article 6(2) and Annex III, point (1)(c) AI Act. Similarly, certain AI-based scoring system, such as those used for credit-scoring or assessing risk in health and life insurance, will be considered high-risk AI systems where they do not fulfil the conditions for the prohibition listed in Article 5(1)(c) AI Act. 31 Another example are AI systems evaluating persons and determining if they are entitled to receive essential public assistance benefits and services, such as healthcare services and social security benefits that are classified as high-risk. 32 If such systems involve unacceptable social scoring and fulfil the conditions of Article 5(1)(c) AI Act, their placing on the market, putting into service and use will be prohibited in the Union.

29 Recital 102 AI Act describes that a release of software and data under free and open-source licence 'allows them to be openly shared and where users can freely access, use, modify and redistribute them or modified versions thereto'.

30 In this list, AI systems based on biometrics are covered, as well as AI systems used for specific purposes in certain domains such as employment, education, access to public and private services, law enforcement etc.

31 This is expressly mentioned in Recital 58 and in Annex III AI Act.

32 Recital 58 AI Act.

In such cases, the risk assessment and management done by the provider and the compliance with the other requirements for high-risk AI systems (e.g., data governance, transparency and human oversight), as well as the deployer's obligations for appropriate use in accordance with the instructions of use and human oversight (Article 26) and in some cases a fundamental rights impact assessment (Article 27), should help to ensure that the high-risk AI system placed on the market or deployed is lawful and does not constitute a prohibited practice.

(38) Finally, AI systems that are exceptionally not considered high-risk based on Article 6(3) AI Act, despite falling under a high-risk use case of Annex III, may still fall within the scope of the prohibitions of Article 5 AI Act. Article 6(3) AI Act only results in an AI system being considered non-high-risk; it does not exclude such AI systems from the scope of the AI Act and the prohibitions.

2.7. Application of the prohibitions to general-purpose AI systems and systems with intended purposes

(39) The prohibitions apply to any AI system, whether with an 'intended purpose' 33 or 'general-purpose' (i.e. that can serve a variety of purposes), for direct use or for integration in other AI systems. 34 Accordingly, each operator should take measures for which they are best placed based on their role and control over the system in the value chain to ensure a responsible and safe provision and use of AI systems, balancing their risks and benefits with a view to achieving the twin objectives of the AI Act. (40) Deployers are thus expected not to use any AI system in a manner prohibited under Article 5 AI Act, including not to bypass any safety guardrails implemented by the providers of the system. While the harm often arises from the way the AI systems are used in practice, providers also have a responsibility not to place on the market or put into service AI systems, including general-purpose AI systems, that are reasonably likely to behave or be directly used in a manner prohibited by Article 5 AI Act. 35 In this context, providers are also expected to take effective and verifiable measures to build in safeguards and prevent and mitigate such harmful behaviour and misuse to the extent they are reasonably foreseeable and the measures are feasible and proportionate depending on the specific AI system and circumstances of the case. In their contractual relationships with deployers (i.e. in the terms of use of the AI system), providers are also expected to exclude use of their AI system for prohibited practices and provide appropriate information in the instructions of use for deployers and regarding the necessary human oversight.

For example, a general-purpose AI system used as a chatbot may deploy manipulative and deceptive techniques which is likely to cause significant harm. To prevent prohibited behaviour of the AI system and uses that are reasonably likely to manipulate,

33

Defined in Article 3(12) AI Act as the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation.

34 See Article 3(66) AI Act.

35 This follows in particular from the reference to 'placing on the market' or 'putting into service' in all prohibitions listed in Article 5 AI Act, with the exception of the prohibition of real-time RBI systems in Article 5(1)(h) that applies only to the use.

deceive and cause significant harms under Article 5(1)(a) AI Act, the provider is expected to take appropriate and proportionate measures (e.g., appropriate safe and ethical design, integration of technical and other safeguards, restrictions of use, transparency and user control, appropriate information in the instructions of use) before the AI system is placed on the market (Article 5(1)(a) AI Act) to ensure the chatbot does not cause significant harm to users or other persons or groups of persons (see also section 3.2.3.(c)).

(41) In certain cases, in particular where the prohibitions are linked to a very specific purpose of the system, 36 providers may have limited possibilities to integrate other preventive and mitigating measures and will have to rely on primarily providing appropriate instructions and information to the deployers and the required human oversight and restricting prohibited use of the system. Where appropriate, such measures may also include monitoring for compliance with that restriction, depending on the means through which the AI system is supplied and the information at the provider's disposal for possible misuse. Any possible monitoring measures to detect misuse should not amount to a general monitoring of the activities of the deployers and should be in line with Union law.

For example, a general-purpose AI system that can recognise or infer emotions should not be used by deployers at the workplaces or in education institutions, unless an exception for medical or safety reasons applies. However, the provider may not be in a position to know the specific context in which the emotion recognition functionality of the system will be used and whether an exception to the prohibition in Article 5(1)(f) AI Act may apply. Such providers may nevertheless explicitly exclude such prohibited use in their terms of use and include appropriate information in the instructions of use to guide deployers. They are also expected to take appropriate measures if they become aware that the system is misused for this specific prohibited purpose by specific deployers, for example, if such misuse is reported or the provider becomes otherwise aware, which may be the case if the system is directly operated through a platform under the control of the provider and the provider performs checks.

2.8. Interplay between the prohibitions and other Union law

(42) The AI Act is a regulation that applies horizontally across all sectors without prejudice to other Union legislation, in particular on the protection of fundamental rights, consumer protection, employment, the protection of workers, and product safety. 37 The AI Act complements such legislation through its preventative and safety logic (AI systems may not be placed on the market or used in a certain way) and provides additional protection by addressing specific harmful AI practices which may not be prohibited by other laws. Furthermore, by addressing the earlier stages of the AI systems' lifecycle (i.e. the placing on the market and putting into service) and

36 Article 5(1)(d)-(h) AI Act.

37 Article 2 and Recital 9 AI Act.

deployment (i.e. the use), the AI Act's prohibitions enable action to be taken against harmful practices involving AI at various points in the AI value chain.

(43) At the same time, the AI Act does not affect prohibitions that apply where an AI practice falls within other Union law. 38 Thus, even where an AI system is not prohibited by the AI Act, its use could still be prohibited or unlawful based on other primary or secondary Union law (e.g., because of the failure to respect fundamental rights in a given case, such as the lack of a legal basis for the processing of personal data required under data protection law, discrimination prohibited by Union law, etc.). The respect of the prohibitions in the AI Act are therefore not a sufficient condition for compliance with other Union legislation that remains applicable to providers and deployers of AI systems.

For example, AI-enabled emotion recognition systems used in the workplace that are exempted from the prohibition in Article 5(1)(f) AI Act, because they are used for medical or safety reasons, remain subject to data protection law and Union and national law on employment and working conditions, including health and safety at work, which may foresee other restrictions and safeguards in relation to the use of such systems. 39

(44) When specific activities related to the placing on the market or use of AI systems are also covered under other Union legislation, the AI Act aims to ensure the consistent implementation of the different provisions. Moreover, it enables effective cooperation between the competent authorities responsible for the enforcement of the AI Act and the authorities protecting fundamental rights pursuant to Article 77 AI Act and other provisions of the AI Act. More generally, in accordance with Article 4(3) TEU, the various authorities concerned are bound to cooperate sincerely when giving effect to their respective tasks under Union law. (45) In the context of the prohibitions, the interplay between the AI Act and Union data protection law is particularly relevant, since AI systems often process information relating to identified or identifiable natural persons ('personal data'). 40 Depending on the prohibition and the context, the most relevant legal acts in relation to such systems are Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation, hereinafter 'GDPR'), Directive (EU) 2016/680 on the protection of natural persons with regard to the processing of personal data by competent authorities for the purposes of the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, and on the free movement of such data (Law Enforcement Directive, hereinafter 'LED'), and Regulation (EU) 2018/1725 which lays down data protection rules for the EU Institutions, bodies, offices and agencies (hereinafter 'EUDPR'). In accordance with Article 2(7) AI Act, these acts remain unaffected and will continue to apply alongside the AI Act, which is consistent and complementary to the EU data protection acquis. Several aspects of these EU data

38 Article 5(8) AI Act.

39 See also Recital 9 AI Act.

40 Article 2(7) AI Act; see also Recital 10 AI Act.

protection rules have been clarified by the CJEU and the European Data Protection Board has adopted a series of guidelines (e.g. on the notion of 'profiling', 41 which is particularly relevant for the prohibition in Article 5(1)(d) AI Act, since it uses the same notion).

(46) Concerning the prohibitions/restrictions on the use of biometric categorisation systems and real-time RBI systems for law enforcement purposes, the AI Act applies as lex specialis to Article 10 LED, thus regulating such use and the processing of biometric data involved in an exhaustive manner. 42 In that context, the AI Act is not intended to provide the legal basis for the processing of personal data under Article 8 of Directive (EU) 2016/680. All other provisions of that Directive apply in addition to the conditions set out in the AI Act, in particular for the use of real-time (RBI) systems for law enforcement purposes when permitted, subject to the limited exceptions in Article 5(1)(h) AI Act. More generally, the LED must also be complied with for any processing of personal data by competent law enforcement authorities (i.e. competent authorities under Article 3(7) LED) when they process the data for law enforcement purposes. (47) In accordance with Article 2(9) AI Act, EU consumer protection and safety legislation also remain fully applicable to AI systems falling within scope of those acts.

For example,

• Social scoring practices by traders (including natural persons acting in a professional capacity in business-to-consumer relations), subject to case-by-case assessment, may also be considered 'unfair' and therefore in breach of consumer law (i.e. Directive 2005/29/EC);
• The use of an AI system to infer emotions may also have to comply with Regulation (EU) 2017/745 (Medical Device Regulation) if the AI system is used for medical diagnosis or medical treatment purposes.

(48) Furthermore, the AI Act applies in conjunction with relevant obligations for providers of intermediary services that embed AI systems or models into their services regulated by Regulation (EU) 2022/2065 ('the Digital Services Act'). Specifically, Article 2(5) AI Act indicates that the AI Act does not affect the application of the provisions on the liability of such providers as set out in Chapter II of the Digital Services Act. (49) In addition, the prohibitions in the AI Act are without prejudice to any liability that the provider or deployer might incur for the harm caused according to applicable Union or national liability laws. 43

41 See also Article 29 Data Protection Working Party, Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679, WP251rev.01, 6.2.2018, and endorsed by the EDPB.

42 Recital 38 AI Act.

43 The conditions for liability (related to damage, liable person, fault or burden of proof, etc) will be determined by the applicable law, such as Directive (EU) 2024/2853 of the European Parliament and of the Council of 23 October 2024 on liability for defective products, (Text with EEA relevance), OJ L, 2024/2853, 18.11.2024 or the applicable national liability laws (see also Proposal for a Directive of the European Parliament and of the Council on adapting non-contractual civil liability rules to artificial intelligence (AI Liability Directive) COM/2022/496 final).

(50) Finally, the prohibitions in Article 5 AI Act and the explicit exceptions to those prohibitions may not be used to circumvent or as a justification to infringe obligations under other Union legislation. (51) As secondary Union legislation, the AI Act must be interpreted in the light of the fundamental rights and freedoms guaranteed by the EU Treaties and the Charter, as well as those protected by international conventions to which the Union is a party. 44 (52) Additional clarifications on the interplay of specific prohibitions with other Union law are provided under the relevant sections below.

2.9. Enforcement of Article 5 AI Act

2.9.1. Market Surveillance Authorities

(53) Market surveillance authorities designated by the Member States as well as the European Data Protection Supervisor (as the market surveillance authority for the EU institutions, agencies and bodies) are responsible for the enforcement of the rules in the AI Act for AI systems, including the prohibitions. Such enforcement takes place within the system of market surveillance and compliance of products established by Regulation (EU) 2019/1020, 45 in line with other Union product safety legislation. The enforcement powers of market surveillance authorities in relation to AI systems are laid down in the AI Act and in Regulation (EU) 2019/1020. Those authorities can take enforcement actions in relation to the prohibitions on their own initiative or following a complaint, which every affected person or any other natural or legal person having grounds to consider such violations has the right to lodge. 46 Member States must designate their competent market surveillance authorities by 2 August 2025. (54) The procedure in the AI Act to deal with AI systems presenting a risk at national level is particularly relevant in the context of enforcing the prohibitions. 47 Where there are cross-border implications beyond the territory of the market surveillance authority, the authority of the Member State concerned must inform the Commission and the market surveillance authorities of other Member States. All market surveillance authorities should follow a Union safeguard procedure with a decision taken by the Commission 48 determining whether the AI system constitutes a prohibited practice. That procedure aims to ensure that the prohibitions are applied uniformly across all Member States, so as to provide legal certainty to both providers and deployers of AI systems. To ensure the uniform application of the AI Act, national market surveillance authorities should also strive for a harmonized application of the prohibitions for comparable cases that do not cross the Member State's territory by drawing inspiration from these Guidelines and cooperating within the AI Board. 49

44 Even if the Union is not yet a party to the European Convention for the Protection of Human Rights and Fundamental Freedoms, Article 59(3) of the Charter states that in so far as the Charter contains rights which correspond to rights guaranteed by the European Convention for the Protection of Human Rights and Fundamental Freedoms, the meaning and scope of those rights shall be the same as those laid down by the said Convention. This provision shall not prevent Union law providing more extensive protection.

45 See also Recital 156 AI Act.

46 Article 85 AI Act.

47 Article 79 AI Act.

48 Article 81 AI Act.

49 Article 65 and 66 AI Act.

16

2.9.2. Penalties

(55) The AI Act follows a tiered approach in setting the penalties for non-compliance with its various provisions, depending on the seriousness of the infringement. Noncompliance with the prohibitions in Article 5 AI Act are considered to constitute the most severe infringement and they are therefore subject to the highest fine. Providers and deployers engaging in prohibited AI practices may be fined up to EUR 35 000 000 or, if the offender is an undertaking, up to 7% of its total worldwide annual turnover for the preceding financial year, whichever is higher. 50 Each Member State should lay down rules if and to the extent that administrative fines may be imposed on public authorities and bodies established in that Member State as providers and deployers of AI systems. EU institutions, bodies and agencies that violate the prohibitions may be subject to administrative fines of up to EUR 1 500 000. 51 (56) It is possible that one and the same prohibited conduct constitutes a violation of two or more provisions of the AI Act (i.e. the non-labelling of deep fakes may also constitute a deceptive technique under Article 5(1)(a) AI Act). In such cases, the principle of ne bis in idem should be respected. In any event, the criteria for determining the penalty as provided for in Article 99(7) AI Act must be taken into account. (57) Since violations of the prohibitions in Article 5 AI Act interfere the most with the freedoms of others and give rise to the highest fines, their scope should be interpreted narrowly.

3. ARTICLE 5(1)(A) AND (B) AI ACT - HARMFUL MANIPULATION, DECEPTION AND EXPLOITATION

(58) The first two prohibitions in Article 5(1)(a) and (b) AI Act aim to safeguard individuals and vulnerable persons from the significantly harmful effects of AI-enabled manipulation and exploitation. Those prohibitions target AI systems that deploy subliminal, purposefully manipulative or deceptive techniques that are significantly harmful and materially influence the behaviour of natural persons or group(s) of persons (Article 5(1)(a) AI Act) or exploit vulnerabilities due to age, disability, or a specific social or economic situation (Article 5(1)(b) AI Act).

3.1. Rationale and objectives

(59) The underlying rationale of these prohibitions is to protect individual autonomy and well-being from manipulative, deceptive, and exploitative AI practices that can subvert and impair an individual's autonomy, decision-making, and free choices. 52 The prohibitions aim to protect the right to human dignity (Article 1 of the Charter), which also constitutes the basis of all fundamental rights and includes individual autonomy as an essential aspect. In particular, the prohibitions aim to prevent manipulation and exploitation through AI systems that reduce individuals to mere tools for achieving certain ends and to safeguard those that are most vulnerable and susceptible to harmful

50 Article 99 AI Act.

51

Article 100 AI Act.

52 Recital 29 AI Act.

17

manipulation and exploitation. The prohibitions against significantly harmful manipulative, deceptive and exploitative AI practices fully align with the broader objectives of the AI Act to promote trustworthy and human-centric AI systems that are safe, transparent, fair and serve humanity and align with human agency and EU values.

3.2. Main components of the prohibition in Article 5(1)(a) AI Act - harmful manipulation

Article 5(1)(a) AI Act provides:

1. The following AI practices shall be prohibited:

(a) the placing on the market, putting into service or use of an AI system that deploys subliminal techniques beyond a person's consciousness or purposefully manipulative or deceptive techniques, with the objective, or the effect of materially distorting the behaviour of a person or a group of persons by appreciably impairing their ability to make an informed decision, thereby causing them to take a decision that they would not have otherwise taken in a manner that causes or is reasonably likely to cause that person, another person or group of persons significant harm;

(60) Several cumulative conditions must be fulfilled for the prohibition in Article 5(1)(a) AI Act to apply: 2. (i) The practice must constitute the 'placing on the market', the 'putting into service', or the 'use' of an AI system. 3. (ii) The AI system must deploy subliminal (beyond a person's consciousness), purposefully manipulative or deceptive techniques. 4. (iii)The techniques deployed by the AI system should have the objective or the effect of materially distorting the behaviour of a person or a group of persons. The distortion must appreciably impair their ability to make an informed decision, resulting in a decision that the person or the group of persons would not have otherwise made. 5. (iv) The distorted behaviour must cause or be reasonably likely to cause significant harm to that person, another person, or a group of persons. (61) For the prohibition to apply, all four conditions must be simultaneously fulfilled and there must be a plausible causal link between the techniques deployed, the material distortion of the behaviour of the person, and the significant harm that has resulted or is reasonably likely to result from that behaviour. (62) The first condition, i.e. the 'placing on the market', the 'putting into service' or the 'use' of an AI system, has already been analysed. The prohibition, therefore, applies to both providers and deployers of AI systems, each within their respective responsibilities, not to place on the market, put into service or use such systems. The next sections focus on the other three conditions.

3.2.1. Subliminal, purposefully manipulative or deceptive techniques

(63) Article 5(1)(a) AI Act prohibits three alternative types of manipulative techniques: a) subliminal techniques beyond a person's consciousness: b) purposefully manipulative techniques; and c) deceptive techniques. To fall within scope of Article 5(1)(a) AI Act, an AI system must deploy one or more of these techniques.

a) Subliminal techniques

(64) While the AI Act does not define 'subliminal techniques', Article 5(1)(a) AI Act specifies that subliminal techniques operate beyond (below or above) the threshold of conscious awareness. Because subliminal techniques and the way they operate are inherently covert, such techniques bypass a person's rational defences against manipulation and are capable of influencing decisions without the conscious awareness of the person, raising significant ethical concerns and impairing individual autonomy, agency and free choice. 53 (65) The subliminal techniques must be capable of influencing behaviour in ways in which the person remains unaware of such influence, how it works, or its effects on the person's decision-making or value- and opinion-formation. In particular, subliminal techniques may use stimuli delivered through audio, visual, or tactile media that are too brief or subtle to be noticed and that have been traditionally known and prohibited in other sectors, such as media advertising. 54 These stimuli, while not consciously perceived, may still be processed by the brain and influence behaviour.

Examples of subliminal techniques (not necessarily prohibited unless all other conditions listed in Article 5(1)(a) AI Act are fulfilled) include:

• Visual Subliminal Messages: an AI system may show or embed images or text flashed briefly during video playback which are technically visible, but flashed too quickly for the conscious mind to register, while still being capable of influencing attitudes or behaviours.
• Auditory Subliminal Messages: an AI system may deploy sounds or verbal messages at low volumes or masked by other sounds, influencing the listener without conscious awareness. These sounds are still technically within the range of hearing, but are not consciously noticed by the listener due to their subtlety or masking by other audio.

-Tactile Subliminal Stimuli: an AI system may stimulate subtle physical sensations that are perceived unconsciously, capable of influencing emotional states or behaviour.

• Subvisual and Subaudible Cueing: an AI system may deploy stimuli that are not just subtle or masked, but are presented in a way that makes them entirely undetectable by the human senses under normal conditions, for example flashing visual stimuli (e.g., flashing images) too quickly for the human eye to detect consciously or playing sounds at volumes imperceptible to the human ear.

53 Recital 29 AI Act.

54 See in particular, the Directive 2010/13/EU of the European Parliament and of the Council of 10 March 2010 on the coordination of certain provisions laid down by law, regulation or administrative action in Member States concerning the provision of audiovisual media services (OJ L 95, 15.4.2010, p. 1) ('AVMSD'). which strictly prohibits subliminal techniques in audiovisual commercial communications.

• Embedded Images: an AI system may hide images within other visual content which are not consciously perceived, but may still be processed by the brain and influence behaviour.
• Misdirection: an AI system may draw attention to specific stimuli or content to prevent noticing other content, often by exploiting cognitive biases and vulnerabilities in attention.
• Temporal manipulation: an AI system may alter the perception of time in user interactions, thus influencing their behaviour and causing impatience and dependence.

(66) The rapid development of AI and related technologies, such as big data analytics, neuro technologies, brain-computer interfaces and virtual reality, heightens the risk of sophisticated subliminal manipulation and its capability to effectively influence human behaviour in a subconscious manner. 55 AI can also extend to emerging machine-brain interfaces and advanced techniques like dream-hacking and brain spyware.

For example, a game can leverage AI-enabled neuro technologies and machine-brain interfaces that permit users to control (parts of) a game with headgear that detects brain activity. AI may be used to train the user's brain surreptitiously and without their awareness to reveal or infer from the neural data information that can be very intrusive and sensitive (e.g., personal bank information, intimate information, etc.) in a manner that can cause them significant harm. The prohibition in Article 5(1)(a) AI Act targets only cases of such significantly harmful subliminal manipulation and not machine-brain interface applications in general when designed in a safe and secure manner and respectful of privacy and individual autonomy.

b) Purposefully manipulative techniques

(67) 'Purposefully manipulative techniques' are not defined in the AI Act, but they should be understood as techniques that are designed or objectively aim to influence, alter, or control an individual's behaviour in a manner that undermines their individual autonomy and free choices. Manipulative techniques are typically designed to exploit cognitive biases, psychological vulnerabilities, or situational factors that make individuals more susceptible to influence. Because of their adaptability, AI systems are also able to respond well to a person's individual circumstances or vulnerabilities and increase the effectiveness and impact of manipulation at scale. While the manipulative capability is an important element to determine the nature of the technique, it is not necessary that the provider or deployer or the system itself deploying the manipulative techniques also intends to cause harm. 56 (68) While not all manipulative techniques operate beyond the threshold of conscious awareness, many do and there may be an overlap with subliminal techniques, since such techniques also ultimately have manipulative effects. Recital 29 AI Act clarifies that the prohibition in Article 5(1)(a) also covers techniques where individuals, even if they are aware of the influence attempt, may not be able to control or resist its manipulative

55 See Recital 29 AI Act.

56 See in this context Recital 28 and sections 3.2.2. and 3.2.3. of the Guidelines.

effect. 57 As a result, individuals are influenced or pushed into behaviour and decisions they would normally not have made if they were not subjected to the manipulative techniques to a point that undermines their individual autonomy or free choice.

An example of purposefully manipulative techniques is sensory manipulation where an AI system deploys background audio or images that lead to mood alterations, for example increasing anxiety and mental distress that influence users' behaviour to the point of creating significant harm.

Another example is personalised manipulation where an AI system that creates and tailors highly persuasive messages based on an individual's personal data or exploits other individual vulnerabilities influences their behaviour or choices to a point of creating significant harm.

(69) The prohibition against purposefully manipulative techniques also covers AI systems that manipulate individuals without any human intending them to do so. Article 5(1)(a) AI Act prohibits AI systems that deploy certain techniques or exhibit a specific manipulative behaviour. Therefore, it could also be the AI system that deploys such manipulative techniques, rather than the provider or the deployer that has designed or used the system in this way.

For example, regardless of whether the provider intends it, an AI system may learn manipulative techniques because the data on which it is trained contain many instances of manipulative techniques, 58 or because reinforcement learning from human feedback can be 'gamed' through manipulative techniques. 59

By contrast, if the manipulative behaviour of the system is merely incidental, the system should not be considered deploying purposefully manipulative techniques as long as the provider has taken appropriate preventive and mitigating measures in case significant harms are reasonably likely to occur (see section 3.2.3.c) below).

c) Deceptive techniques

(70) The AI Act does not define 'deceptive techniques'. Recital 29 AI Act clarifies that these are techniques that subvert or impair a person's autonomy, decision-making or free choice in ways that the person is not consciously aware or, where it is aware, can still be deceived or is not able to control or resist them. 'Deceptive techniques' deployed by AI systems should be understood to involve presenting false or misleading information with the objective or the effect of deceiving individuals and influencing their behaviour in a manner that undermines their autonomy, decision-making and free choices. (71) In this context, the interplay between the prohibition in Article 5(1)(a) AI Act and the deployer's obligations in Article 50(4) AI Act to label 'deep fakes' and certain AI-

57 Recital 28 AI Act.

58 M. Carroll et al., Characterising Manipulation from AI Systems, In Equity and Access in Algorithms,Mechanisms, and Optimization (EAAMO '23), October 30-November 1, 2023, Boston, MA, USA. ACM, New York, NY, USA, 13 pages. https://doi.org/10. 1145/3617694.3623226 |:2303.09387.

59 D. Amodei, et al., Concrete Problems in AI Safety, 36th Conference on Neural Information Processing Systems (NeurIPS 2022). arXiv:1606.06565; J. Skalse et al. Defining and Characterizing Reward Gaming, Advances in Neural Information Processing Systems 35 (NeurIPS 2022) C. Denison et al., Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language, 36th Conference on Neural Information Processing Systems (NeurIPS 2022), Models, arXiv:2406.10162.

generated text publications on matters of public interest, 60 as well as the provider's obligation to ensure AI systems interacting with people are designed in a way to inform people that they are interacting with AI and not a human, 61 should be clarified. Such visible disclosure constitutes a mitigating measure that should also be enabled through design features embedded in the AI system provided by the provider, including technical measures enabling the detection of AI-generated and manipulated content. 62 The visible labelling of 'deep fakes' and chatbots reduces the risk of deception that is likely to arise once the AI-generated content is disseminated to the public and reduces the risk of harmful distorting effects on the individual's opinion- and belief-formation and behaviour.

(72) By contrast, the prohibition in Article 5(1)(a) AI Act has a much more limited scope. It may, for example, cover cases where a chatbot or deceptive AI-generated content presents false or misleading information in ways that aim to or have the effect of deceiving individuals and distorting their behaviour that would not have happened if they were not exposed to the interaction with the AI system or the deceptive AI generated content, in particular if this has not been visibly disclosed. 63 (73) As with purposefully manipulative techniques, the prohibition of deceptive techniques may also cover AI systems that deceive individuals without any human intending them to do so (see section 3.2.1.b)above). For example, regardless of whether their providers intend such an outcome, AI systems may learn deceptive techniques simply because this increases their performance for the task for which they were developed, for example by reinforced learning. 64

An example of deceptive techniques that may be deployed by AI is an AI chatbot that impersonates a friend of a person or a relative with synthetic voice and tries to pretend it is the person causing scams and significant harms.

Another example is an AI system that learns to identify when it is under evaluation and temporarily halts any undesired behaviour, only to resume such behaviour once the evaluation period is over. 65 Such deceptive behaviour is particularly dangerous, since it defies any external human oversight over the system and may be prohibited if it is reasonably likely to cause significant harms.

60 Article 50(4) AI Act.

61 Article 50(1) AI Act.

62 Article 50(2) AI Act.

63 While in principle the transparency obligations in Article 50 AI Act aim to minimise the manipulative effects of deep fakes and chatbots, there might be instances and contexts where despite the information notices these deceptive techniques may still have significant effects on individuals and distort their behaviour to a point that undermine persons' individual autonomy and informed decision-making, so they should not be misused for disinformation and manipulation purposes and might still be covered in some cases by the prohibition in Article 5(1)a) if all other conditions of the ban are fulfilled (including the significant harms).

64 F. Ward, F. Toni, F. Belardinelli, T. Everitt, Honesty Is the Best Policy: Defining and Mitigating AI Deception (neurips.cc); Advances in Neural Information Processing Systems 36 (NeurIPS 2023); P. Park. et al. AI deception: A survey of examples, risks, and potential solutions [2406.10162] Patterns, Volume 5, Issue 5, 100988.

65 J. Lehman, J. Clune, D. Misevic, C. Adami, L. Altenberg, J. Beaulieu, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. Artificial life, 26(2):274-306, 2020.

By contrast, a generative AI system that incidentally presents false or misleading information and hallucinates 66 may not be considered to deploy deceptive techniques within the meaning of Article 5(1)(a) AI Act, taking into account the limitations and the state of the art of generative AI. In particular, this may be the case where the provider of the system has properly informed users about the system's limitations and integrated appropriate safeguards into the system to minimise such outcomes and provided that the system is not intended for, nor deployed in, sensitive contexts (e.g., health, education, elections) where serious harmful consequences are likely to occur (see also considerations in section 3.2.3.c) below).

d) Combination of techniques

(74) Article 5(1)(a) AI Act applies to subliminal, purposefully manipulative, or deceptive techniques, or to combinations of such techniques that can have a compound impact. As stated above, purposefully manipulative techniques may be also subliminal in nature, if they operate beyond the threshold of conscious awareness. (75) Furthermore, when purposefully manipulative and deceptive techniques are applied in combination, this may significantly influence the behaviour of individuals, leading them to make decisions based on unconscious manipulations and false beliefs. This combination may create a feedback loop where individuals are less likely to question or critically evaluate the information received, since the manipulative elements have already primed their cognitive biases and emotional responses.

3.2.2. With the objective or the effect of materially distorting the behaviour of a person or a group of persons

(76) A third condition for the prohibition in Article 5(1)(a) AI Act to apply is that the deployed subliminal, purposefully manipulative, or deceptive technique must have 'the objective, or the effect of materially distorting the behaviour of a person or a group of persons'. This implies a substantial impact on the behaviour where a person's autonomy and free choices are undermined, rather than a minor influence. However, intent is not a necessary requirement, since Article 5(1)(a) AI Act also covers practices that may only have the 'effect' of causing material distortion. There should be a plausible/reasonably likely causal link between the potential material distortion of the behaviour and the subliminal, purposefully manipulative or deceptive technique deployed by the AI system.

a) The concept of 'material distortion of the behaviour'

(77) The concept of 'material distortion of the behaviour' of a person or a group of persons is central to Article 5(1)(a) AI Act. It involves the deployment of subliminal, purposefully manipulative or deceptive techniques that are capable of influencing people's behaviour in a manner that appreciably impairs their ability to make an

66 'Hallucination' is a term used to describe a technical flaw in generative AI systems when they generate unwanted information that is fabricated or factually incorrect without this being intended by their developers. See more Ji Ziwei et al., Survey of Hallucination in Natural Language Generation | ACM Computing Surveys, 55, Issue 12, Article No.: 248, Pages 1 - 38.

informed decision, thereby causing them to behave in a way or to take a decision that they would otherwise not have taken.

(78) 'Appreciable impairment' refers to a substantially reduced ability to make informed and autonomous decisions, thereby causing individuals to behave in a way or to take a decision that they would otherwise not have taken. It goes beyond minor or negligible impacts and involves a significant distortion or hindrance in decision-making and free choice, including in relation to opinion- and belief-formation. This suggests that 'material distortion' involves a degree of coercion, manipulation, or deception that goes beyond lawful persuasion, which falls outside the scope of the prohibition (see section 3.5.1. below). (79) An 'informed decision' requires an understanding and knowledge of the relevant information, including the available options, the risks and benefits of each choice, the possible effects of the AI system on their behaviour, and, as appropriate, other contextual information that is important for the decision-making or the behaviour of the person. (80) For the interpretation of the concept of 'material distortion of behaviour', Union consumer protection law, in particular, Directive 2005/29/EC (Unfair Commercial Practices Directive or 'UCPD'), may constitute a valid source of inspiration. The UCPD prohibits various unfair, misleading, and aggressive commercial practices (Articles 5 to 9 UCPD) capable of causing consumers to make transactional decisions that they would otherwise not have made. According to the CJEU and the Commission guidance on the UCPD, 67 there is no need to prove that a consumer's economic behaviour has been distorted, it suffices to establish that a commercial practice is 'likely' (i.e. capable) of impacting an average consumer's transactional decision. 68 The CJEU has also underscored that even accurate information may be misleading if presented in a way that distorts the consumer's decision-making process. 69 National enforcement authorities are tasked to investigate the specific facts and circumstances of each case ( in concreto) and to evaluate the potential impact of the practice on the average consumer's decision-making process ( in abstract ). 70 For that purpose, they must take the point of view of the 'average' consumer, which is the benchmark developed by the CJEU, now integrated in the UCPD. 71

67 See also Commission Guidance on the interpretation and application of Directive 2005/29/EC of the European Parliament and of the Council concerning unfair business-to-consumer commercial practices in the internal market, (OJ C 526, 29.12.2021, p. 1) . EU:C:2016:800 , Case C-611/14,

68 Judgment of the Court of Justice (Fifth Chamber) of 26 October 2016. Canal Digital Danmark A/S para 73.

69 Judgment of the Court of Justice of 19 December 2013, Trento Sviluppo and Centrale Adriatica, C-281/12, EU:C:2013:859.

70 Commission Notice - Guidance on the interpretation and application of Directive 2005/29/EC of the European Parliament and of the Council concerning unfair business-to-consumer commercial practices in the internal market (OJ C 526, 29.12.2021, p. 1).

71 See Recitals 18 and 19 UCPD. 'Average consumer' is a person who is reasonably well informed and reasonably observant and circumspect, considering social, cultural and linguistic factors. The average consumer test is not a statistical test (i.e. it does not require to prove that a certain percentage of consumers would have been materially distorted/appreciably impaired by a business practice). The test is based on the principle of proportionality. The UCPD adopted this notion to strike the right balance between the need to protect consumers and the promotion of free trade in an openly competitive market. Courts and authorities will have to exercise their own faculty of judgment to determine the typical reaction of the average consumer in a given case. In the UCPD Guidance, the Commission advised them to make use of behavioural insights and other data. Case C-646/22, Compass Banca , clarifies that the definition of the average consumer does not exclude the possibility that an individual's decision-making capacity may be impaired by constraints, such as cognitive biases. Judgment of the Court (Fifth Chamber) of 14 November 2024. Compass Banca SpA v Autorità Garante della Concorrenza e del Mercato (AGCM), Case C-646/22, EU:C:2024:957.

(81) In the context of the prohibition in Article 5(1)(a) AI Act, market surveillance authorities must also investigate each case's specific facts and circumstances, assessing whether the subliminal, purposefully manipulative or deceptive technique deployed by the AI system is likely to appreciably impair the decision-making, individual autonomy and free choice of an 'average' individual within a targeted group when the system affects a group of persons in a manner that is reasonably likely to cause significant harm. Such an interpretation seems justified given that the AI Act intends to complement the UCPD 72 and must be applied in a consistent manner. At the same time, given that Article 5(1)(a) AI Act also refers to the possibility to distort the 'behaviour of a natural person' and, if the perspective of the 'average' individual proves difficult or ineffective to assess in certain contexts (for example due to very tailored or 'personalised' manipulation or harmful effects on specific vulnerable groups), specific cases may be examined also from the perspective of specific individuals by assessing to what extent an AI system deploying subliminal, purposefully manipulate or deceptive techniques is capable of undermining their individual autonomy in concrete cases and significant harm has occurred or is likely to occur.

b) Scenario 1: Prohibited AI systems 'with the objective to' materially distort behaviour

(82) Article 5(1)(a) AI Act applies to AI systems deploying the above-mentioned techniques and having as a first scenario 'the objective to materially distort the behaviour of a person or a group of persons' . Such an objective may be pursued by the provider or the deployer of the AI system, or by the system itself within the implicit objectives it may pursue. 73 This objective should be distinguished from the 'intended purpose' of the AI system (Article 3(12) AI Act). Even if intended by the provider, the manipulative objective is in most cases not the purpose of the use for which the system is offered and it is often neither transparent, nor specified as such in the information supplied by the provider (e.g., in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation).

For example, a chatbot that may be used in different contexts is designed to use subliminal messaging techniques, such as flashing brief visual cues and embedding inaudible auditory signals or to exploit emotional dependency or specific vulnerabilities of users in advertisements. These techniques are deployed 'with the objective to' materially distort users' behaviour, since they are objectively a design feature that aims to influence consumers' purchasing decisions without their conscious awareness, to push people to take significantly harmful financial decisions.