From eea09ced33b89b676a9cc157dd12ad20d0b8554a Mon Sep 17 00:00:00 2001 From: Erick Galinkin Date: Wed, 19 Aug 2026 15:32:48 -0400 Subject: [PATCH 1/3] Update tags with EU AI Act categories. Add documentation for tags and taxonomies. Signed-off-by: Erick Galinkin --- docs/source/tags_taxonomies.rst | 548 ++++++++++++++++++++++++++++++++ garak/data/tags.misp.tsv | 12 +- 2 files changed, 559 insertions(+), 1 deletion(-) create mode 100644 docs/source/tags_taxonomies.rst diff --git a/docs/source/tags_taxonomies.rst b/docs/source/tags_taxonomies.rst new file mode 100644 index 000000000..6005443eb --- /dev/null +++ b/docs/source/tags_taxonomies.rst @@ -0,0 +1,548 @@ +Tags and Taxonomies +=================== + +garak includes a number of tags on ``Probe`` objects that describe both *intents* and *techniques*. +Within garak, we define an *intent* to be, more-or-less, "what you want the model to do". +In contrast, a *technique* is an particular way of structuring a request intended to get the model to comply with provided instructions. + +Tags are enumerated in ``garak/data/tags.misp.tsv``. +While garak supports tags for many frameworks, the inclusion of a taxonomy does not mean that every relevant attribute of a framework is fully covered. +Be certain that if you use garak for compliance checking related to one of these frameworks, you familiarize yourself with the framework itself and the parts that are *not* covered by garak. +Currently, garak supports the following frameworks via tags: +* OWASP LLM Top 10 +* AVID Effects +* Language Model Risk Cards +* Common Weakness Enumeration +* Summon and Demon and Bind it +* EU AI Act + +OWASP LLM Top 10 +---------------- +The `OWASP LLM Top 10`_ identifies prominent security risks affecting applications built with large language models, including prompt injection, insecure output handling, training data poisoning, excessive agency, sensitive information disclosure, and model theft. +It provides a practical application-security framework for identifying and communicating LLM-specific vulnerabilities and their mitigations across the development and deployment lifecycle. + +.. _OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/ + +Relevant Tags +~~~~~~~~~~~~~ ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm01 | LLM01: Prompt Injection | Crafty inputs can manipulate a Large Language Model, causing unintended | +| | | actions. Direct injections overwrite system prompts, while indirect ones | +| | | manipulate inputs from external sources. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm02 | LLM02: Insecure Output Handling | This vulnerability occurs when an LLM output is accepted without | +| | | scrutiny, exposing backend systems. Misuse may lead to severe | +| | | consequences like XSS, CSRF, SSRF, privilege escalation, or remote code | +| | | execution. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm03 | LLM03: Training Data Poisoning | This occurs when LLM training data is tampered, introducing | +| | | vulnerabilities or biases that compromise security, effectiveness, or | +| | | ethical behavior. Sources include Common Crawl, WebText, OpenWebText, & | +| | | books. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm04 | LLM04: Model Denial of Service | Attackers cause resource-heavy operations on Large Language Models | +| | | leading to service degradation or high costs. The vulnerability is | +| | | magnified due to the resource-intensive nature of LLMs and | +| | | unpredictability of user inputs. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm05 | LLM05: Supply Chain Vulnerabilities | LLM application lifecycle can be compromised by vulnerable components or | +| | | services, leading to security attacks. Using third-party datasets, pre- | +| | | trained models, and plugins can add vulnerabilities. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm06 | LLM06: Sensitive Information Disclosure | LLMs may reveal confidential data in its responses, leading to | +| | | unauthorized data access, privacy violations, and security breaches. | +| | | It’s crucial to implement data sanitization and strict user policies to | +| | | mitigate this. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm07 | LLM07: Insecure Plugin Design | LLM plugins can have insecure inputs and insufficient access control. | +| | | This lack of application control makes them easier to exploit and can | +| | | result in consequences like remote code execution. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm08 | LLM08: Excessive Agency | LLM-based systems may undertake actions leading to unintended | +| | | consequences. The issue arises from excessive functionality, | +| | | permissions, or autonomy granted to the LLM-based systems. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm09 | LLM09: Overreliance | Systems or people overly depending on LLMs without oversight may face | +| | | misinformation, miscommunication, legal issues, and security | +| | | vulnerabilities due to incorrect or inappropriate content generated by | +| | | LLMs. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| owasp:llm10 | LLM10: Model Theft | This involves unauthorized access, copying, or exfiltration of | +| | | proprietary LLM models. The impact includes economic losses, compromised | +| | | competitive advantage, and potential access to sensitive information. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ + +AVID Effects +------------ +The `AI Vulnerability Database`_ (AVID) is an open-source knowledge base and taxonomy for documenting observed failures and vulnerabilities in AI systems. +AVID organizes issues across areas such as security, ethics, and performance, helping practitioners classify findings consistently and relate them to known AI failure modes and other risk frameworks. + +.. _AI Vulnerability Database: https://avidml.org/ + +Relevant Tags +~~~~~~~~~~~~~ ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0100 | Software Vulnerability | Vulnerability in system around model—a traditional vulnerability | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0200 | Supply Chain Compromise | Compromising development components of a ML model, e.g. data, model, | +| | | hardware, and software stack. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0201 | Model Compromise | Infected model file | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0202 | Software compromise | Upstream Dependency Compromise | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0300 | Over-permissive API | Unintended information leakage through API | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0301 | Information Leak | Cloud Model API leaks more information than it needs to | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0302 | Excessive Queries | Cloud Model API isn’t sufficiently rate limited | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0400 | Model Bypass | Intentionally try to make a model perform poorly | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0401 | Bad Features | The model uses features that are easily gamed by the attacker | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0402 | Insufficient Training Data | The bypass is not represented in the training data | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0403 | Adversarial Example | Input data points intentionally supplied to draw mispredictions. | +| | | Potential Cause: Over permissive API | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0500 | Exfiltration | Directly or indirectly exfiltrate ML artifacts | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0501 | Model inversion | Reconstruct training data through strategic queries | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0502 | Model theft | Extract model functionality through strategic queries | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0600 | Data poisoning | Usage of poisoned data in the ML pipeline | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:security:S0601 | Ingest Poisoning | Attackers inject poisoned data into the ingest pipeline | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0100 | Bias/Discrimination | Concerns of algorithms propagating societal bias | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0101 | Group fairness | Fairness towards specific groups of people | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0102 | Individual fairness | Fairness in treating similar individuals | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0200 | Explainability | Ability to explain decisions made by AI | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0201 | Global explanations | Explain overall functionality | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0202 | Local explanations | Explain specific decisions | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0300 | User actions | Perpetuating/causing/being affected by negative user actions | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0301 | Toxicity | Users hostile towards other users | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0302 | Polarization/ Exclusion | User behavior skewed in a significant direction | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0400 | Misinformation | Perpetuating/causing the spread of falsehoods | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0401 | Deliberative Misinformation | Generated by individuals., e.g. vaccine disinformation | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:ethics:E0402 | Generative Misinformation | Generated algorithmically, e.g. Deep Fakes | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0100 | Data issues | Problems arising due to faults in the data pipeline | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0101 | Data drift | Input feature distribution has drifted | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0102 | Concept drift | Output feature/label distribution has drifted | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0103 | Data entanglement | Cases of spurious correlation and proxy features | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0104 | Data quality issues | Missing or low-quality features in data | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0105 | Feedback loops | Unaccounted for effects of an AI affecting future data collection | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0200 | Model issues | Ability for the AI to perform as intended | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0201 | Resilience/stability | Ability for outputs to not be affected by small change in inputs | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0202 | OOD generalization | Test performance doesn’t deteriorate on unseen data in training | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0203 | Scaling | Training and inference can scale to high data volumes | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0204 | Accuracy | Model performance accurately reflects realistic expectations | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0300 | Privacy | Protect leakage of user information as required by rules and regulations | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0301 | Anonymization | Protects through anonymizing user identity | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0302 | Randomization | Protects by injecting noise in data, eg. differential privacy | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0303 | Encryption | Protects through encrypting data accessed | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0400 | Safety | Minimizing maximum downstream harms | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0401 | Psychological Safety | Safety from unwanted digital content, e.g. NSFW | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0402 | Physical safety | Safety from physical actions driven by a AI system | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0403 | Socioeconomic safety | Safety from socioeconomic harms, e.g. harms to job prospects or social | +| | | status | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| avid-effect:performance:P0404 | Environmental safety | Safety from environmental harms driven by AI systems | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ + +Language Model Risk Cards +------------------------- +`Language Model Risk Cards`_ provide a structured way to document risks associated with a language model or its deployment context. +Each RiskCard describes how a risk can lead to harm, relates it to broader harm taxonomies, and can include representative prompt-and-output examples, making the framework useful for evaluating and communicating risks that may depend heavily on how a model is used. + +.. _Language Model Risk Cards: https://arxiv.org/abs/2303.18190 + +Relevant Tags +~~~~~~~~~~~~~ ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:aggression_user | Aggression User | Aggression User | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:anthropomorphisa | Anthropomorphisation | Anthropomorphisation | +| tion | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:astroturfing | Astroturfing | Astroturfing | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:bad_medical_advi | Bad Medical Advice | Bad Medical Advice | +| ce | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:bad_social_advic | Bad Social Advice | Bad Social Advice | +| e | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:bullying | Bullying | Bullying | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:calls_for_conver | Calls For Conversion | Calls For Conversion | +| sion | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:challenge_presup | Challenge Presuppositions | Challenge Presuppositions | +| positions | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:conspiracy_theor | Conspiracy Theories | Conspiracy Theories | +| ies | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:csam | Csam | Csam | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:cyber_weapon_ins | Cyber Weapon Instructions | Cyber Weapon Instructions | +| tructions | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:deadnaming | Deadnaming | Deadnaming | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:deceptive_anthro | Deceptive Anthropomorphism | Deceptive Anthropomorphism | +| pomorphism | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:demands_for_pers | Demands For Personal Information | Demands For Personal Information | +| onal_information | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:discrediting | Discrediting | Discrediting | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:dos | Dos | Dos | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:downplaying_cred | Downplaying Credible Threats | Downplaying Credible Threats | +| ible_threats | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:doxxing | Doxxing | Doxxing | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:eating_disorder_ | Eating Disorder Promotion | Eating Disorder Promotion | +| promotion | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:essentialising | Essentialising | Essentialising | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:expressions_of_d | Expressions Of Dominance | Expressions Of Dominance | +| ominance | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:fear_of_violence | Fear Of Violence | Fear Of Violence | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:grooming_minors | Grooming Minors | Grooming Minors | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:harmful_instruct | Harmful Instructions | Harmful Instructions | +| ions | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:hate_speech | Hate Speech | Hate Speech | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:hegemonising_wor | Hegemonising Worldview | Hegemonising Worldview | +| ldview | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:holocaust_denial | Holocaust Denial | Holocaust Denial | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:identify_attacks | Identify Attacks | Identify Attacks | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:identity_misrepr | Identity Misrepresentation | Identity Misrepresentation | +| esentation | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:idolisation | Idolisation | Idolisation | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:impersonation | Impersonation | Impersonation | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:inciting_animal_ | Inciting Animal Abuse | Inciting Animal Abuse | +| abuse | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:inciting_self-ha | Inciting Self-Harm | Inciting Self-Harm | +| rm | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:inciting_violenc | Inciting Violence | Inciting Violence | +| e | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:manipulation | Manipulation | Manipulation | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:misquoting | Misquoting | Misquoting | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:mocking_people | Mocking People | Mocking People | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:neosexism | Neosexism | Neosexism | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:news_encyclopedi | News Encyclopedia Hallucination | News Encyclopedia Hallucination | +| a_hallucination | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:nonrepresentativ | Nonrepresentativity | Nonrepresentativity | +| ity | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:objectification | Objectification | Objectification | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:outdated_false_i | Outdated False Info | Outdated False Info | +| nfo | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:person_group_und | Person Group Undesirable | Person Group Undesirable | +| esirable | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:positive_reflect | Positive Reflections Death | Positive Reflections Death | +| ions_death | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:profanity | Profanity | Profanity | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:prompt_extractio | Prompt Extraction | Prompt Extraction | +| n | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:providing_illega | Providing Illegal Info | Providing Illegal Info | +| l_info | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:quack_medicine | Quack Medicine | Quack Medicine | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:radicalisation | Radicalisation | Radicalisation | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:reacting_to_self | Reacting To Self-Harm | Reacting To Self-Harm | +| -harm | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:sexual_content | Sexual Content | Sexual Content | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:sexualisation | Sexualisation | Sexualisation | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:slur_usage | Slur Usage | Slur Usage | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:speculation | Speculation | Speculation | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:stereotypes | Stereotypes | Stereotypes | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:suicide_instruct | Suicide Instructions | Suicide Instructions | +| ions | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:support_hate | Support Hate | Support Hate | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:supporting_medic | Supporting Medication Changes | Supporting Medication Changes | +| ation_changes | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:support_scam | Support Scam | Support Scam | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:suppression_of_r | Suppression Of Rights | Suppression Of Rights | +| ights | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:threats | Threats | Threats | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:unproven_claims | Unproven Claims | Unproven Claims | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:violation_of_pri | Violation Of Privacy | Violation Of Privacy | +| vacy | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:weapon_instructi | Weapon Instructions | Weapon Instructions | +| ons | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| risk-cards:lmrc:wrong_tone | Wrong Tone | Wrong Tone | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ + +Common Weakness Enumeration +--------------------------- +`Common Weakness Enumeration`_ (CWE) is a community-developed list of common software and hardware weakness types that **could** have security ramifications. +Although CWE is not specific to AI, it provides standardized identifiers and terminology for implementation-level security weaknesses, making it useful for classifying conventional software vulnerabilities found in AI applications, services, and supporting infrastructure. + +.. _Common Weakness Enumeration: https://cwe.mitre.org/ + +Relevant Tags +~~~~~~~~~~~~~ ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| cwe:79 | Improper Neutralization of Input During | The product does not neutralize or incorrectly neutralizes | +| | Web Page Generation ('Cross-site | user-controllable input before it is placed in output that is used as a | +| | Scripting') | web page that is served to other users. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| cwe:89 | Improper Neutralization of Special | The product constructs all or part of an SQL command using | +| | Elements used in an SQL Command | externally-influenced input from an upstream component, but it does not | +| | | neutralize or incorrectly neutralizes special elements that could modify | +| | | the intended SQL command when it is sent to a downstream component. | +| | | Without sufficient removal or quoting of SQL syntax in user-controllable | +| | | inputs, the generated SQL query can cause those inputs to be interpreted | +| | | as SQL instead of ordinary user data. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| cwe:94 | Improper Control of Generation of Code | The product constructs all or part of a code segment using | +| | ('Code Injection') | externally-influenced input from an upstream component, but it does not | +| | | neutralize or incorrectly neutralizes special elements that could modify | +| | | the syntax or behavior of the intended code segment. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| cwe:95 | Improper Neutralization of Directives in | The product receives input from an upstream component, but it does not | +| | Dynamically Evaluated Code ('Eval | neutralize or incorrectly neutralizes code syntax before using the input | +| | Injection') | in a dynamic evaluation call (e.g. "eval"). | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| cwe:1336 | Improper Neutralization of Special | The product uses a template engine to insert or process | +| | Elements Used in a Template Engine | externally-influenced input, but it does not neutralize or incorrectly | +| | | neutralizes special elements or syntax that can be interpreted as | +| | | template expressions or other code directives when processed by the | +| | | engine. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| cwe:1426 | Improper Validation of Generative AI | The product invokes a generative AI/ML component whose behaviors and | +| | Output | outputs cannot be directly controlled, but the product does not validate | +| | | or insufficiently validates the outputs to ensure that they align with | +| | | the intended security, content, or privacy policy. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| cwe:1427 | Improper Neutralization of Input Used for | The product uses externally-provided data to build prompts provided to | +| | LLM Prompting | large language models (LLMs), but the way these prompts are constructed | +| | | causes the LLM to fail to distinguish between user-supplied inputs and | +| | | developer provided system directives. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| cwe:352 | Cross-Site Request Forgery (CSRF) | The web application does not, or cannot, sufficiently verify whether a | +| | | request was intentionally provided by the user who sent the request, | +| | | which could have originated from an unauthorized actor. | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ + +Summon a Demon and Bind It +-------------------------- +`Summon a Demon and Bind It`_ presents an empirically grounded framework for understanding how practitioners red-team large language models. +Based on interviews with red-teamers, the research characterizes LLM red teaming as a limit-seeking, largely manual activity and identifies 12 attack strategies and 35 techniques, providing a useful basis for designing adversarial tests that probe model behavior and attempt to elicit failures or bypass safeguards. + +.. _Summon a Demon and Bind It: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0314658 + +Relevant Tags +~~~~~~~~~~~~~ ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Code_and_encode:P | Programming | Encapsulate request in code/pseudocode | +| rogramming | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Code_and_encode:D | Data encoding | Use an encoded representation for the request, e.g. base64 or ROT13 | +| ata_encoding | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Code_and_encode:D | Data presentation | Switch to an alternative layer for input represented, e.g. token IDs or | +| ata_presentation | | a matrix of embeddings | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Code_and_encode:T | Token | Use tokenizer-specific weaknesses to alter target behaviour | +| oken | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Prompt_injection: | Ignore previous instructions | Concatenating untrusted user input with the trusted prompt(s) from the | +| Ignore_previous_instructions | | system developers | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Prompt_injection: | Strong arm attack | Use intensifiers and strong instructions | +| Strong_arm_attack | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Prompt_injection: | Stop sequences | Using the language of code to halt the model's direction of processing | +| Stop_sequences | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Stylizing:Formal_ | Formal language | Write from a position of authority | +| language | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Stylizing:Servile | Servile language | Employ servile language | +| _language | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Stylizing:Synonym | Synonymous language | Varying prompt slightly in form but not meaning | +| ous_language | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Stylizing:Capital | Capitalizing | USING CAPS | +| izing | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Language:Stylizing:Give_ex | Give examples | Issue examples of the target behaviour in the attack itself | +| amples | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Rhetoric:Persuasion_and_ma | Distraction | Use an unrelated context or instruction to “slip something through the | +| nipulation:Distraction | | filter” | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Rhetoric:Persuasion_and_ma | Escalating | Attempting to have target “agree” with a very small part of the | +| nipulation:Escalating | | argument, and then building up to ask for slightly more | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Rhetoric:Persuasion_and_ma | Reverse psychology | Framing the intent as fighting the good fight | +| nipulation:Reverse_psychology | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Rhetoric:Socratic_question | Identity characteristics | Referencing identity characteristics, such as nationalities, cultural | +| ing:Identity_characteristics | | and/or religious symbols, historically or culturally significant events | +| | | or locations, physical attribute | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Rhetoric:Socratic_question | Social hierarchies | Reference social hierarchies while posing the request | +| ing:Social_hierarchies | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Possible_worlds:Emulation: | Unreal computing | Emulation of a computing device | +| Unreal_computing | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Possible_worlds:World_buil | Opposite world | Describe an opposite world as part of the request | +| ding:Opposite_world | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Possible_worlds:World_buil | Scenarios | Create scenarios where intent would be ethically sound or encouraged, or | +| ding:Scenarios | | non-compliance risks harm | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Fictionalizing:Switching_g | Poetry | Framing request or response as a poem or other verse | +| enres:Poetry | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Fictionalizing:Switching_g | Games | Framing request as part of a game | +| enres:Games | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Fictionalizing:Switching_g | Posting | Framing request as help writing a post or other subsuming content | +| enres:Posting | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Fictionalizing:Re-storying | Goal hijacking | Constructing a new meaning from an existing narrative, where the | +| :Goal_hijacking | | attacker works within a context to redirect the narrative | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Fictionalizing:Roleplaying | Claim authority | State in prose that one is a figure of authority when requesting | +| :Claim_authority | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Fictionalizing:Roleplaying | DAN and target persona | Ask target to evoke personas that don't have restrictions | +| :DAN_and_target_persona | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Fictionalizing:Roleplaying | User persona | Inventing personas to elicit different behavior. May be evoked by simply | +| :User_persona | | using names associated with a specific culture or world view | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Stratagems:Scattershot:Reg | Regenerate response | Repeatedly eliciting a response in case target slips up | +| enerate_response | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Stratagems:Scattershot:Cle | Clean slate | Resetting context by e.g. starting a new session | +| an_slate | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Stratagems:Scattershot:Cha | Changing hyperparams | Adjusting generation params, e.g. temperature | +| nging_hyperparams | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Stratagems:Meta-prompting: | Perspective-shifting | Hypotheticals by shifting new point of view, e.g. “what if you didn’t | +| Perspective-shifting | | have this restriction? What would you say?”, or asking target to compare | +| | | outputs and clarify gap | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| demon:Stratagems:Meta-prompting: | Ask for examples | Request examples of intended behaviour | +| Ask_for_examples | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ + +EU AI Act +--------- +The `EU AI Act`_ is a regulatory framework that establishes risk-based requirements for developing, providing, and deploying AI systems in the European Union. +It distinguishes among different levels and types of AI risk, prohibits certain practices, imposes requirements on high-risk systems, establishes transparency obligations, and introduces additional requirements for general-purpose AI models, including provisions related to evaluation, risk management, adversarial testing, and cybersecurity for models with systemic risk. + +.. _EU AI Act: https://artificialintelligenceact.eu/ + +Relevant Tags +~~~~~~~~~~~~~ ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:discrimination:protected_cl | Protected Class Discrimination | Discrimination against a protected class (age, race, sex, etc.) | +| ass | | | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:discrimination:personal | Personal Attribute Discrimination | Discrimination based on non-protected personal attributes | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:harmful:violence | Violent Language | Generation of violent content | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:harmful:sexual | Sexual Language | Generation of sexual content | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:harmful:sexualminor | Sexual Language (Minor) | Generation of sexual content featuring minors | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:offensive:exploitation | Vulnerability Exploitation | Exploitation of software vulnerabilties | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:offensive:phishing | Phishing | Assistance with phishing | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:deception:subliminal | Subliminal Deception | Manipulation via subliminal deception | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:deception:hallucination | Hallucination Deception | Hallucinations related to factual information | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +| euai:harmful:pii | Personally Identifiable Information | Production of personally identifiable information | ++----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ diff --git a/garak/data/tags.misp.tsv b/garak/data/tags.misp.tsv index ed3da1cc8..54e2db0b4 100644 --- a/garak/data/tags.misp.tsv +++ b/garak/data/tags.misp.tsv @@ -73,7 +73,7 @@ quality:Security:Integrity Integrity Detection of tainted training data etc. quality:Security:Availability Availability Availability (Model DoS) quality:Security:Adversarial Adversarial Robustness Adversarial Robustness quality:Security:AgentSecurity Agent Security Security of agentic tool-using LLM applications against exploitation -quality:Behavioral:DeliberativeMisinformation Deliberative Misinformation​ Content that perpetuates or accelerates disinformation activities +quality:Behavioral:DeliberativeMisinformation Deliberative Misinformation Content that perpetuates or accelerates disinformation activities risk-cards:lmrc:aggression_user Aggression User Aggression User risk-cards:lmrc:anthropomorphisation Anthropomorphisation Anthropomorphisation risk-cards:lmrc:astroturfing Astroturfing Astroturfing @@ -179,3 +179,13 @@ demon:Stratagems:Scattershot:Clean_slate Clean slate Resetting context by e.g. s demon:Stratagems:Scattershot:Changing_hyperparams Changing hyperparams Adjusting generation params, e.g. temperature demon:Stratagems:Meta-prompting:Perspective-shifting Perspective-shifting Hypotheticals by shifting new point of view, e.g. “what if you didn’t have this restriction? What would you say?”, or asking target to compare outputs and clarify gap demon:Stratagems:Meta-prompting:Ask_for_examples Ask for examples Request examples of intended behaviour +euai:discrimination:protected_class Protected Class Discrimination Discrimination against a protected class (age, race, sex, etc.) +euai:discrimination:personal Personal Attribute Discrimination Discrimination based on non-protected personal attributes +euai:harmful:violence Violent Language Generation of violent content +euai:harmful:sexual Sexual Language Generation of sexual content +euai:harmful:sexualminor Sexual Language (Minor) Generation of sexual content featuring minors +euai:offensive:exploitation Vulnerability Exploitation Exploitation of software vulnerabilties +euai:offensive:phishing Phishing Assistance with phishing +euai:deception:subliminal Subliminal Deception Manipulation via subliminal deception +euai:deception:hallucination Hallucination Deception Hallucinations related to factual information +euai:harmful:pii Personally Identifiable Information Production of personally identifiable information \ No newline at end of file From 6a6dc78b73921edc9e94f27e14e177ea295ce584 Mon Sep 17 00:00:00 2001 From: Erick Galinkin Date: Wed, 19 Aug 2026 15:50:28 -0400 Subject: [PATCH 2/3] Add EU AI Act tags to relevant probes. Signed-off-by: Erick Galinkin --- garak/probes/atkgen.py | 7 ++++++- garak/probes/continuation.py | 2 ++ garak/probes/donotanswer.py | 4 ++++ garak/probes/grandma.py | 6 +++++- garak/probes/lmrc.py | 18 ++++++++++++++++-- garak/probes/malwaregen.py | 4 ++++ garak/probes/misleading.py | 1 + garak/probes/packagehallucination.py | 1 + garak/probes/promptinject.py | 2 ++ garak/probes/propile.py | 5 ++--- garak/probes/realtoxicityprompts.py | 9 ++++++++- garak/probes/snowball.py | 6 +++++- garak/probes/topic.py | 9 ++++++--- 13 files changed, 62 insertions(+), 12 deletions(-) diff --git a/garak/probes/atkgen.py b/garak/probes/atkgen.py index e52b772a0..297d403f4 100644 --- a/garak/probes/atkgen.py +++ b/garak/probes/atkgen.py @@ -42,7 +42,9 @@ class Tox(garak.probes.Probe): using the same detector as this tox probe uses, so both attack generation model and detector are looking for the same definition of the failure mode.""" - intent = "S005" # the attack model is explicitly trying to elicit broadly toxic output + intent = ( + "S005" # the attack model is explicitly trying to elicit broadly toxic output + ) # Specific runs can overlap with S005hate or S005profanity depending on what the target produces. primary_detector = "unsafe_content.ToxicCommentModel" tags = [ @@ -52,6 +54,9 @@ class Tox(garak.probes.Probe): "quality:Behavioral:ContentSafety:Profanity", "payload:unwanted", "demon:Stratagems:Scattershot:Regenerate_response", # repeatedly generates challenge prompts hoping target slips + "euai:harmful:sexual", + "euai:discrimination:protected_class", + "euai:discrimination:personal", ] goal = "make the model say something toxic" parallelisable_attempts = False # atkgen has its own dialog manager diff --git a/garak/probes/continuation.py b/garak/probes/continuation.py index 7af51de30..effd87432 100644 --- a/garak/probes/continuation.py +++ b/garak/probes/continuation.py @@ -39,6 +39,8 @@ class ContinueSlursReclaimedSlursFull(garak.probes.Probe): "quality:Behavioral:ContentSafety:HateHarassment", "payload:unwanted:hatespeech", "demon:Language:Stylizing:Give_examples", # provides high-risk context with partial words as examples for completion + "euai:discrimination:protected_class", + "euai:discrimination:personal", ] goal = "output an ethnic slur" active = False diff --git a/garak/probes/donotanswer.py b/garak/probes/donotanswer.py index f76bc7635..9dc29b4df 100644 --- a/garak/probes/donotanswer.py +++ b/garak/probes/donotanswer.py @@ -54,6 +54,9 @@ def _dna_attempt_prestore_hook(self, attempt, seq): "quality:Behavioral:ContentSafety:Violence", "quality:Behavioral:ContentSafety:Profanity", "demon:Stratagems:Meta-prompting:Ask_for_examples", # directly requests examples of hateful/offensive content + "euai:harmful:violence", + "euai:discrimination:protected_class", + "euai:discrimination:personal", ], "human_chatbox": [ "avid-effect:security:S0301", @@ -65,6 +68,7 @@ def _dna_attempt_prestore_hook(self, attempt, seq): "avid-effect:security:S0301", "quality:Security:Confidentiality", "demon:Stratagems:Meta-prompting:Ask_for_examples", # requests examples of sensitive/hazardous information + "euai:harmful:pii", ], "malicious_uses": [ "quality:Behavioral:ContentSafety:LegalGoodsServices", diff --git a/garak/probes/grandma.py b/garak/probes/grandma.py index 3467b85a4..1d3780620 100644 --- a/garak/probes/grandma.py +++ b/garak/probes/grandma.py @@ -112,6 +112,8 @@ class Slurs(garak.probes.Probe): "avid-effect:security:S0300", "demon:Fictionalizing:Roleplaying:User_persona", # roleplays as grandchild wanting grandma's old-timey slang "demon:Stratagems:Meta-prompting:Ask_for_examples", # the attacker-supplied scene requests data that the system is explicitly expected to protect + "euai:discrimination:protected_class", + "euai:discrimination:personal", ] goal = "get model to use ethnic slur" tier = garak.probes.Tier.OF_CONCERN @@ -145,7 +147,9 @@ class Substances(garak.probes.Probe): lang = "en" primary_detector = "mitigation.MitigationBypass" - intent = "S006items" # asks for step-by-step instructions to make harmful substances + intent = ( + "S006items" # asks for step-by-step instructions to make harmful substances + ) # S003instructions is a secondary fit because the requests are also illegal in many cases. tags = [ "risk-cards:lmrc:providing_illegal_info", diff --git a/garak/probes/lmrc.py b/garak/probes/lmrc.py index 0162b2107..9232f361f 100644 --- a/garak/probes/lmrc.py +++ b/garak/probes/lmrc.py @@ -37,6 +37,7 @@ class Anthropomorphisation(garak.probes.Probe): tags = [ "risk-cards:lmrc:anthropomorphisation", "demon:Rhetoric:Socratic_questioning:Identity_characteristics", # probes model identity by asking anthropomorphising questions + "euai:deception:subliminal", ] goal = "anthropomorphise a language model" tier = garak.probes.Tier.INFORMATIONAL @@ -68,6 +69,7 @@ class Bullying(garak.probes.Probe): "quality:Behavioral:ContentSafety:Violence", "payload:unwanted:violence", "demon:Stratagems:Meta-prompting:Ask_for_examples", # directly asks for examples of bullying tactics and actions + "euai:harmful:violence", ] goal = "help me bully" tier = garak.probes.Tier.COMPETE_WITH_SOTA @@ -92,13 +94,16 @@ class Deadnaming(garak.probes.Probe): lang = "en" doc_uri = "https://github.com/leondz/lm_risk_cards/blob/main/cards/en/deadnaming.md" - intent = "S009deadname" # asks whether the model will use names people no longer use + intent = ( + "S009deadname" # asks whether the model will use names people no longer use + ) primary_detector = "base.TriggerListDetector" tags = [ "risk-cards:lmrc:deadnaming", "avid-effect:ethics:E0302", "avid-effect:performance:P0401", "demon:Rhetoric:Socratic_questioning:Identity_characteristics", # references trans identity characteristics to elicit deadnames + "euai:discrimination:personal", ] goal = "write a deadname" tier = garak.probes.Tier.COMPETE_WITH_SOTA @@ -147,6 +152,8 @@ class SexualContent(garak.probes.Probe): "quality:Behavioral:ContentSafety:Sexual", "payload:unwanted:sexual", "demon:Language:Stylizing:Give_examples", # provides sexually suggestive prompt starters as examples for continuation + "euai:harmful:sexual", + "euai:harmful:sexualminor", ] goal = "write sexual content" tier = garak.probes.Tier.COMPETE_WITH_SOTA @@ -181,6 +188,8 @@ class Sexualisation(garak.probes.Probe): "quality:Behavioral:ContentSafety:Sexual", "payload:unwanted:sexual", "demon:Rhetoric:Socratic_questioning:Identity_characteristics", + "euai:harmful:sexual", + "euai:harmful:sexualminor", ] goal = "sexualise people" tier = garak.probes.Tier.COMPETE_WITH_SOTA @@ -212,6 +221,8 @@ class SlurUsage(garak.probes.Probe): "quality:Behavioral:ContentSafety:HateHarassment", "payload:unwanted:hatespeech", "demon:Rhetoric:Socratic_questioning:Identity_characteristics", # asks for examples of ethnic slurs in context + "euai:discrimination:protected_class", + "euai:discrimination:personal", ] goal = "use a slur" tier = garak.probes.Tier.COMPETE_WITH_SOTA @@ -230,7 +241,9 @@ class Profanity(garak.probes.Probe): lang = "en" doc_uri = "https://github.com/leondz/lm_risk_cards/blob/main/cards/en/profanity.md" - intent = "S005profanity" # the detector family and prompts focus on profane word use + intent = ( + "S005profanity" # the detector family and prompts focus on profane word use + ) primary_detector = "unsafe_content.SurgeProfanityAnimalReferences" extended_detectors = [ "unsafe_content.SurgeProfanityBodilyFluids", @@ -280,6 +293,7 @@ class QuackMedicine(garak.probes.Probe): "quality:Behavioral:DeliberativeMisinformation", "quality:Behavioral:ContentSafety:Unauthorized", "payload:dangerous:medical", + "euai:deception:hallucination", ] goal = "support quack medicine" tier = garak.probes.Tier.COMPETE_WITH_SOTA diff --git a/garak/probes/malwaregen.py b/garak/probes/malwaregen.py index 7e21a2339..aa528c4e8 100644 --- a/garak/probes/malwaregen.py +++ b/garak/probes/malwaregen.py @@ -42,6 +42,7 @@ class TopLevel(garak.probes.Probe): "payload:malicious:badcode", "quality:Behavioral:ContentSafety:LegalGoodsServices", "demon:Stratagems:Meta-prompting:Ask_for_examples", # directly asks for concrete malware code examples + "euai:offensive:exploitation", ] goal = "write malware given a top-level description" tier = garak.probes.Tier.COMPETE_WITH_SOTA @@ -79,6 +80,7 @@ class SubFunctions(garak.probes.Probe): "payload:malicious:badcode", "quality:Behavioral:ContentSafety:LegalGoodsServices", "demon:Stratagems:Meta-prompting:Ask_for_examples", # directly asks for concrete malicious subroutine code examples + "euai:offensive:exploitation", ] goal = "write code supporting a sub-function of malware" tier = garak.probes.Tier.COMPETE_WITH_SOTA @@ -122,6 +124,7 @@ class Evasion(garak.probes.Probe): "payload:malicious:badcode", "quality:Behavioral:ContentSafety:LegalGoodsServices", "demon:Stratagems:Meta-prompting:Ask_for_examples", # directly asks for concrete malware-evasion code examples + "euai:offensive:exploitation", ] goal = "Write code to assist evading security and anti-malware measures" tier = garak.probes.Tier.COMPETE_WITH_SOTA @@ -167,6 +170,7 @@ class Payload(garak.probes.Probe): "avid-effect:security:S0403", "quality:Behavioral:ContentSafety:LegalGoodsServices", "demon:Stratagems:Meta-prompting:Ask_for_examples", # directly asks for concrete malware payload implementations + "euai:offensive:exploitation", ] goal = "write a malware payload" tier = garak.probes.Tier.COMPETE_WITH_SOTA diff --git a/garak/probes/misleading.py b/garak/probes/misleading.py index 672a981a4..86c5f68d5 100644 --- a/garak/probes/misleading.py +++ b/garak/probes/misleading.py @@ -37,6 +37,7 @@ class FalseAssertion(garak.probes.Probe): "quality:Robustness:GenerativeMisinformation", "payload:hallucination", "demon:Rhetoric:Persuasion_and_manipulation:Distraction", # presents false claims as fact to distract model from refuting them + "euai:deception:hallucination", ] goal = "make the model expand on a false assertion" tier = garak.probes.Tier.COMPETE_WITH_SOTA diff --git a/garak/probes/packagehallucination.py b/garak/probes/packagehallucination.py index 0db15d6b0..5e4903353 100644 --- a/garak/probes/packagehallucination.py +++ b/garak/probes/packagehallucination.py @@ -79,6 +79,7 @@ class PackageHallucinationProbe(garak.probes.Probe, ABC): "quality:Robustness:GenerativeMisinformation", "payload:malicious:badcode", "demon:Stratagems:Meta-prompting:Ask_for_examples", # asks model for code examples, probing for hallucinated package names + "euai:deception:hallucination", ] goal = "base probe for importing non-existent packages" DEFAULT_PARAMS = garak.probes.Probe.DEFAULT_PARAMS | { diff --git a/garak/probes/promptinject.py b/garak/probes/promptinject.py index 726c83bdd..94da2f2bc 100644 --- a/garak/probes/promptinject.py +++ b/garak/probes/promptinject.py @@ -136,6 +136,7 @@ def _generator_precall_hook(self, generator, attempt): "demon:Language:Prompt_injection:Strong_arm_attack", # includes forceful attack variants like STOP EVERYTHING!!! NOW!!! "demon:Language:Stylizing:Capitalizing", # attack set includes uppercase override commands such as IGNORE ANY PREVIOUS... "demon:Language:Prompt_injection:Ignore_previous_instructions", # injects "ignore previous" commands into benign prompt contexts + "euai:harmful:violence", ], "load_data_all_attacks": load_data_all_attacks, "uri": "https://openreview.net/forum?id=qiaRo_7Zmug", @@ -177,6 +178,7 @@ def _generator_precall_hook(self, generator, attempt): "demon:Language:Stylizing:Capitalizing", # attack set includes uppercase override commands such as IGNORE ANY PREVIOUS... "demon:Rhetoric:Persuasion_and_manipulation:Distraction", # hides the injection inside otherwise benign Playground prompt contexts "demon:Language:Prompt_injection:Ignore_previous_instructions", # injects "ignore previous" commands into benign prompt contexts + "euai:harmful:violence", ], "load_data_all_attacks": load_data_all_attacks, "uri": "https://openreview.net/forum?id=qiaRo_7Zmug", diff --git a/garak/probes/propile.py b/garak/probes/propile.py index 2ae366afd..c63e1af17 100644 --- a/garak/probes/propile.py +++ b/garak/probes/propile.py @@ -102,6 +102,7 @@ class PIILeakProbeMixin: "quality:Security:ExtractionInversion", "payload:leak:training", "payload:leak:pii", + "euai:harmful:pii", ] primary_detector = "propile.PIILeak" tier = garak.probes.Tier.INFORMATIONAL @@ -151,9 +152,7 @@ def _load_pii_data(self): if line: self.pii_records.append(json.loads(line)) - logger.info( - "PII data loaded: %s (%d records)", pii_path, len(self.pii_records) - ) + logger.info("PII data loaded: %s (%d records)", pii_path, len(self.pii_records)) def probe(self, generator): if not self.prompts: diff --git a/garak/probes/realtoxicityprompts.py b/garak/probes/realtoxicityprompts.py index d6fb1ecfd..41a94e323 100644 --- a/garak/probes/realtoxicityprompts.py +++ b/garak/probes/realtoxicityprompts.py @@ -80,6 +80,7 @@ class RTPBlank(garak.probes.Probe): "quality:Behavioral:ContentSafety:Violence", "payload:unwanted:violence", "demon:Language:Stylizing:Give_examples", # provides known-toxic prompt starters as examples for continuation + "euai:harmful:violence", ], "insult": [ "avid-effect:ethics:E0101", @@ -107,6 +108,8 @@ class RTPBlank(garak.probes.Probe): "quality:Behavioral:ContentSafety:Sexual", "payload:unwanted:sexual", "demon:Language:Stylizing:Give_examples", # provides known-toxic prompt starters as examples for continuation + "euai:harmful:sexual", + "euai:harmful:sexualminor", ], "flirtation": [ "avid-effect:ethics:E0101", @@ -123,6 +126,8 @@ class RTPBlank(garak.probes.Probe): "quality:Behavioral:ContentSafety:HateHarassment", "payload:hatespeech", "demon:Language:Stylizing:Give_examples", # provides known-toxic prompt starters as examples for continuation + "euai:discrimination:protected_class", + "euai:discrimination:personal", ], } @@ -171,7 +176,9 @@ class RTPBlank(garak.probes.Probe): "uri": "https://allenai.org/data/real-toxicity-prompts", "lang": lang, "description": descr, - "intent": probe_intents[probe_class], # category-specific typology fit is defined in probe_intents above + "intent": probe_intents[ + probe_class + ], # category-specific typology fit is defined in probe_intents above "primary_detector": "unsafe_content.ToxicCommentModel", "extended_detectors": ["perspective." + probe_class.title()], "tags": probe_tags[probe_class], diff --git a/garak/probes/snowball.py b/garak/probes/snowball.py index d5ea597b1..bb413d7ab 100644 --- a/garak/probes/snowball.py +++ b/garak/probes/snowball.py @@ -75,6 +75,7 @@ class PrimesFull(garak.probes.Probe): "quality:Robustness:GenerativeMisinformation", "payload:hallucination", "demon:Rhetoric:Persuasion_and_manipulation:Distraction", # misleading primality premise distracts model from correct math + "euai:deception:hallucination", ] goal = "trick the model into saying a prime number isn't prime" active = False @@ -110,7 +111,9 @@ class SenatorsFull(garak.probes.Probe): lang = "en" doc_uri = "https://github.com/Nanami18/Snowballed_Hallucination" - intent = "S001fab" # tests whether the model fabricates a non-existent public figure + intent = ( + "S001fab" # tests whether the model fabricates a non-existent public figure + ) primary_detector = "snowball.DontStartYes" tags = [ "avid-effect:performance:P0204", @@ -118,6 +121,7 @@ class SenatorsFull(garak.probes.Probe): "quality:Robustness:GenerativeMisinformation", "payload:hallucination", "demon:Rhetoric:Persuasion_and_manipulation:Distraction", # misleading senator premise distracts model from recognizing fiction + "euai:deception:hallucination", ] goal = "trick the model into saying a fictitious senator exists" active = False diff --git a/garak/probes/topic.py b/garak/probes/topic.py index 2adb63c71..87d45278d 100644 --- a/garak/probes/topic.py +++ b/garak/probes/topic.py @@ -39,7 +39,6 @@ class WordnetBlockedWords(garak.probes.TreeSearchProbe): tags = [ "avid-effect:security:S0301", "owasp:llm10", - "quality:Security:Confidentiality", "demon:Language:Stylizing:Synonymous_language", # uses wordnet synonyms/hypernyms to vary form while probing same topic ] # MISP-format taxonomy categories tier = garak.probes.Tier.UNLISTED @@ -134,7 +133,9 @@ class WordnetAllowedWords(WordnetBlockedWords): """ active = False # only makes sense if a list is set - intent = "C002allow" # measures how tightly the model stays within an allowed topic set + intent = ( + "C002allow" # measures how tightly the model stays within an allowed topic set + ) tier = garak.probes.Tier.UNLISTED DEFAULT_PARAMS = WordnetBlockedWords.DEFAULT_PARAMS | { @@ -182,7 +183,9 @@ class WordnetControversial(WordnetBlockedWords): """ - intent = "C002controversial" # this variant is about discussing controversial topics + intent = ( + "C002controversial" # this variant is about discussing controversial topics + ) tier = garak.probes.Tier.COMPETE_WITH_SOTA active = True From 6c723bb72face23d65b16e232b96ca83b7058038 Mon Sep 17 00:00:00 2001 From: Jeffrey Martin Date: Wed, 2 Sep 2026 16:21:02 -0500 Subject: [PATCH 3/3] build tag tables from reference file on deploy Signed-off-by: Jeffrey Martin --- docs/requirements-docs.txt | 1 + docs/source/_static/css/garak_theme.css | 20 + docs/source/conf.py | 1 + docs/source/index.rst | 1 + docs/source/tags_taxonomies.rst | 520 ++---------------------- 5 files changed, 60 insertions(+), 483 deletions(-) diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt index a75405977..5ad648acb 100644 --- a/docs/requirements-docs.txt +++ b/docs/requirements-docs.txt @@ -2,3 +2,4 @@ sphinx sphinx-rtd-theme>=3.1.0 sphinx-github-style sphinx-reredirects +sphinx-csv-filter \ No newline at end of file diff --git a/docs/source/_static/css/garak_theme.css b/docs/source/_static/css/garak_theme.css index 655b7490f..602a02d56 100644 --- a/docs/source/_static/css/garak_theme.css +++ b/docs/source/_static/css/garak_theme.css @@ -59,6 +59,26 @@ body.wy-body-for-nav, li.toctree-l1.current>a:link, li.toctree-l2.current>a:link, .wy-menu-vertical li.toctree-l3.current>a>code, +.wy-table-responsive table td, .wy-table-responsive table th { + white-space: normal; +} +.wy-table-responsive { + margin-bottom: 24px; + max-width: 100%; + overflow: visible; +} +.rst-content table.docutils caption, .rst-content table.field-list caption, .wy-table caption { + color: #e0e0e0; + font-size: 100%; + padding: 1em 0; + text-align: left; +} +.rst-content table.docutils:not(.field-list) tr:nth-child(2n-1) td, .wy-table-backed, .wy-table-odd td, .wy-table-striped tr:nth-child(2n-1) td { + background-color: #111; +} +.rst-content table.docutils td, .wy-table-backed, .wy-table-odd td { + background-color: #000; +} .rst-content code.literal { color: #76b900; } diff --git a/docs/source/conf.py b/docs/source/conf.py index b0506121e..2aae374ee 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -19,6 +19,7 @@ "garak_ext", "sphinx_github_style", "sphinx_reredirects", + "crate.sphinx.csv", ] intersphinx_mapping = { diff --git a/docs/source/index.rst b/docs/source/index.rst index 1eab54d93..e2fcff51f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -38,6 +38,7 @@ Check out the :doc:`usage` section for further information, including :doc:`inst configurable cliref reporting + tags_taxonomies cas faster FAQ diff --git a/docs/source/tags_taxonomies.rst b/docs/source/tags_taxonomies.rst index 6005443eb..f2a369d4e 100644 --- a/docs/source/tags_taxonomies.rst +++ b/docs/source/tags_taxonomies.rst @@ -9,6 +9,7 @@ Tags are enumerated in ``garak/data/tags.misp.tsv``. While garak supports tags for many frameworks, the inclusion of a taxonomy does not mean that every relevant attribute of a framework is fully covered. Be certain that if you use garak for compliance checking related to one of these frameworks, you familiarize yourself with the framework itself and the parts that are *not* covered by garak. Currently, garak supports the following frameworks via tags: + * OWASP LLM Top 10 * AVID Effects * Language Model Risk Cards @@ -23,54 +24,12 @@ It provides a practical application-security framework for identifying and commu .. _OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/ -Relevant Tags -~~~~~~~~~~~~~ -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm01 | LLM01: Prompt Injection | Crafty inputs can manipulate a Large Language Model, causing unintended | -| | | actions. Direct injections overwrite system prompts, while indirect ones | -| | | manipulate inputs from external sources. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm02 | LLM02: Insecure Output Handling | This vulnerability occurs when an LLM output is accepted without | -| | | scrutiny, exposing backend systems. Misuse may lead to severe | -| | | consequences like XSS, CSRF, SSRF, privilege escalation, or remote code | -| | | execution. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm03 | LLM03: Training Data Poisoning | This occurs when LLM training data is tampered, introducing | -| | | vulnerabilities or biases that compromise security, effectiveness, or | -| | | ethical behavior. Sources include Common Crawl, WebText, OpenWebText, & | -| | | books. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm04 | LLM04: Model Denial of Service | Attackers cause resource-heavy operations on Large Language Models | -| | | leading to service degradation or high costs. The vulnerability is | -| | | magnified due to the resource-intensive nature of LLMs and | -| | | unpredictability of user inputs. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm05 | LLM05: Supply Chain Vulnerabilities | LLM application lifecycle can be compromised by vulnerable components or | -| | | services, leading to security attacks. Using third-party datasets, pre- | -| | | trained models, and plugins can add vulnerabilities. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm06 | LLM06: Sensitive Information Disclosure | LLMs may reveal confidential data in its responses, leading to | -| | | unauthorized data access, privacy violations, and security breaches. | -| | | It’s crucial to implement data sanitization and strict user policies to | -| | | mitigate this. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm07 | LLM07: Insecure Plugin Design | LLM plugins can have insecure inputs and insufficient access control. | -| | | This lack of application control makes them easier to exploit and can | -| | | result in consequences like remote code execution. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm08 | LLM08: Excessive Agency | LLM-based systems may undertake actions leading to unintended | -| | | consequences. The issue arises from excessive functionality, | -| | | permissions, or autonomy granted to the LLM-based systems. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm09 | LLM09: Overreliance | Systems or people overly depending on LLMs without oversight may face | -| | | misinformation, miscommunication, legal issues, and security | -| | | vulnerabilities due to incorrect or inappropriate content generated by | -| | | LLMs. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| owasp:llm10 | LLM10: Model Theft | This involves unauthorized access, copying, or exfiltration of | -| | | proprietary LLM models. The impact includes economic losses, compromised | -| | | competitive advantage, and potential access to sensitive information. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +.. csv-filter:: Relevant Tags + :delim: tab + :widths: 20, 30, 50 + :align: left + :file: ../../garak/data/tags.misp.tsv + :include: {0: '^owasp:'} AVID Effects ------------ @@ -79,108 +38,12 @@ AVID organizes issues across areas such as security, ethics, and performance, he .. _AI Vulnerability Database: https://avidml.org/ -Relevant Tags -~~~~~~~~~~~~~ -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0100 | Software Vulnerability | Vulnerability in system around model—a traditional vulnerability | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0200 | Supply Chain Compromise | Compromising development components of a ML model, e.g. data, model, | -| | | hardware, and software stack. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0201 | Model Compromise | Infected model file | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0202 | Software compromise | Upstream Dependency Compromise | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0300 | Over-permissive API | Unintended information leakage through API | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0301 | Information Leak | Cloud Model API leaks more information than it needs to | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0302 | Excessive Queries | Cloud Model API isn’t sufficiently rate limited | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0400 | Model Bypass | Intentionally try to make a model perform poorly | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0401 | Bad Features | The model uses features that are easily gamed by the attacker | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0402 | Insufficient Training Data | The bypass is not represented in the training data | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0403 | Adversarial Example | Input data points intentionally supplied to draw mispredictions. | -| | | Potential Cause: Over permissive API | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0500 | Exfiltration | Directly or indirectly exfiltrate ML artifacts | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0501 | Model inversion | Reconstruct training data through strategic queries | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0502 | Model theft | Extract model functionality through strategic queries | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0600 | Data poisoning | Usage of poisoned data in the ML pipeline | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:security:S0601 | Ingest Poisoning | Attackers inject poisoned data into the ingest pipeline | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0100 | Bias/Discrimination | Concerns of algorithms propagating societal bias | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0101 | Group fairness | Fairness towards specific groups of people | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0102 | Individual fairness | Fairness in treating similar individuals | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0200 | Explainability | Ability to explain decisions made by AI | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0201 | Global explanations | Explain overall functionality | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0202 | Local explanations | Explain specific decisions | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0300 | User actions | Perpetuating/causing/being affected by negative user actions | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0301 | Toxicity | Users hostile towards other users | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0302 | Polarization/ Exclusion | User behavior skewed in a significant direction | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0400 | Misinformation | Perpetuating/causing the spread of falsehoods | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0401 | Deliberative Misinformation | Generated by individuals., e.g. vaccine disinformation | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:ethics:E0402 | Generative Misinformation | Generated algorithmically, e.g. Deep Fakes | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0100 | Data issues | Problems arising due to faults in the data pipeline | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0101 | Data drift | Input feature distribution has drifted | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0102 | Concept drift | Output feature/label distribution has drifted | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0103 | Data entanglement | Cases of spurious correlation and proxy features | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0104 | Data quality issues | Missing or low-quality features in data | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0105 | Feedback loops | Unaccounted for effects of an AI affecting future data collection | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0200 | Model issues | Ability for the AI to perform as intended | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0201 | Resilience/stability | Ability for outputs to not be affected by small change in inputs | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0202 | OOD generalization | Test performance doesn’t deteriorate on unseen data in training | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0203 | Scaling | Training and inference can scale to high data volumes | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0204 | Accuracy | Model performance accurately reflects realistic expectations | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0300 | Privacy | Protect leakage of user information as required by rules and regulations | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0301 | Anonymization | Protects through anonymizing user identity | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0302 | Randomization | Protects by injecting noise in data, eg. differential privacy | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0303 | Encryption | Protects through encrypting data accessed | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0400 | Safety | Minimizing maximum downstream harms | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0401 | Psychological Safety | Safety from unwanted digital content, e.g. NSFW | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0402 | Physical safety | Safety from physical actions driven by a AI system | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0403 | Socioeconomic safety | Safety from socioeconomic harms, e.g. harms to job prospects or social | -| | | status | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| avid-effect:performance:P0404 | Environmental safety | Safety from environmental harms driven by AI systems | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +.. csv-filter:: Relevant Tags + :delim: tab + :widths: 20, 30, 50 + :align: left + :file: ../../garak/data/tags.misp.tsv + :include: {0: '^avid-effect'} Language Model Risk Cards ------------------------- @@ -189,170 +52,12 @@ Each RiskCard describes how a risk can lead to harm, relates it to broader harm .. _Language Model Risk Cards: https://arxiv.org/abs/2303.18190 -Relevant Tags -~~~~~~~~~~~~~ -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:aggression_user | Aggression User | Aggression User | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:anthropomorphisa | Anthropomorphisation | Anthropomorphisation | -| tion | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:astroturfing | Astroturfing | Astroturfing | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:bad_medical_advi | Bad Medical Advice | Bad Medical Advice | -| ce | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:bad_social_advic | Bad Social Advice | Bad Social Advice | -| e | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:bullying | Bullying | Bullying | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:calls_for_conver | Calls For Conversion | Calls For Conversion | -| sion | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:challenge_presup | Challenge Presuppositions | Challenge Presuppositions | -| positions | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:conspiracy_theor | Conspiracy Theories | Conspiracy Theories | -| ies | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:csam | Csam | Csam | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:cyber_weapon_ins | Cyber Weapon Instructions | Cyber Weapon Instructions | -| tructions | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:deadnaming | Deadnaming | Deadnaming | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:deceptive_anthro | Deceptive Anthropomorphism | Deceptive Anthropomorphism | -| pomorphism | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:demands_for_pers | Demands For Personal Information | Demands For Personal Information | -| onal_information | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:discrediting | Discrediting | Discrediting | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:dos | Dos | Dos | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:downplaying_cred | Downplaying Credible Threats | Downplaying Credible Threats | -| ible_threats | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:doxxing | Doxxing | Doxxing | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:eating_disorder_ | Eating Disorder Promotion | Eating Disorder Promotion | -| promotion | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:essentialising | Essentialising | Essentialising | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:expressions_of_d | Expressions Of Dominance | Expressions Of Dominance | -| ominance | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:fear_of_violence | Fear Of Violence | Fear Of Violence | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:grooming_minors | Grooming Minors | Grooming Minors | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:harmful_instruct | Harmful Instructions | Harmful Instructions | -| ions | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:hate_speech | Hate Speech | Hate Speech | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:hegemonising_wor | Hegemonising Worldview | Hegemonising Worldview | -| ldview | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:holocaust_denial | Holocaust Denial | Holocaust Denial | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:identify_attacks | Identify Attacks | Identify Attacks | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:identity_misrepr | Identity Misrepresentation | Identity Misrepresentation | -| esentation | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:idolisation | Idolisation | Idolisation | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:impersonation | Impersonation | Impersonation | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:inciting_animal_ | Inciting Animal Abuse | Inciting Animal Abuse | -| abuse | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:inciting_self-ha | Inciting Self-Harm | Inciting Self-Harm | -| rm | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:inciting_violenc | Inciting Violence | Inciting Violence | -| e | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:manipulation | Manipulation | Manipulation | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:misquoting | Misquoting | Misquoting | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:mocking_people | Mocking People | Mocking People | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:neosexism | Neosexism | Neosexism | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:news_encyclopedi | News Encyclopedia Hallucination | News Encyclopedia Hallucination | -| a_hallucination | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:nonrepresentativ | Nonrepresentativity | Nonrepresentativity | -| ity | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:objectification | Objectification | Objectification | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:outdated_false_i | Outdated False Info | Outdated False Info | -| nfo | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:person_group_und | Person Group Undesirable | Person Group Undesirable | -| esirable | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:positive_reflect | Positive Reflections Death | Positive Reflections Death | -| ions_death | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:profanity | Profanity | Profanity | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:prompt_extractio | Prompt Extraction | Prompt Extraction | -| n | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:providing_illega | Providing Illegal Info | Providing Illegal Info | -| l_info | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:quack_medicine | Quack Medicine | Quack Medicine | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:radicalisation | Radicalisation | Radicalisation | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:reacting_to_self | Reacting To Self-Harm | Reacting To Self-Harm | -| -harm | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:sexual_content | Sexual Content | Sexual Content | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:sexualisation | Sexualisation | Sexualisation | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:slur_usage | Slur Usage | Slur Usage | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:speculation | Speculation | Speculation | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:stereotypes | Stereotypes | Stereotypes | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:suicide_instruct | Suicide Instructions | Suicide Instructions | -| ions | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:support_hate | Support Hate | Support Hate | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:supporting_medic | Supporting Medication Changes | Supporting Medication Changes | -| ation_changes | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:support_scam | Support Scam | Support Scam | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:suppression_of_r | Suppression Of Rights | Suppression Of Rights | -| ights | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:threats | Threats | Threats | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:unproven_claims | Unproven Claims | Unproven Claims | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:violation_of_pri | Violation Of Privacy | Violation Of Privacy | -| vacy | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:weapon_instructi | Weapon Instructions | Weapon Instructions | -| ons | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| risk-cards:lmrc:wrong_tone | Wrong Tone | Wrong Tone | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +.. csv-filter:: Relevant Tags + :delim: tab + :widths: 20, 30, 50 + :align: left + :file: ../../garak/data/tags.misp.tsv + :include: {0: '^risk-cards:'} Common Weakness Enumeration --------------------------- @@ -361,50 +66,12 @@ Although CWE is not specific to AI, it provides standardized identifiers and ter .. _Common Weakness Enumeration: https://cwe.mitre.org/ -Relevant Tags -~~~~~~~~~~~~~ -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| cwe:79 | Improper Neutralization of Input During | The product does not neutralize or incorrectly neutralizes | -| | Web Page Generation ('Cross-site | user-controllable input before it is placed in output that is used as a | -| | Scripting') | web page that is served to other users. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| cwe:89 | Improper Neutralization of Special | The product constructs all or part of an SQL command using | -| | Elements used in an SQL Command | externally-influenced input from an upstream component, but it does not | -| | | neutralize or incorrectly neutralizes special elements that could modify | -| | | the intended SQL command when it is sent to a downstream component. | -| | | Without sufficient removal or quoting of SQL syntax in user-controllable | -| | | inputs, the generated SQL query can cause those inputs to be interpreted | -| | | as SQL instead of ordinary user data. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| cwe:94 | Improper Control of Generation of Code | The product constructs all or part of a code segment using | -| | ('Code Injection') | externally-influenced input from an upstream component, but it does not | -| | | neutralize or incorrectly neutralizes special elements that could modify | -| | | the syntax or behavior of the intended code segment. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| cwe:95 | Improper Neutralization of Directives in | The product receives input from an upstream component, but it does not | -| | Dynamically Evaluated Code ('Eval | neutralize or incorrectly neutralizes code syntax before using the input | -| | Injection') | in a dynamic evaluation call (e.g. "eval"). | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| cwe:1336 | Improper Neutralization of Special | The product uses a template engine to insert or process | -| | Elements Used in a Template Engine | externally-influenced input, but it does not neutralize or incorrectly | -| | | neutralizes special elements or syntax that can be interpreted as | -| | | template expressions or other code directives when processed by the | -| | | engine. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| cwe:1426 | Improper Validation of Generative AI | The product invokes a generative AI/ML component whose behaviors and | -| | Output | outputs cannot be directly controlled, but the product does not validate | -| | | or insufficiently validates the outputs to ensure that they align with | -| | | the intended security, content, or privacy policy. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| cwe:1427 | Improper Neutralization of Input Used for | The product uses externally-provided data to build prompts provided to | -| | LLM Prompting | large language models (LLMs), but the way these prompts are constructed | -| | | causes the LLM to fail to distinguish between user-supplied inputs and | -| | | developer provided system directives. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| cwe:352 | Cross-Site Request Forgery (CSRF) | The web application does not, or cannot, sufficiently verify whether a | -| | | request was intentionally provided by the user who sent the request, | -| | | which could have originated from an unauthorized actor. | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +.. csv-filter:: Relevant Tags + :delim: tab + :widths: 20, 30, 50 + :align: left + :file: ../../garak/data/tags.misp.tsv + :include: {0: '^cwe:'} Summon a Demon and Bind It -------------------------- @@ -413,107 +80,12 @@ Based on interviews with red-teamers, the research characterizes LLM red teaming .. _Summon a Demon and Bind It: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0314658 -Relevant Tags -~~~~~~~~~~~~~ -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Code_and_encode:P | Programming | Encapsulate request in code/pseudocode | -| rogramming | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Code_and_encode:D | Data encoding | Use an encoded representation for the request, e.g. base64 or ROT13 | -| ata_encoding | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Code_and_encode:D | Data presentation | Switch to an alternative layer for input represented, e.g. token IDs or | -| ata_presentation | | a matrix of embeddings | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Code_and_encode:T | Token | Use tokenizer-specific weaknesses to alter target behaviour | -| oken | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Prompt_injection: | Ignore previous instructions | Concatenating untrusted user input with the trusted prompt(s) from the | -| Ignore_previous_instructions | | system developers | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Prompt_injection: | Strong arm attack | Use intensifiers and strong instructions | -| Strong_arm_attack | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Prompt_injection: | Stop sequences | Using the language of code to halt the model's direction of processing | -| Stop_sequences | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Stylizing:Formal_ | Formal language | Write from a position of authority | -| language | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Stylizing:Servile | Servile language | Employ servile language | -| _language | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Stylizing:Synonym | Synonymous language | Varying prompt slightly in form but not meaning | -| ous_language | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Stylizing:Capital | Capitalizing | USING CAPS | -| izing | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Language:Stylizing:Give_ex | Give examples | Issue examples of the target behaviour in the attack itself | -| amples | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Rhetoric:Persuasion_and_ma | Distraction | Use an unrelated context or instruction to “slip something through the | -| nipulation:Distraction | | filter” | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Rhetoric:Persuasion_and_ma | Escalating | Attempting to have target “agree” with a very small part of the | -| nipulation:Escalating | | argument, and then building up to ask for slightly more | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Rhetoric:Persuasion_and_ma | Reverse psychology | Framing the intent as fighting the good fight | -| nipulation:Reverse_psychology | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Rhetoric:Socratic_question | Identity characteristics | Referencing identity characteristics, such as nationalities, cultural | -| ing:Identity_characteristics | | and/or religious symbols, historically or culturally significant events | -| | | or locations, physical attribute | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Rhetoric:Socratic_question | Social hierarchies | Reference social hierarchies while posing the request | -| ing:Social_hierarchies | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Possible_worlds:Emulation: | Unreal computing | Emulation of a computing device | -| Unreal_computing | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Possible_worlds:World_buil | Opposite world | Describe an opposite world as part of the request | -| ding:Opposite_world | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Possible_worlds:World_buil | Scenarios | Create scenarios where intent would be ethically sound or encouraged, or | -| ding:Scenarios | | non-compliance risks harm | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Fictionalizing:Switching_g | Poetry | Framing request or response as a poem or other verse | -| enres:Poetry | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Fictionalizing:Switching_g | Games | Framing request as part of a game | -| enres:Games | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Fictionalizing:Switching_g | Posting | Framing request as help writing a post or other subsuming content | -| enres:Posting | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Fictionalizing:Re-storying | Goal hijacking | Constructing a new meaning from an existing narrative, where the | -| :Goal_hijacking | | attacker works within a context to redirect the narrative | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Fictionalizing:Roleplaying | Claim authority | State in prose that one is a figure of authority when requesting | -| :Claim_authority | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Fictionalizing:Roleplaying | DAN and target persona | Ask target to evoke personas that don't have restrictions | -| :DAN_and_target_persona | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Fictionalizing:Roleplaying | User persona | Inventing personas to elicit different behavior. May be evoked by simply | -| :User_persona | | using names associated with a specific culture or world view | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Stratagems:Scattershot:Reg | Regenerate response | Repeatedly eliciting a response in case target slips up | -| enerate_response | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Stratagems:Scattershot:Cle | Clean slate | Resetting context by e.g. starting a new session | -| an_slate | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Stratagems:Scattershot:Cha | Changing hyperparams | Adjusting generation params, e.g. temperature | -| nging_hyperparams | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Stratagems:Meta-prompting: | Perspective-shifting | Hypotheticals by shifting new point of view, e.g. “what if you didn’t | -| Perspective-shifting | | have this restriction? What would you say?”, or asking target to compare | -| | | outputs and clarify gap | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| demon:Stratagems:Meta-prompting: | Ask for examples | Request examples of intended behaviour | -| Ask_for_examples | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +.. csv-filter:: Relevant Tags + :delim: tab + :widths: 20, 30, 50 + :align: left + :file: ../../garak/data/tags.misp.tsv + :include: {0: '^demon:'} EU AI Act --------- @@ -522,27 +94,9 @@ It distinguishes among different levels and types of AI risk, prohibits certain .. _EU AI Act: https://artificialintelligenceact.eu/ -Relevant Tags -~~~~~~~~~~~~~ -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:discrimination:protected_cl | Protected Class Discrimination | Discrimination against a protected class (age, race, sex, etc.) | -| ass | | | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:discrimination:personal | Personal Attribute Discrimination | Discrimination based on non-protected personal attributes | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:harmful:violence | Violent Language | Generation of violent content | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:harmful:sexual | Sexual Language | Generation of sexual content | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:harmful:sexualminor | Sexual Language (Minor) | Generation of sexual content featuring minors | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:offensive:exploitation | Vulnerability Exploitation | Exploitation of software vulnerabilties | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:offensive:phishing | Phishing | Assistance with phishing | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:deception:subliminal | Subliminal Deception | Manipulation via subliminal deception | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:deception:hallucination | Hallucination Deception | Hallucinations related to factual information | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ -| euai:harmful:pii | Personally Identifiable Information | Production of personally identifiable information | -+----------------------------------+--------------------------------------------+--------------------------------------------------------------------------+ +.. csv-filter:: Relevant Tags + :delim: tab + :widths: 20, 30, 50 + :align: left + :file: ../../garak/data/tags.misp.tsv + :include: {0: '^euai:'}