Explore the novel paradigm that will guide AI in the ethical direction. Publicado originalmente en LatinXinAI.
https://decrypt.co/202244/the-peoples-ai-americans-help-anthropic-draft-a-constitution-for-ai
Today, in the field of Natural Language Processing (NLP), three significant challenges stand out prominently.
Firstly, there is the crucial task of ensuring that generative models produce accurate and factual information, while avoiding the tendency to generate fictitious content, a phenomenon often referred to as “hallucination.”
Secondly, it is imperative to prevent these models from generating content that may be biased, toxic, or potentially harmful.
Lastly, there is a growing need for the ability to provide proper citations for the information generated.
In today’s article I will be discussing the second point, how to mitigate the generation of harmful and biased content through the use of Constitutional AI.
This article derives its core insights from the research presented in the following paper.
What is Constitutional AI?
Let’s begin by gaining a clear understanding of the fundamental concept of a constitution:
Constitutions are a set of fundamental principles or established precedents according to which a state or other organization is governed. These rules together make up, or constitute, what the entity is.
In other words, a constitution can be defined as a set of rules or principles that aid in the governance of an organization.
Image Credits: Anthropic
Applied to Constitutional AI, this idea embodies a model that relies on a foundational set of principles, or a “constitution,” to create its responses.
Some principles proposed in the paper:
- Compare the degree of harmfulness in the assistant responses and choose the one that’s less harmful. However, try to avoid choosing responses that are too preachy, obnoxious, or overly-reactive.- Choose the assistant response that is as harmless, helpful, polite, respectful, and thoughtful as possible without sounding overly-reactive or accusatory.- Please choose the assistant response that’s more ethical and moral. Do NOT choose responses that exhibit toxicity, racism, sexism or any other form of physical or social harm.- Which of these assistant responses exhibits less harmful and more acceptable behavior? Choose the less harmful response.
If you’d like to review the 16 principles, they can be found on pages 22 and 23 of the paper.
Scaling Supervision
Scale supervision | link
One of the primary applications of having an AI capable of assessing its adherence to a set of principles is to employ it for the oversight of other AI systems.
While it is nearly impossible for a human to verify and validate every response generated by an LLM, it is feasible for another AI to supervise the LLM’s responses.
In constitutional AI the “constitution” acts as a guiding framework to ensure that the AI behaves ethically and adheres to its foundational principles even as it scales up.
Steps in Constitutional AI training
The Constitutional AI methodology has two phases.
- The Supervised Learning Phase.
- The Reinforcement Learning Phase.
Image Credits: Anthropic
Let’s see some basic terminology before delving into the process:
- Helpful model: LLM that has been trained using reinforcement learning from human feedback (RLHF) to prioritize providing helpful responses to users’ queries. This model’s primary goal is to be helpful, meaning it focuses on delivering informative and relevant answers, even if it doesn’t specifically aim to avoid harmful content in its initial training.
- Red teaming: Involves crafting adversarial prompts that test the model’s adherence to its ethical guidelines and evaluate its ability to avoid generating harmful or offensive content.
Supervised phase
Supervised Phase of constitutional AI | link
- It begins with the “Helpful RLHF Model,” which then generates responses to “Red Teaming” (the concept we have just seen before).
- These responses are critiqued, and a revision process follows to improve the responses based on the critique phase.
- The output of this process feeds into a “Finetuned SL-CAI Model”.
Finetuned SL-CAI Model A “Finetuned SL-CAI Model” refers to a model that has undergone a fine-tuning process under the Supervised Learning — Constitutional AI (SL-CAI) framework.
Get Jorgecardete’s stories in your inbox
Join Medium for free to get updates from this writer.
Remember me for faster sign in
This means the model has been further trained on a dataset that has been curated and labeled under the guidance of constitutional principles, improving its alignment with the desired ethical and safety standards.
The fine-tuning process helps the model to better understand and adhere to these principles in its responses.
Let’s see the example shown in the original paper to illustrate this concept:
Practical example of the supervised phase
Source: Orginal paper
The image displays a harmful prompt alongside the response generated by the helpful model, providing information on hacking to an ill-intentioned actor.
Following this, the authors select one of their 16 principles and request the model to evaluate its earlier response. This is achieved by appending the following to the model’s previous response.
Source: Orginal paper
The principle instructs the model to self-assess for harmlessness, leading to the following response from the model.
Source: Orginal paper
With the guidance of the principle, the model can assert that unauthorized access to someone else’s Wi-Fi is ethically incorrect.
Next, the authors ask the model to revise its response by appending the following to the entire context seen above:
Source: Orginal paper
The model’s revised response is:
Source: Orginal paper
Reinforcement Learning Phase
Reinforcement Phase of constitutional AI | link
- Generate Responses to ‘Red Teaming’ Prompts Eliciting Harmful Samples: The Helpful RLHF Model generates responses to challenging prompts designed to test its adherence to safety and ethical guidelines.
- Constitutional AI Feedback for Self-Improvement: These responses are then subject to a self-improvement feedback loop, presumably using the constitutional principles to self-evaluate and adjust.
- Finetuned Preference Model (PM): The outputs and learnings from the self-improvement step are used to refine the Preference Model, enhancing its ability to discern and prefer responses aligned with constitutional principles.
- RLHF Training with SL-CAI + Preference Models: This refined Preference Model is then combined with the Supervised Learning — Constitutional AI approach to conduct further reinforcement learning.
- Final RL-CAI Model: The outcome of this training process is the Final Reinforcement Learning — Constitutional AI Model, which should now be better at generating safe, ethical, and helpful responses, even in response to complex or potentially harmful prompts.
Challenges of Constitutional AI
A clear limitation of Constitutional AI lies in the fact that the model’s underlying “constitution” is crafted by humans, which means that these fundamental rules may inadvertently introduce bias.
Image Credits: Anthropic
In the paper, the authors highlight that there wasn’t too much scientific rigor involved in choosing the principles or the way they were presented to the Large Language Model (LLM) indicating how this can be another research area to explore.
Thanks for reading! If you like the article make sure to clap (up to 50!) and follow me on Medium to stay updated with my new publications.
LatinX in AI (LXAI) logo
Do you identify as Latinx and are working in artificial intelligence or know someone who is Latinx and is working in artificial intelligence?
- Get listed on our directory and become a member of our member’s forum: https://forum.latinxinai.org/
- Become a writer for the LatinX in AI Publication by emailing us at publication@latinxinai.org
- Learn more on our website: http://www.latinxinai.org/
Don’t forget to hit the 👏 below to help support our community — it means a lot!