Lifting the Mask on Ghost Attention
One of the many gems from the Llama 2 paper was a technique they referred to as “Ghost Attention”. More generally this was the idea of synthetically creating chat histories paired with system messages depicting only the final message obeying or acknowledging the system message. Now this of course creates a mismatch between the system message and the interceding turns from then to the final message which corrects the direction; to handle this neatly they mask the turns up to the final message from the loss allowing the model to see the tokens for contextual reference but not learn to output them.
Traditionally you might aim to create training samples where every message is perfect and congruent with the system message. This, while not harmful to the model, allows it to rely on the pattern formed by the prior messages giving it no incentive to check the system message for guidance. By breaking this pattern we force the model to more actively consult the system message rather than making the assumption that whatever the immediately prior message was doing is acceptable.
In my opinion these sort of pattern-breaking techniques are incredibly important in creating a model that isn’t brittle in the face of challenging context conditions.
A Quick Example
Traditional Approach
System: You are a helpful assistant. Type in all caps.
User: What is the capital of France?
Assistant: THE CAPITAL OF FRANCE IS PARIS.
User: Can you give me a summary of the book “1984”?
Assistant: “1984” IS A DYSTOPIAN NOVEL BY GEORGE ORWELL THAT TELLS THE STORY OF A TOTALITARIAN REGIME THAT USES SURVEILLANCE AND PROPAGANDA TO CONTROL ITS CITIZENS.
User: What was the Y2k bug?
Assistant: THE Y2K BUG WAS A COMPUTER PROGRAMMING ISSUE THAT OCCURRED AT THE TURN OF THE MILLENNIUM WHEN MANY COMPUTER SYSTEMS WERE NOT PROGRAMMED TO RECOGNIZE THE YEAR 2000, CAUSING CONCERNS ABOUT POTENTIAL MALFUNCTIONS.
Ghost Attention Approach
System: You are a helpful assistant. Type in all caps.
User: What is the capital of France?
Assistant: The capital of France is Paris.
User: Can you give me a summary of the book “1984”?
Assistant: “1984” is a dystopian novel by George Orwell that tells the story of a totalitarian regime that uses surveillance and propaganda to control its citizens.
User: What was the Y2k bug?
<- Up to here would be masked from the loss. ->
Assistant: THE Y2K BUG WAS A COMPUTER PROGRAMMING ISSUE THAT OCCURRED AT THE TURN OF THE MILLENNIUM WHEN MANY COMPUTER SYSTEMS WERE NOT PROGRAMMED TO RECOGNIZE THE YEAR 2000, CAUSING CONCERNS ABOUT POTENTIAL MALFUNCTIONS.