The PocketDoc Labs flagship model, Dans-PersonalityEngine-V1.2.0-24b, is good to be sure. However, we are very far from the maximum that can be achieved on a given base model. I want V1.3.0 to feel like a noticeable step towards that potential. The following significant problems have…

The Problem Historically open weights models have struggled with recovering from repetitive spirals both in the immediate prior few tokens, i.e. repeating the same token endlessly, and more structurally in phrasing or general writing flow. Easy, we train the model on sections of low quality…

One of the many gems from the Llama 2 paper was a technique they referred to as “Ghost Attention”. More generally this was the idea of synthetically creating chat histories paired with system messages depicting only the final message obeying or acknowledging the system message.