Not even a week after the release of V1.2.0 and I was approached by a sponsor interested in putting up the resources and funding for an update. To me V1.2.0 was already a great model and no small amount of work went into it, so…

Recently I’ve been experimenting with optimizers other than AdamW and its kin using Mistral-7B-V0.3 as a basis. Of them one that showed a lot of promise was CAME. It offers memory overhead similar to Adafactor but with convergence like AdamW. In my usage, I’ve found…

The PocketDoc Labs flagship model, Dans-PersonalityEngine-V1.2.0-24b, is good to be sure. However, we are very far from the maximum that can be achieved on a given base model. I want V1.3.0 to feel like a noticeable step towards that potential. The following significant problems have…

The Problem Historically open weights models have struggled with recovering from repetitive spirals both in the immediate prior few tokens, i.e. repeating the same token endlessly, and more structurally in phrasing or general writing flow. Easy, we train the model on sections of low quality…

One of the many gems from the Llama 2 paper was a technique they referred to as “Ghost Attention”. More generally this was the idea of synthetically creating chat histories paired with system messages depicting only the final message obeying or acknowledging the system message.