Making Dans-PersonalityEngine V1.3.0
Not even a week after the release of V1.2.0 and I was approached by a sponsor interested in putting up the resources and funding for an update. To me V1.2.0 was already a great model and no small amount of work went into it, so I really had to dig into the data and see what was possible.
Taking Stock of the Roleplay Data Mix
After careful analysis of V1.2.0’s performance and user feedback, several key areas emerged that needed improvement. The roleplay data mix, while cleaner than previous versions, still had some lingering issues with refusals and broken responses slipping through the filters. The N-gram filter in particular wasn’t performing as intended, and the filter list for “slops” was not comprehensive enough. Users were also reporting that the model was not staying in character as much as they would like, often overstepping their bounds and narrating actions for the users character.
To address these issues, a classifier model was trained to specifically clean the RP data mix. The N-gram algorithm was refined to properly handle thinking block formats, and a comprehensive list of common problematic phrases was implemented as an additional filtering layer. A new targeted dataset was created to help guide the model toward better character consistency during roleplay interactions.
The Technical Data Mix
While in much better shape than the RP data mix, the technical data mix was not without its issues. V1.2.0 was signicantly lacking in the way of reasoning and problem solving data. The coding and general assistant data was somewhat dated and not as potent as I would like. A diverse set of reasoning data was assembled custom made for this version and a huge amount of new assistant oriented data was added covering a wide range of domains. Performance can be expected to increase in philosophy, law, finance, medicine, and numerous STEM domains. In addition to this a new set of multilingual data was added to help prevent the model from forgetting languages it had been trained on.
Other Notable Changes to the Data Mix
Almost all models when let write by themselves for thousands of tokens will devolve in some way or another. Now the traditional school of thought is to tackle this using RL however through clever use of masking I was able to help bake these tendencies into the model during the SFT process. While not a “magic bullet” it should help to some degree.
Dataset in Total
The new data mix is 1.7B tokens depending on the specific tokenizer used. The mix comes out to be around 35% instruct, assistant and problem solving tasks, 65% writing, roleplay, and other creative tasks.
Training Parameters
There were a number of departures on the hyperparameters used in this training run as compared to the previous version. Previously I used adamw for the optimizer with a linear warmup and cosine decay schedule. I switched to ademamix for the optimizer with a linear warmup and REX learning rate scheduler. In addition to this I switched from using a gradient clip of 3.0-15.0 with a weight decay of 0.1 to a gradient clip of 0.001 with no weight decay. I will make a write up on the training parameters and the reasoning behind them in a future post.
Future Work
Now that the data is in a good place I will be taking a break from the SFT data mix and working on RL methods. There may be further trains on more model sizes using the same data mix before then as its not that much work to do. I have a lot of hopes for the RL and its ability to push the model even further.