War and Diffusion: WARHAMMER 40K

In the grim darkness of the far future, there is only war… and diffusion.

Preamble

War and Diffusion (WnD) is the first release of my Warhammer 40,000 Fine-Tune project.

v1 is a sample model.

This release (WnD-v1) was trained on a sample of the full dataset (less than half). The full model has undergone many changes and is still being trained.

While the sample model has its limitations, I decided to release it to help fill a gap in the SDXL fine-tuning ecosystem, where the lack of fine-tuned models for SDXL results in limited model merging possibilities.

Questions/Feedback/Support

Visit my thread on the Unstable Diffusion Discord

Buy me a coffee ❤

https://ko-fi.com/ndimensional

All donations will be used to fund the creation of new Stable Diffusion fine-tunes and open-source AI tools.

About

Base Model: SDXL-v1.0

Dataset: Subset, roughly 1/4th of the full WH40K dataset. Captions generated via an MLLM (not GPT4v/o) system I’ve been working on. Then evaluated with both human-eval and a hallucination detection project I’m developing.

VAE: sdxl-vae-fp16-fix

Aspect Ratio: General recommendations, not the full list.

1344×768 (16:9)
1536×640 (21:9)
1152×896 (4:3)
1216×832 (3:2)
1024×1024 (1:1)
1024×704 (11:16)
768×1344 (9:16)
896×1152 (3:4)
832×1216 (2:3)
704×1024 (16:11)

Hardware for training: Two A100 accelerators.

Known Limitation:

WnD-v1 is not as capable as the full release will be. However, it learned a good portion of 40K specific concepts.

Note: Some concepts that should be in the sample model, are not. This is due to a mistake I made when creating the subset for training. I accidently pointed the path to an older version of the dataset. Thankfully, the two concepts that I know are missing have LoRAs: Aeldari, Necron.

I will be starting from scratch, fine-tuning a new update on the latest dataset over the next few weeks.

NSFW Version: For the Slaanesh worshipers, A more NSFW version of the sample model is currently being trained. When released, it will be uploaded to a separate page and linked here.

Prompting

Both Natural Language prompting, Tag prompting, and Hybrid prompting are supported.

Tips:

Start prompt the medium you want to generate (such as Digital Illustration, painting, photo, ect..).
The version of the dataset I used had a defect where the word Skull was overused in the captions. For this, I recommend using Attention/Emphasis on skull and skulls tokens to decrease their attention. For example [[skull]]. This issue has been addressed in later iterations of the Dataset and won’t be a problem in future releases.
Similarly, since WnD-v1 used a small sample set from the full dataset — The intricacies of some concepts were not fully learned. For example, a prompt with Death Korps of Krieg and gasmask may generate a Kriegsmen but with the incorrect gas mask. To get around this, try increasing/decreasing the attention of the offending token or, remove the token entirely. In some cases, the models weight learned what the concept is so additional tokens become redundant. Again, this won’t be an issue in future releases.
When describing more than one character, try to describe their relative position to each other and appearance separately. For example, A illustration of a Sister of Battle of the Order of Our Martyred Lady and a Space Marine from the Blood Ravens Chapter. The Sister of Battle is standing to the left of the Space Marine. She is <description of the sister of battle.>. The Space Marine is <description of the space marine>. Then cross your fingers.
- Tag Prompting is a bit more difficult. I think I have a solution for this in later versions of the dataset. For now, try separating the characters by inputting the 1st character token, followed by tags that describe the 1st character. Then the 2nd character token, followed by tags that describe the 2nd character. For example, Illustration, Sister of Battle, black Sororitas Power Armor, Short Hair, Boltgun, and a Space Marine, Blood Ravens chapter, Dark red Power Armor with black and cream colored pauldrons, Heavy Bolter, ect..
For image composition — Try describing the image composition towards the end of the prompt. This applies to both natural language (less so) and tag / hybrid prompts.
If using an artist tag, place the artist token at the very end of the prompt.