Skip to content

feat: add LLaDA-Image support - #1968

Draft
fszontagh wants to merge 3 commits into
leejet:masterfrom
fszontagh:feat/llada-image
Draft

feat: add LLaDA-Image support#1968
fszontagh wants to merge 3 commits into
leejet:masterfrom
fszontagh:feat/llada-image

Conversation

@fszontagh

@fszontagh fszontagh commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds text-to-image and instruction-guided editing for
LLaDA-Image (Apache-2.0). The denoiser is a
Lumina2/Z-Image-style NextDiT conditioned by a LLaDA2-MoE diffusion-LLM text encoder, and it
reuses the Flux.2 VAE.

Most of it reuses what is already here:

  • The DiT instantiates ZImage::JointTransformerBlock / FinalLayer; the hyperparameters are
    identical. Z-Image's blocks gained two defaulted parameters, norm_elementwise_affine
    (LLaDA's norms are parameter-free) and split_qkv (its checkpoint keeps q/k/v separate).
  • The text encoder is a new LLMArch::LLADA2_MOE on the existing MoE path, adding
    grouped-sigmoid routing with an expert bias, a shared expert, partial rotary and
    bidirectional attention.
  • New: QueryFormer, text projection, and SigVQ (editing only, constructed only when those
    weights are present), plus the llada_image reference-image preset and sigma schedule.

Editing runs the clean reference and the noisy target in one sequence. Because adaLN is a
linear map of the timestep embedding, feeding a per-token embedding selects the right
modulation exactly, so no new modulation machinery was needed.

One change outside the model: tensor_should_be_converted now skips *_pad_token.
LLaDA-Image stores those around 7.7e24, so f16 storage and q8_0's f16 block scale both
overflow to inf, and every GGUF I converted rendered blank white until this was added.

Docs in docs/llada_image.md; rows added to README.md and docs/edit.md. The connector
components ship in separate directories, so the docs show how to merge them into the one file
--embeddings-connectors expects, and link pre-merged copies.

Related Issue / Discussion

Closes #1937

Additional Information

Verified on an RTX 3060 12GB, CUDA, with LLaDA-Image-Turbo int8_tensorwise and with q8_0 and
f16 GGUFs converted by -M convert, against a Q4_K text encoder. No offload flags are needed.

512x512 / 768x768 / 1024x1024 10.5 s / 22.3 s / 35.5 s
1024x576, 576x1024 22.4 s
editing, 512x512 107 s

assets/llada_image/example.png is "a lovely cat holding a sign says 'llada.cpp'";
edit_example.png is that image with "change the sign text to 'sd.cpp'".

Segmented execution was checked against unconstrained execution at 512x512: --max-vram 6,
--max-vram 4, --max-vram 3 and --offload-to-cpu all produce byte-identical output, so the
roughly 16 GB of weights run in a 3 GB budget.

Editing packs reference and target in one sequence, so it needs about twice the tokens of
text-to-image at the same size; at 1024x1024 on 12 GB it needs --diffusion-fa.

Not implemented: generation_mode="vq", where the text encoder block-diffusion-decodes VQ
tokens before diffusion. Text-to-image and editing do not use that path. Tested with
int8_tensorwise and GGUF quantizations only.

Checklist

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please put merge and vocab on a single line, and remove the helper scripts.

@fszontagh
fszontagh marked this pull request as draft September 13, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] LLaDA-Image 6B

2 participants