How-To: Making Artistic QR Codes With Stable Diffusion and ControlNet

Early in June 2023, two individuals named Lu Xin and Sun Gouyo showcased images of QR codes that resembled art rather than typical digital tags for scanning with a smartphone. You can find one of the articles discussing this here: link. According to the creator, they had trained a new specialized controlnet model called "controlnet qrcode_webui_fp16" and shared some highly impressive QR code images.

As of now, the creators have not made this controlnet model public, much to the frustration of many individuals. However, creative and technical enthusiasts are persistently trying to uncover the secrets behind their model and process. The creators did provide some hints regarding the process and models used, mentioning two controlnet models called "control_v1p_sd15_brightness" and "control_v1p_sd15_illumination."

UPDATE June 19 2023: New ControlNets are available on Hugging Face for QR code generation. Additionally a free publicly available QR art generator is also now available: https://huggingface.co/spaces/huggingface-projects/QR-code-AI-art-generator

Numerous people have presented their ideas, models, and workflows on platforms like Reddit, Facebook, and other social media groups, aiming to create their own artistic QR codes. I have compiled these ideas and incorporated some of my own workflow to attempt this myself.

To clarify, none of these attempts have achieved the same level of success as the QR code images shared by the original creators and their customized controlnet. Nevertheless, some of these attempts are truly impressive. It will be fascinating to observe how far they can push the boundaries while still ensuring compatibility with the majority of smartphones for scanning the QR codes.

Prerequisites:

To proceed, you will need the following:

1. Access to Stable Diffusion (SD) with ControlNet installed
2. Models for both SD and ControlNet
3. A QR code

There are multiple ways to run SD, and a good starting point is the How-tos provided by Eight Buffalo Media Groups. If our documented solution does not meet your requirements, you can search on YouTube for alternative options that suit your needs.

If you choose to use Automatic1111 for installing SD, as we have documented, you can install Controlnet as an extension through the extension tab. Refer to our How-to article 7 for instructions on installing extensions, and install the 'sd-webui-controlnet' extension just like any other extension. Once again, you can search on YouTube to find the best options for you, including newer versions of Controlnet and newer models.

There are numerous models available for SD, but we (shocker) recommend using ours. You can download it, along with many other SD models, from Hugging Face or Civitai:

Civitai: link
Hugging Face: link

For ControlNet, I suggest grabbing multiple models to experiment with. All the common models can be found here (Hugging Face: link), including one called 'control_v11f1e_sd15_tile' that some users have found much success with. However, I recommend installing all of them if you have sufficient storage space:

The controlnet models mentioned in the article, namely 'control_v1p_sd15_brightness' and 'control_v1p_sd15_illumination,' can be found here at Hugging Face: link

Lastly, you will need a QR code. I used a website called QRCode Monkey, which provides free PNG files. You can access it here: link.

Now that we have documented the prerequisites, let's proceed with the workflow.

Workflow:

It's important to note that this entire process is experimental. The original QR codes that were posted (available in Chinese here) and inspired us all to explore this creative avenue were created using a custom trained controlnet model. Therefore, it's best to keep your expectations low when starting this endeavor.

Below each image, I have shared my prompts and settings. You'll notice that there are a few settings that remained consistent for each image. However, for every successful QR code I managed to scan on my iPhone, I had three images that didn't work. Based on comments and feedback from others, it seems that different images may yield varying results depending on the phone used, including the original images with the used controlnet.

Here's a list of things that worked and didn't work for me:

• I observed more consistent results when using two different controlnet models rather than just one.
• The two models that yielded the most success for me were 'control_v1p_sd15_brightness' and 'control_v11f1e_sd15_tile.' I achieved the best results when using them together, although the QR code still remained more noticeable than desired.
• QR codes for longer URLs (e.g., https://sites.google.com/eightbuff.com/eight-buffalo-media-group-llc/lowly-fans) compared to shorter domain names (e.g., http://eightbuff.com) have more blocks in the QR code and proved more challenging to generate successfully. Therefore, I recommend opting for shorter URLs when creating QR codes.
• The image you select matters. Be creative and think of images where the targeting marks (the three large squares or circles in your QR code) can be obscured.
• Adjust the "Weight" and "Guidance End" parameters to minimize the visibility of the QR code while making the image more prominent. I had to adjust these parameters for each generated image, sometimes across all the controlnets I was using.
• Setting the Controlnet Canvas height to 512x512 and the image to 768x768 provided the best results for me. I suggest adjusting the size settings, but keep in mind that changing the size of your image for many SD models can significantly alter the generated image.

As you can see, many of my QR codes still have prominent visibility and require further refinement. However, they serve as a starting point for my improvement iteration until I decide to invest time in creating a controlnet or until someone provides one for public use.

Please share your successes in generating QR code images in the comments or tag us on Facebook. I would like to express my gratitude to everyone who shared their workflows on public forums, which saved us many hours of research. If anyone would like to be mentioned or have their link work included at the end of this post, please let us know at info@eightbuff.com, and we will gladly add you to the acknowledgments in this post.

If you would like to have us generate images of QR codes for you, join our Pateron site and get image generation as a free benefit! https://patreon.com/EightBuffaloMediaGroup 

Positive prompt: herd of buffalo on the open plains,
Negative prompt: poor quality, ugly, blurry, boring, text, blurry, pixelated, ugly, username, worst quality, (((watermark))), ((signature)), face, worst quality, painting, copyright, unrealistic, (((text)))Steps: 100, Sampler: Euler a, CFG scale: 8, Seed: 1687514050, Size: 768x768, Model hash: 4a54cde627, Model: 8buffGen_v10Noemapruned, ControlNet-0 Enabled: True, ControlNet-0 Module: none, ControlNet-0 Model: control_v1p_sd15_brightness [5f6aa6ed], ControlNet-0 Weight: 0.6, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 0.83, Version: v1.3.2

Positive prompt: Beautiful mountain town,
Negative prompt: poor quality, ugly, blurry, boring, text, blurry, pixelated, ugly, username, worst quality, (((watermark))), ((signature)), face, worst quality, painting, copyright, unrealistic, (((text)))
Steps: 100, Sampler: Euler a, CFG scale: 8, Seed: 3274261988, Size: 1024x1024, Model hash: 4a54cde627, Model: 8buffGen_v10Noemapruned, ControlNet-0 Enabled: True, ControlNet-0 Module: none, ControlNet-0 Model: control_v1p_sd15_brightness [5f6aa6ed], ControlNet-0 Weight: 0.5, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 0.8, Version: v1.3.2

Positive prompt:big city at sunset, skyscrapers,
Negative prompt: poor quality, ugly, blurry, boring, text, blurry, pixelated, ugly, username, worst quality, (((watermark))), ((signature)), face, worst quality, painting, copyright, unrealistic, (((text)))
Steps: 100, Sampler: Euler a, CFG scale: 10, Seed: 40917465, Size: 768x768, Model hash: 431208a773, Model: 8buff_GEN_FP16_v1, ControlNet-0 Enabled: True, ControlNet-0 Module: none, ControlNet-0 Model: control_v11p_sd15_scribble [d4ba51ff], ControlNet-0 Weight: 0.4, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 0.55, ControlNet-1 Enabled: True, ControlNet-1 Module: none, ControlNet-1 Model: control_v1p_sd15_brightness [5f6aa6ed], ControlNet-1 Weight: 0.5, ControlNet-1 Guidance Start: 0, ControlNet-1 Guidance End: 0.8, Version: v1.3.2

Positive prompt: big city at sunset, skyscrapers,
Negative prompt: poor quality, ugly, blurry, boring, text, blurry, pixelated, ugly, username, worst quality, (((watermark))), ((signature)), face, worst quality, painting, copyright, unrealistic, (((text)))
Steps: 100, Sampler: Euler a, CFG scale: 10, Seed: 1687391782, Size: 768x768, Model hash: 431208a773, Model: 8buff_GEN_FP16_v1, ControlNet-0 Enabled: True, ControlNet-0 Module: none, ControlNet-0 Model: control_v11f1e_sd15_tile [a371b31b], ControlNet-0 Weight: 0.15, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 0.2, ControlNet-1 Enabled: True, ControlNet-1 Module: none, ControlNet-1 Model: control_v1p_sd15_brightness [5f6aa6ed], ControlNet-1 Weight: 0.5, ControlNet-1 Guidance Start: 0, ControlNet-1 Guidance End: 0.8, Version: v1.3.2

Positive prompt: snowy mountain temple,
Negative prompt: poor quality, ugly, blurry, boring, text, blurry, pixelated, ugly, username, worst quality, (((watermark))), ((signature)), face, worst quality, painting, copyright, unrealistic, (((text)))
Steps: 100, Sampler: Euler a, CFG scale: 9, Seed: 3763461602, Size: 768x768, Model hash: 431208a773, Model: 8buff_GEN_FP16_v1, ControlNet-0 Enabled: True, ControlNet-0 Module: none, ControlNet-0 Model: control_v11f1e_sd15_tile [a371b31b], ControlNet-0 Weight: 0.15, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 0.2, ControlNet-1 Enabled: True, ControlNet-1 Module: none, ControlNet-1 Model: control_v1p_sd15_brightness [5f6aa6ed], ControlNet-1 Weight: 0.6, ControlNet-1 Guidance Start: 0, ControlNet-1 Guidance End: 0.82, Version: v1.3.2

Positive prompt: person, long hair, black hair, Realistic, armor, full body, intricately detailed, crafted, meticulous, magnificent, maximum details, extremely hyper aesthetic, Sketch, Manga Sketch, Pencil drawing, Black and White, Manga, Manga style, Line art, vector art, Monochromatic, by katsuhiro otomo and masamune shirow and studio ghilibi,
Negative prompt: poor quality, ugly, blurry, boring, text, blurry, pixelated, ugly, username, worst quality, (((watermark))), ((signature)), face, worst quality, copyright, unrealistic, (((text))), nsfw, nude,
Steps: 100, Sampler: Euler a, CFG scale: 11, Seed: 291441644, Size: 768x768, Model hash: 431208a773, Model: 8buff_GEN_FP16_v1, ControlNet-0 Enabled: True, ControlNet-0 Module: none, ControlNet-0 Model: control_v1p_sd15_brightness [5f6aa6ed], ControlNet-0 Weight: 0.45, ControlNet-0 Guidance Start: 0, ControlNet-0 Guidance End: 0.82, Version: v1.3.2