Creating layer masks for your images

SOFTWARE: Photoshop (intermediate difficulty).

Among the best inexpensive stock art resources for DMs Guild creators are the free WOTC-provided DMs Guild Creator Resource images. They appear in a lot of DMs Guild titles—but none (or at least most) of the images have transparent backgrounds, and many of the DMs Guild creators don’t know the best ways to make transparent backgrounds from flat/opaque stock art.

Today I’ll show you a (usually) quick and powerful technique for layer mask creation using one of Adobe Photoshop more advanced features, the Calculations command.

But first, let’s look at a couple of other, commonly used techniques—and why they’re less than ideal.

NOTE: This is an intermediate level tutorial! If you don’t already know the Photoshop basics and understand what layer masks are, I recommend familiarizing yourself with them (especially layer masks) before proceeding. This page on masks from Adobe’s online Photoshop User Guide explains the basic idea and the two most common techniques for creating masks for art in Photoshop: vector masks and (raster) layer masks. (You can skip the bits about vector masks, because they’re not relevant to this tutorial.)

I am inordinately fond of goblins, and this is an excellent painting, so I selected this piece of WOTC art, on a mock-up page with the sort of “old paper” background that’s used in a whole bunch of DMs Guild and other 5e-compatible titles.

Obviously, the unaltered original doesn’t work on this background. You could reduce it within the margins and crop it down a bit, but the angle of the top of the ilustration fits so nicely with the text above it! Why lose that?

One quick and dirty technique I’ve seen is to simply change the blend mode of the image to Multiply in InDesign or whatever page layout program you’re using. (Homebrewery-built titles use this frequently, through the mix-blend-mode CSS property.) This is what that looks like:

It looks clean and seamless, but because all of the color in the image gets blended into the background, this technique makes the whole image look kind of jaundiced (at least on a yellowish background like the one I’m using). Here, it’s not too bad, because the image is already pretty yellow, but it also looks darker. And with other images, this color shift might look pretty bad.

Another technique is using a clipping path (a.k.a. a vector mask). InDesign can detect the edges of your illustration and generate an automatic clipping path.

At first glance, this doesn’t look bad, either! But looking closely, you can see that there are lighter areas where the clipping path doesn’t quite nail it.

Drawing a vector mask by hand would work better than automatic paths… but drawing complex vector paths is time-consuming—and the hard edges of a clipping path are a poor fit for illustrations with soft edges or vignettes like this image. If you’re a layout artist making less than $5 a page, the last thing you want to do is spend a lot of time doing something that will only look okay.

Better than all of these, in most cases, is the Calculations technique. It’s (usually) quick, but it does require a little manual labor. The results speak for themselves, though: the background vignette is blended seamlessly into the paper background without a yellow cast to the characters or background—so you don’t lose the highlights, as with the Multiply method:

Admittedly, because the original image is a little yellow, this is not easy to see, but if you look carefully, the brightest highlight on the skull is slightly brighter than the color of the paper:

Here’s how the technique works:

1. Prepare your file.

The safest way to work in Photoshop is to ensure that you can’t irreversibly screw anything up. I like to duplicate the original artwork layer and put it at the bottom of the file, just in case I start drawing on the wrong layer.

Since were’re making the image transparent, we want to be able to preview and test that transparency easily. To do that, I create two additional layers: one white, and the other filled with any color that will stand out from the rest of the image; I like pure cyan (RGB 0/168/236).

Name your layers intuitively! Even if you don’t have collaborators on a project, the names help you find your bearings when you need to look in old files.

So now my file looks like this:

2. Identify the darkest channels.

RGB colors are comprised of information in three separate channels: Red, Green, and Blue. The Channels palette allows you to look at and edit each of these channels separately.

Always do any Photoshop work in RGB (or whatever the native colorspace of the original image is) and convert to CMYK only when absolutely necessary and at the last possible moment. The wider gamut of RGB gives you more room to manipulate the relationships between light and dark (and other colors), which is helpful for creating high quality masks and any other image editing you need to do.

Looking at your image’s channels, identify which of the channels is the darkest for various parts of the image. Yellows show up in the Blue channel, magentas show up in the Green channel, and cyans show up Red channel. (I know that it’s because RGB is additive not subtractive, but I can’t really wrap my head around that. You don’t really need to understand why it’s the case, though; you just need to be able to use the channel information.)

For this goblin illustration, the red channel gives me the best information about the blue “sky” in the background, while the blue channel looks pretty good for everything else.

3. Use Calculations to create a selection.

Next, open the Calculations panel (under the Image menu). Using this command, you will be taking the information from channel or channels you identified in Step 2 and blend them together in order to give you a solid starting point for your layer mask.

Change the Source 1 and 2 channels to the one(s) you identified in Step 2 and change the Blending mode. Multiply or Linear Burn work the best for creating a dark, reasonably solid looking result.

With Preview checked, play around with different channels and blending modes to get a feel for what the Calculations panel is actually doing! I ultimately decided that mixing the Green and Blue channels gave me the most solid mask on the figures, with solid-enough values in the sky.

You want the darkest areas to be as solid/black as possible, while being careful not to make the edges too harsh, because then your mask will look harsh. To soften the results, you can adjust the opacity of the blending mode (or simply use a different blending mode). The mask does not need to be totally solid! This is only a starting point.

Finally, set the Result to Selection and hit OK to process the results.

4. Convert the selection to a layer mask.

You now have the results of the Calculations command selected. Make sure that the Artwork layer is selected in the Layers panel, then click the “Add Layer Mask” button at the bottom of it (it’s technically a circle on a rectangular field, but I think it looks like the Japanese flag, so I remember it that way).

That will give you something that looks like this:

The visible art is where your mask is NOT yet covering. As you can see, there’s way too much of the background showing, and a lot of the interior areas that we want to be opaque (on the goblins and their weapons, on the rocks) are not. As I said in the last step: this is a starting point.

To preview your mask, you can select the layer mask invert it (Cmd-I or Ctrl-I in Windows). I use the cyan layer and white layers created in step 1 to preview the transparency more easily, as seen here:

5. Adjust the layer mask.

Next we need to solidify the mask as much as possible—again, without making it too harsh.

Select the and use either Curves or Levels to finetune the mask. You want to darken up the dark areas (by sliding up the black point) and eliminate any noise from the lighter areas (by sliding down the white point).

It doesn’t matter whether you use Curves or Levels, as long as the results look solid and smooth. I prefer using Curves, because it gives me more control over how much to adjust the values of both light and dark in a single adjustment. Here, the curve between the white and black points helps to darken the mask, while leaving intact the areas that are supposed to retain some degree of transparency (the background).

It’s okay if the background is still a little transparent; that will let the color blend into the background of the page better in InDesign. (Note: If you’re working on a DMs Guild Creator Resource image, you may notice some blocking in the transparent areas; this is due to JPG compression in the original image and is unavoidable. It will be less noticeable in the final image.)

I find it easier to refine the layer mask by looking for elements that the mask isn’t covering up—but should be, generally the brightest highlights in the image. For instance, on the bones and on the rock in the background.

We’ll have to manually fill those areas in the next step.

6. Clean up the mask as needed.

Using the Brush tool, fill in the mask where needed (the highlights on the bones, the rock in the background, etc.). Since your mask should already be pretty dark overall, you should use a solid black, 100% hard brush to fill in those areas. You may need to erase (or draw over with white) any noisy areas in the art, as well.

You can also solidify some of the background with a 100% soft brush if you want, such as in the blue (sky) background. Take care not to paint in areas that you want be be transparent, like gaps in the figures or solid parts of the background (the rock and ground). The softness of the brush allows a smooth, subtle transition between the image and the page background of your book.

The mask you make may not be perfect, but that’s okay! If you compare my final mask with the original image above, I’ve lost a little bit of the vignetted blue and orange in the background. But more importantly, I’ve preserved the whites and highlights in the focal points of the image: the goblins themselves.

This looks more complicated than it is! Once you’ve learned the technique, you can easily get this masking process for most images down to about five minutes for most images. In the video here, I create a mask for a different image in real-time:

During the cleanup phase for this video example, I noticed that the mask was including a slim hairline around the entire image and used Select / Modify / Contract to eliminate that. I did these touch-ups with a mouse, because they were super simple, but having a basic graphics tablet like a Wacom Bamboo can help.

Some images will make your life difficult and require more manual touching up than others (particularly light objects against light backgrounds), but this Calculations command will usually be your fastest and most effective way to create an alpha channel.

——————————

LEGAL STUFF: This post (“Creating layer masks for your images”) on the Polyhedral website and its associated video tutorial are unofficial Dungeons & Dragons Fan Content permitted under the Fan Content Policy. It is not approved, endorsed, or sponsored by Wizards of the Coast. Portions of the materials used are property of Wizards of the Coast. © Wizards of the Coast LLC.