Control of texture mask

I want to be able to select which pixels are masked, and I want masked pixels to be able to be other colors than pure black. I want to use a color you can actually see to select the mask, and then color the masked pixels with an averge hue of the non-masked pixels.

My trees look cel-shaded right now because of the black masked pixels creeping in.