I'm not sure if I get you right. Maybe you're talking about darkish blended contours of the masked textures, like a dark border? This is well known "feature". Since masked textures require to use black as transparent color (even with TGA), and Texturefiltering will kind of blur things (as well as mipmapping does), you will be forced to use
A) A solution posted in the archives that will paint black texels with a neutral background color while leaving the alpha channel of the loaded mesked texture intact. I have posted such a solution too, not so long ago.
B) Use the new texture formats DDS, specificly DDS DXT3 with alpha information. You need to cut out the black parts, so they become transparent (not talking about a 4th channel for alpha!). Then select the outline of the masked thing, select border (eg. 4), then blur it (more), so the alpha channel will also be blured and allow directx to use smooth vector curves for the mask contour. In short: use DXT3 with alpha, then load it in mask mode. This is the reccomended way.
(nvidia offers a free PS export plugin I think)
A) A solution posted in the archives that will paint black texels with a neutral background color while leaving the alpha channel of the loaded mesked texture intact. I have posted such a solution too, not so long ago.
B) Use the new texture formats DDS, specificly DDS DXT3 with alpha information. You need to cut out the black parts, so they become transparent (not talking about a 4th channel for alpha!). Then select the outline of the masked thing, select border (eg. 4), then blur it (more), so the alpha channel will also be blured and allow directx to use smooth vector curves for the mask contour. In short: use DXT3 with alpha, then load it in mask mode. This is the reccomended way.
(nvidia offers a free PS export plugin I think)