First of all: never use jpg for masked textures. You cannot prevent the formerly 0,0,0 black pixels from becoming other almost black shades that won't work as a mask. Use bmp or png. Unfortunately you still have to use black as the mask color here. The only exception is DDS. If you have photoshop or gimp there are export plugins.
You need to leave the masked parts unpainted. With the lasso you can easily cut out to-be-masked parts. In photoshop the checkerboard will then shine trough and represent a transparent or "non-painted" area. Blur the contours, then save it as DXT3 DDS with alpha information, then load it in BLitz with the mask flag.
Currently that's the best masked textures you can get in BLitz3D.
EDIT: oops I just see you want to use an alpha filter and not really masks. However, these notes may be useful noneless.
You need to leave the masked parts unpainted. With the lasso you can easily cut out to-be-masked parts. In photoshop the checkerboard will then shine trough and represent a transparent or "non-painted" area. Blur the contours, then save it as DXT3 DDS with alpha information, then load it in BLitz with the mask flag.
Currently that's the best masked textures you can get in BLitz3D.
EDIT: oops I just see you want to use an alpha filter and not really masks. However, these notes may be useful noneless.