You could do this easy enough by following old film special effects tricks (matte paintings ect.)
create your terrain...load in the rock texture...load in the mask texture...make a copy of the terrain, make a copy of the mask...and load in the grass texture.
so you should have two terrains, and two mask textures...
apply the rock texture to the first terrain, apply the grass texture to the second (copied) terrain.
Now comes the fun part...if the mask texture is ment for the grass texture (that is the white areas are where the grass will be visable, black where it is invisable) then multiply blend apply one of the masks to the terrain with the grass texture.
for the other mask...go through each pixel and invert it, that is you want a negative mask where all black areas become white, and white areas become black (newred = 255 - oldred, newgreen = 255 - oldgreen, etc..)...then multiply blend the negative mask on the terrain with the rock texture.
at this point the rock terrain will have darker black areas where the grass is suppost to show...and the grass terrain will have darker black areas where the rock is to show through...so to make them blend together correctly addiveblend the grass terrain entity...this allows the grass to show in the darkened areas of rock terrain, but allows the rock terrain to show in the darked areas of the grass terrain...
create your terrain...load in the rock texture...load in the mask texture...make a copy of the terrain, make a copy of the mask...and load in the grass texture.
so you should have two terrains, and two mask textures...
apply the rock texture to the first terrain, apply the grass texture to the second (copied) terrain.
Now comes the fun part...if the mask texture is ment for the grass texture (that is the white areas are where the grass will be visable, black where it is invisable) then multiply blend apply one of the masks to the terrain with the grass texture.
for the other mask...go through each pixel and invert it, that is you want a negative mask where all black areas become white, and white areas become black (newred = 255 - oldred, newgreen = 255 - oldgreen, etc..)...then multiply blend the negative mask on the terrain with the rock texture.
at this point the rock terrain will have darker black areas where the grass is suppost to show...and the grass terrain will have darker black areas where the rock is to show through...so to make them blend together correctly addiveblend the grass terrain entity...this allows the grass to show in the darkened areas of rock terrain, but allows the rock terrain to show in the darked areas of the grass terrain...