The problem is you're taking a half-assed approach to stencil shadows.
There are two halves to your problem here.
The first half is that you are not rendering the shadows correctly. Imagine rendering all the shadows first, pure black, onto a white background. If they overlap, it doesn't matter. That area is pure black no matter what. After you're done rendering the shadows, then you can brighten them up a bit. Maybe make them fall off from the light source by distance. Don't ask me how to do this, I don't know.
The second half of your problem is that in Doom, EVERYTHING cast shadows from EVERY light source. Assuming you have one light source immuminating your scene, and everything cast shadows, and you rendered the shadows first, and black, as described above "this is crap" would not be visible, because that area would be in shadow because of the walls.
There probably isn't a way to render stencil shadows only for some things but not for others. If there is, I bet it requires the latest version of DirectX.
As for multiple light sources, I don't know how you do those. I suspect Doom didn't do them, cause I don't remember seeing anything other than single shadows in any area. I don't know how they could have done transitions between light sources though.
Hope that's of some help. My own shadow system does not stop at walls either. In order to do that, I would have to treat the projected shadow as a plane, clip all the polygons in the light's frustrum, sort them from front to back, and then clip away bits of that plane as I applied the shadow to each polygon from front to back until I ran out of shadow or polygons to clip it. This would be slow. Though probably not as alow as clipping all those polygons in the first place which I already have to do.
PS:
I could be wrong about that first bit. Maybe I am not seeing what I think I see in the first image cause of all your light sources. And maybe you can fix the problem with clever stencil shaodw stuff.
There are two halves to your problem here.
The first half is that you are not rendering the shadows correctly. Imagine rendering all the shadows first, pure black, onto a white background. If they overlap, it doesn't matter. That area is pure black no matter what. After you're done rendering the shadows, then you can brighten them up a bit. Maybe make them fall off from the light source by distance. Don't ask me how to do this, I don't know.
The second half of your problem is that in Doom, EVERYTHING cast shadows from EVERY light source. Assuming you have one light source immuminating your scene, and everything cast shadows, and you rendered the shadows first, and black, as described above "this is crap" would not be visible, because that area would be in shadow because of the walls.
There probably isn't a way to render stencil shadows only for some things but not for others. If there is, I bet it requires the latest version of DirectX.
As for multiple light sources, I don't know how you do those. I suspect Doom didn't do them, cause I don't remember seeing anything other than single shadows in any area. I don't know how they could have done transitions between light sources though.
Hope that's of some help. My own shadow system does not stop at walls either. In order to do that, I would have to treat the projected shadow as a plane, clip all the polygons in the light's frustrum, sort them from front to back, and then clip away bits of that plane as I applied the shadow to each polygon from front to back until I ran out of shadow or polygons to clip it. This would be slow. Though probably not as alow as clipping all those polygons in the first place which I already have to do.
PS:
I could be wrong about that first bit. Maybe I am not seeing what I think I see in the first image cause of all your light sources. And maybe you can fix the problem with clever stencil shaodw stuff.