I think basically you do this:
For any boolean operation:
Split all the polygons in A with the planes of the polygons in B, and vice versa.
Operate on these split versions of the meshes.
To subtract B from A:
1. Flip polygons in B.
2. Find all the polygons of B which lie inside A. Add those to final mesh.
3. Find all polygons in A which lie outside B. Add those to the final mesh.
4. Weld vertices of final mesh if desired.
To add A and B:
1. Find all polygons of B which lie outside A. Keep them.
2. Find all polygons of A which lie outside B. Keep them.
4. Weld vertices of final mesh if desired.
AND and NOT operations aren't all that useful so I won't bother to explain those.
For any boolean operation:
Split all the polygons in A with the planes of the polygons in B, and vice versa.
Operate on these split versions of the meshes.
To subtract B from A:
1. Flip polygons in B.
2. Find all the polygons of B which lie inside A. Add those to final mesh.
3. Find all polygons in A which lie outside B. Add those to the final mesh.
4. Weld vertices of final mesh if desired.
To add A and B:
1. Find all polygons of B which lie outside A. Keep them.
2. Find all polygons of A which lie outside B. Keep them.
4. Weld vertices of final mesh if desired.
AND and NOT operations aren't all that useful so I won't bother to explain those.