So dSetInternalSpaceCollideMode() enables/disables all collisions for objects inside that space? And spaces are hierarchical?
Yep, internal collide mode only applies to the sub-spaces though (not the top level space).
So I could create WorldSpace, PropSpace(worldspace) and ActorsSpace(worldspace). PropSpace and ActorsSpace would be subspaces of WorldSpace. Everything in worldspace, including everything in PropSpace and ActorsSpace, can collide. Let's say I create a dozen boxes in PropSpace and a dozen spheres in ActorsSpace, these could all bash into each other.
Yep, the objects within the same sub-spaces will collide with each other unless you disable internal collide mode.
Now to disable the boxes from colliding with each other, but still collide with spheres and allow spheres to collide with spheres, I can use dSetInternalSpaceCollideMode(). But how do I specify which spaces this should apply to? Or does it apply to ALL subspaces of the top-level space, so it would disable all cube<>cube and sphere<>sphere collisions?
Disabling the internal collide mode will prevent all collisions of objects within the same sub-space. All objects in the top level space will still collide with all sub-space objects. You would specify which objects collide with other objects by the way you design the sub-spaces. The simplest way to think about it is to put all objects that don't need collision detection between each other into the same sub-space. So, you could have all scene objects (terrain, plane, static scenery objects) into a sub-space of space (PropSpace). With collide mode disabled, if you want your actor objects to collide with each other, then you would put them into the top level space (Space). If not then you could create another sub-space for actor objects (ActorSpace). All objects created in the top level space (Space) will collide with the PropSpace objects and the ActorSpace objects.
While we're at it, looking for clues in the .decls file, what do...
dSetInternalContactSurfaceMode(mode):"odeSetInternalContactSurfaceMode@4" and
dGetInternalContactSurfaceMode%():"odeGetInternalContactSurfaceMode@0"
dSetInternalContactSurfaceMode(mode):"odeSetInternalContactSurfaceMode@4" and
dGetInternalContactSurfaceMode%():"odeGetInternalContactSurfaceMode@0"
Those are the functions you requested, their functionality is described up there ^^ (See the JV-ODE Version 1.05 Released post)
Also, does dSpaceAdd(space,geom) duplicate a geom's data into space, or just include in that space along with the geom's creation space, like box=dCreateBox(Space) creates the box geom in Space, would dSpaceAdd(PropSpace,box) just add collision calculations inside PropSpace? Or does that move box into PropSpace so it's collisions internal to PropSpace can be disabled?
I think it just adds a copy of the geom to the space (duplication), so you would need to use dSpaceRemove(space,geom) to delete the original if required.