Function that returns a type

The reason that you have two instances is that you created two.

Type coo
	Field a
	Field b
End Type

test.coo = New coo ;<--- First instance created here, delete this line
test = jack(1,2);<------ Second instance created here, should be "test.coo = jack(1,2)"
Print test\a
Print test\b
WaitKey()
End

Function jack.coo (aa, bb)
	Local pff.coo = New coo
	pff\a = aa
	pff\b = bb
	Return pff
End Function

test is merely a pointer to one "coo" (yes, blitz uses pointers. Handle(test) returns the actual pointer instead of the object to which it points. Object.coo(<pointer>) will return the object that the <pointer> is pointing to.)
All custom type objects are global. What is confusing you and making you create a second object is that test and pff are both locals, they are pointers to objects that are global. You are creating an object and pointing test to it but then creating another object in a function, pointing another variable to it, then the function returns and changes where test is point so test is now pointing to the second object instead of the first.

Here is a silly illustration of what I mean:

Put a fork on my plate.
Put a piece of cake on my plate.
Aim my fork at the piece of cake. (test.coo = New coo)
Put another piece of cake on my plate. 
Put another fork on my plate and aim it at the second piece of cake. (pff.coo = New coo)
Move my first fork so it is aimed at the same cake as my second fork. (return pff)
Remove the second fork. (pff is deleted when the function closes because it is a local)

I now have 2 pieces of cake (coo objects) but only one fork aimed at the second one (test).