Here is an alternative:
If the distances of the mouse to each point added together is almost equal to the distance of one point to another then the mouse is on the line. Pseudo code:
If Abs((Dist2D(mousex,mousey,p1x,p1y)+Dist2D(mousex,mousey,p2x,p2y))-Dist2D(p1x,p1y,p2x,p2y)) < line_thickness#
;mouse is on line
EndIf
This also works in 3D.
If the distances of the mouse to each point added together is almost equal to the distance of one point to another then the mouse is on the line. Pseudo code:
If Abs((Dist2D(mousex,mousey,p1x,p1y)+Dist2D(mousex,mousey,p2x,p2y))-Dist2D(p1x,p1y,p2x,p2y)) < line_thickness#
;mouse is on line
EndIf
This also works in 3D.