I would say that approach is doomed.
Notice that you have different responses for vertical and horizontal walls.
But the collision response with the rectangle is the same in all cases, i.e. the geometry is completely ignored.
Notice that you have different responses for vertical and horizontal walls.
But the collision response with the rectangle is the same in all cases, i.e. the geometry is completely ignored.