There are ways of avoiding them, but I don't remember ;] Either they're actually on patrol and move around occasionally, or you might be able to destroy a structure up north and get their attention that way, then run back around before they figure out you're screwing with them =]