The thing with AI technologies is that they have the ability to be unpredicatable. A property known as Emgergent Behaviour where abilities of an AI system emerge from the combination of 2 or more rules allow the AI system to perform actions it wasn't programmed to.
An example is the game Baldurs Gate, where the testers thought that they would be ok to build a wall around their town to stop the invading orcs from getting in. By using the knowledge of the wall size and the knowledge that various objects could be walked upon, the orcs were able to use obstacles to help them climb over the walls - something that they weren't programmed to do.
Sounds a bit far fetched, but just goes to show that people shouldn't mess with technologies that they can't guarantee complete control of. What happens when you have emergent behaviours emerging from combinations of emergent behaviours?
An example is the game Baldurs Gate, where the testers thought that they would be ok to build a wall around their town to stop the invading orcs from getting in. By using the knowledge of the wall size and the knowledge that various objects could be walked upon, the orcs were able to use obstacles to help them climb over the walls - something that they weren't programmed to do.
Sounds a bit far fetched, but just goes to show that people shouldn't mess with technologies that they can't guarantee complete control of. What happens when you have emergent behaviours emerging from combinations of emergent behaviours?