Did you speak to the prisoner in the cage at the top of the tower?
There was no prisoner, however i found out i was in the place that opened the gates that allowed me into an even larger area within oblivion where there was a prisoner at the top, i got the stone and all is done :)
Now, I must have theived somethig without even realising because I stoopped to talk to this dude on a horse and promptly got arrested.
Now, how do I go and get my horse back? :S