The bottom line is, if you aren't doing anything wrong then what are you afraid of?
Im sure average joe german thought that too in 1932. (Not to mention quite a few russians before that)
Occasionally it is wise to not blindly follow authority without questioning it.