i was wondering if setting up an svn would slow don my site and if multiple up/downloads would cause any noticeable slow downs
That completely depends on the server and/or bandwidth you have available -- on a good server with a ton of bandwidth it would be virtually unnoticable, on a crappy server or low bandwidth connection it may be torture.