How many of the games in that 20+ years had more than 100 people working on it?
I think you'll find that even today, the number of games produced by more than 100 people is less than 1%. If you won't take my word for it (or Johns word for it) industry veteran Daniel Irish (Mobygames Bio), claims the average team size is between 20-60 people. (Game Producers Handbook, you can get it of Amazon, I suggest you read it). We actually went over all this in a design class at UAT. We had big huge discussions about design documents and what they are good for.
They are good for keeping a project focused, and preventing scope creep / feature creep. The conclusion was that as the number of people working on a game (and the focus of the school is obviously on the AAA industry, not indie or even small scale games) goes up, the design document becomes more and more essential.
Well the conclusion was wrong. As the number of people go up, design documents become less important, and DUR's become more important. The ideal configuration for productivity is autonomous groups of between 4 and 8 people. So if you have a larger project, you break it down into tasks that are solvable within your timebox (usually 14 days), by the team that's taken on the task.You then use some sort of process management tools (you can use Gantt charts and Microsoft Project, or you can use SCRUM and divide your work into Sprint, Release and Backlogs, or you can use XP, TQM, or whatever other process management tools take your fancy. You can even mix and match between them).
There are even jobs becoming available in some of the bigger development 'mills' that focus ONLY on document upkeep.
Yes. Why do you think these jobs are available? To paraphrase Fred Brooks: "It is more cost effective to have good programmers develop and mediocre programmers document". Congratulations, you just landed a job that involves keeping 1000's of pages of information up to date, so you can tell the programmers what they need to know when they ask, so they don't have to spend their valuable time sifting through hundreds of pages of irrelevant drivel. Additionally. Design Documents are the perfect storage medium for game ideas.
Actually that would be Treatments. The Design Documents main purpose is to prevent feature creep and thus it's a really poor choice for keeping track of game ideas. Who should I trust more. John Pickford someone who has worked on several true classics, or Mark Baldwin a man with some game of the years under his belt.
Like I said, those who can, do. Those who can't, teach. In any case Mark Baldwin is not credited with Metal Fatigue on his Mobygames Bio, he might want to rectify that if he did indeed work on the game. From what I can see tho' he was mainly involved with Empire and Worms sequels. As for who you should trust more, I don't think anyone really cares - should you land a job as a designer, you'll learn soon enough how many people bother to read design documents. We can talk again then. I see a lot of games being designed through a wiki. Is that really any different when you get down to the intention?
Yes. Although it greatly depends on how concise the designers are.