One issue I keep seeing in comments here and elsewhere on this issue is that academia is very competitive, with everyone worried about their rank. In my last post, I admitted that it would be hard to completely deny that there is competition, particularly when people are younger, which tends to come out when jobs are at stake. But, for the most part, I think the role of competition is completely exaggerated, strangely so. Academia -- at least, certainly, my branch of it -- thrives on collaboration, and I believe people who go into it thinking that it is a big competition are going to lose out, both professionally and in their enjoyment of the profession. (Indeed, one of the reasons I write these posts is because I don't want undergraduate and graduate students getting what I think is a one-sided, incorrect view of how academics work.)
First, I would again like to compare academics with other professions. I've seen comments (including on this blog) that other professions are less competitive than academia, and people are less worried about rank. I think people who are making that suggestion need to check in with people working in those professions, because I think they're ridiculously wrong. Lawyers coming out of school to get jobs, doctors trying to get fellowships or residencies, consultants at consulting firms -- all very competitive. As you move up, this continues. For lawyers, there's who can bill the most hours, and the challenge to make partner; for doctors, who can get the best positions at hospitals; and for businesspeople, every promotion is a step up. Even for software engineering types, there's competition. When I was a graduate student, I recall visiting friends who had gone to a large well-known company, and for a large part of the evening all they talked about was what "level" they and others were in the company and the annual reviews and who was doing what project that might get them ahead. So let's not be delusional and start by acknowledging that there's competition everywhere, and that's unsurprising when jobs and money are at stake. While I'm not suggesting I have deep knowledge of all of these careers, I think academics have much less competition than most.
If academics appear like they're concerned about ranking, perhaps it's because they appear to be easy to rank. First, as I pointed out last post, there's not that many of us. Second, there are obvious metrics everyone can understand: number of papers published, number of papers appearing in "top" conferences, and h-index stand out. I'm not suggesting these are good metrics -- but they're easy and to a first order give (potentially) some useful information. They're a quick way of bucketing or sorting people, particularly those without an established track record and are therefore not necessarily widely known or visible in the field, and therefore have more of an impression and an impact on younger academics.
But very quickly after your PhD, this sort of ranking loses its importance, and the very idea of ranking starts to lose its value -- as many have noted in a variety of venues. In academia, there's lots of ways to be successful, many points on the Pareto frontier. There are many great results waiting to be found and many different subfields to work in. At the end of the day, a history of good work and specific achievements is what people look for; there's not really a finite pool of such things for which to compete. Indeed, I'm not sure how I would go about competing with the top people in the field, except to try to do interesting work, which is what I'm trying to do anyway. (A secondary way to compete is just to make sure people know about your work. But giving talks is less important to being successful than doing the work that goes into the talks in the first place; again, it can have a bigger impact for people in the early stages of their career.)
Against this idea of competition, just look at how people in academia work together. In computer science theory, in particular, most papers have several authors working together. In a number of cases these are students with their advisors, but a closer look reveals that in many cases, they are not. Credit can be shared easily in academia, and collaborations can lead to results that individuals could not get alone. Working in groups is a way for people to get more done. Instead of competition, collaboration often yields the path to having a greater impact on the field. Rather than being a "competitive game", research is more a "cooperative game". (As an aside, this is why theory's approach of alphabetical order for authors rather than some sort of implicit "credit scheme" based on author order makes such great sense.) In most areas of computer science that I've participated in, a similar spirit prevails.
I encourage graduate students to team up pick out projects to work on together (and have seen this at other places, also -- one of my best experiences as a graduate student was such a project). It gives them something to do and a sense of ownership outside of working with their advisor. And, importantly, in reinforces that these other students are their colleagues, and that working together is a great idea that can gain for everyone. Hopefully, they also learn that working together is more fun and generally more productive than working alone. When it comes to hiring time, it's nice to see students who have worked on such team projects, because I typically prefer colleagues with a track record of working well with others.
Sometimes small competitions break out, sure -- multiple groups are working on the same or similar problems. Often, though, this is a very healthy competition, pushing progress forward in an area over a series of papers. I remember in the past an argument with another group when we were working on similar problems and an issue of "credit" in the writeup of the various papers came up. A week later, we were starting collaborations together on new ideas. That's not exactly the sign of a super-competitive landscape.
It could be that I've just got a mistaken impression of the field. Harvard is a wonderfully collaborative place, and personally I've found overall I like working with others more than on my own. But when I think of cutthroat competition, I don't think of the job I'm in.
To conclude the post, I think what may be going on is people confuse "competition" with "drive". Most academics are smart, successful people, who are internally driven by a desire to do great work. To many, that must appear like "competition", but if so, it's internal competition -- you're not out to beat others, but to be our best. And I think it's very possible academia does have more "Type A" personalities that have this internal drive that is, surely, not always healthy. It's not clear to me that this academia's fault -- such people would be similarly driven in any career -- but, if it is true, then it suggests we might consider if this is best for our field, and how we might open up the field to a wider set of personalities or how we might make work in this field healthier for this type of personality.