How Life Sciences Actually Work: Findings of a Year-Long Investigation

created: ; modified:

See discussion on /r/slatestarcodex (a) (25 comments), Hacker News (a) (17 comments), Twitter (1 (a), 2 (a)) (>300 retweets), Marginal Revolution (a) (39 comments)

Summary: academia has a lot of problems and it could work much better. However, these problems are not as catastrophic as an outside perspective would suggest. My (contrarian, I guess) intuition is that scientific progress in biology is not slowing down. Specific parts of academia that seem to be problematic: rigid, punishing for deviation, career progression; peer review; need to constantly fundraise for professors. Parts that seem to be less of a problem than I initially thought: short-termism; lack of funding for young scientists.

2021 update: I became quite a bit more pessimistic about the future of academia since I wrote this essay and am now working on New Science.

Introduction

In his Asymmetric Weapons Gone Bad (a), Scott Alexander notes that for some areas of inquiry, studying them a little bit leads you astray and only studying them a lot makes it possible to understand what’s really going on:

Maybe with an unlimited amount of resources, our investigations would naturally converge onto the truth. Given infinite intelligence, wisdom, impartiality, education, domain knowledge, evidence to study, experiments to perform, and time to think it over, we would figure everything out.

But just because infinite resources will produce truth doesn’t mean that truth as a function of resources has to be monotonic. …

Some hard questions might be epistemic traps – problems where the more you study them, the wronger you get, up to some inflection point that might be further than anybody has ever studied them before.

I think that this observation is a general one – true for almost all areas of study – and moreover lends itself to a much simpler explanation: almost everyone has an agenda. Almost no one has the exact right picture of what’s going on in their mind.

Thus, when starting to study something and reading one or two books or talking to a couple of people, you will inevitably overload on their agenda, personal biases, personal ignorance, and all the other factors that do not lead to the correct model of the subject.

Then, if you read a few more books or talk to several more people, you will see some of the biases that influenced your model-building. However, this also will not be enough. When picking the books to read or people to talk to, your sample will be either too narrow (i.e. people you talk to will be very similar to each other) or too broad (i.e. their experiences will be so discordant that it will too difficult for you to be able to cross-reference them and piece them into a coherent picture).

When I talked to the first 5 people about how life sciences research works, I had a perfect picture in my mind, fully in accordance with all the rumors about academia’s sclerosis, risk-aversion, and short-termism, I’d been hearing for the last few years.

When I talked to 10 more people, I realized that something was off, but couldn’t quite piece it all together.

In the end, I ended up interviewing more than 60 people involved with life sciences research – mostly grad students, postdocs, and PIs (principal investigators, heads of laboratories), but also, for example, with people from philanthropic organizations and venture funds investing in life sciences.

You could view this essay as a culmination of a year of my pursuit of applied progress studies (a). I feel like I’m finally able to converge on something, even though this something seems to be somewhat different from what almost any single person I talked to has as their model of what’s going on in the field.

[Life] science is not slowing down

It’s tempting to look at the negative side of academia. Bureaucratization, seeming risk-aversion, loss of freedom… and conclude that therefore science is working poorly. This is a mistake. Yes, funding agencies are risk-averse; yes, academia now selects for things you probably don’t want it to select for, like conformity and high conscientiousness; yes, an average scientist is not in academia for the love of science (and maybe the productivity of an average scientist is decreasing I would argue that comparing an average scientist today and in 1973 is a bit like comparing an average high-school drop-out today and in 1973. (a) ).

However, all of this does not mean that science is stagnating or even that it is slowing down. The pace of discovery in all areas of biology I looked at is astounding. And not only the pace is astounding, but many researchers are indeed working on the highest expected value projects they have with little restrictions and are taking in the smartest people they can find and throwing money at them (the way it works in biology is that PIs fundraise and manage/mentor, while graduate students and postdocs do the work).

If you look at Harvard or MIT or Stanford or many other universities – they are all filled with amazing researchers working on great budgets, truly passionate about discovery and invention. HHMI (a) is actually giving people unrestricted grants; CZI (a) is funding people specifically to work on software (which is usually hard to get funded) and to work on their riskiest, usually unfundable ideas; and many other foundations are trying to correct the inefficiencies they see in resource allocation.

I think that the perception of stagnation in science – and in biology specifically – is basically fake news, driven by technological hedonic treadmill and nostalgia. We rapidly adapt to technological advances – however big they are – and we always idealize the past – however terrible it was.

I mean – we can just go to Wikipedia’s 2018 in science (a) and see how much progress we made last year:

Doesn’t seem like much? Here’s the kicker: this is not 2018. This is January 2018.

If you actually go and look at the major discoveries made in any single year in the first half of the 20th century and compare it to 2018, you’ll probably conclude that the pace of scientific progress is only getting faster. A potential concern here is that if the amount of resources spent on science has increased 100x but the pace of progress has remained constant or only slightly increased, then the marginal returns of doing science have decreased dramatically. I am undecided on this argument.

All in all, after a year of studying how life science research actually works and progresses, I’m way more optimistic about it.

Nothing works the way you would naively think it works (for better and for worse): 3 Examples

1. “NIH’s risk-aversion makes it very hard to fund innovative science”

Observation: NIH is risk-averse to the point that even when it explicitly asks for radical proposals Open Philanthropy evaluating NIH Director’s Transformative Research Award proposals (a): ‘we thought many of the proposals we reviewed were similar to proposals in more typical RFPs in terms of their novelty and potential impact. In other words, we considered many of the submitted proposals to be a bit on the conventional side. This surprised us given the “transformative” premise and focus of the TRA program. We speculate that this may be due to the constraints within which applicants feel they must work to get through panel reviews.’ , nobody believes it will actually fund anything radical, so people don’t even bother applying with high-risk research

Naive conclusion: it’s impossible to get high-risk projects funded by NIH, so scientists end up working on incremental science

Reality: NIH doesn’t force scientists to adhere 100% to their submitted plans, so what ends up happening is that scientists apply for a grant with a project for which they have supporting preliminary data (so that NIH doesn’t consider it high-risk) and spend most of the funds on that project. The left-over funds then can produce preliminary data for their high-risk ideas, making them appear less risky and more easily fundable by the NIH in the future. Sometimes this takes a more extreme form where researchers actually promise NIH good results from experiments they already performed but have not publicized yet.

So, in reality, distortion is still present because you have to figure out rather arcane ways to get anything interesting funded, but the amount of distortion is way less than we would naively expect.

2. “NIH’s biases make it very hard to fund methodological research”

Observation: NIH doesn’t like to fund purely methodological studies (e.g. development of better software)

Naive conclusion: it’s impossible to get funded for methods development by NIH

Reality: you can dress up methods development for NIH, e.g. by providing a concrete biological goal for which insufficient methods are the bottleneck and show NIH that you will be able to achieve concrete progress on something that matters to them using your better methods

Again, yes, doing this introduces obvious frictions and inefficiencies and skews what scientists work on towards things that are easy to dress up for NIH, rather than things that they believe are most important. But the amount of frictions and inefficiencies is way less than we would naively expect.

3. “NIH severely underfunds young researchers”

WEAKNESSES: “This R01 proposal would appear to be Dr. Greene’s first NIH grant, as per the biosketch." (a)

Observation:

the median age of first-time recipients of R01 grants, the most common and sought-after form of N.I.H. funding, is 42, while the median age of all recipients is 52. More people over 65 are funded with research grants than those under age 35. (NYT (a))

Naive conclusion: young scientists lack the resources to pursue their research. NIH should allocate more funds to investigators under age 35 and 40.

Reality: Many extremely capable people end up working in well-funded labs whose PIs are pretty hands-off and who allow their grad students and postdocs to work on whatever they want and not really think about money. In statistics, this registers as “more money than ever goes to older/famous PIs” and “young investigators can’t start their independent careers”. On the ground this means that until age 35 (i.e. when your creativity is the highest) you are isolated from management and fundraising (and endless administrative responsibilities bestowed on any tenure-track professor) and can 100% focus on doing science and publishing papers, while getting mentoring from your senior PI and while being helped by all the infrastructure established labs have (administrative assistants, lab technicians, etc.).

One could make an argument that due to all of these factors, the present funding structure is beneficial to production of original research by those most capable of it, and that letting more young people set up their labs will mostly result in more people attending useless faculty meetings and more people writing grants, instead of doing research.

When you see “X’s [where X is a famous scientist] lab discovers Y”, chances are it was Z – X’s grad student or a postdoc – who came up with the idea, did all the work (with help of other students), and wrote the paper. X’s contribution was probably in providing the research environment in which all of these were possible, money, and talking to Z every other week about the progress with the idea. I’m not saying that X’s contribution is not important – it usually is – but press releases focused on the PI miss the fact that money is not actually spent directly by them, systematically biasing our perception of how funds are allocated.

In the end, yes, distortion is there. It is indeed difficult to become independent and establish your own personal research agenda if you’re young, but the end negative effect on science done by young researchers is way less than we would naively expect.

Two side points

Also see How policies designed to improve academia are actually messing it up (a)

If you’re smart and driven, you’ll find your way in

There are many labs in biology that have good funding and open and risk-loving PIs. This means that if you’re very smart, ambitious, and driven, you’ll likely be able to get in and work on cool stuff, even with a questionable background, although you might have to spend a year or so working as a Research Assistant to prove your worth. PIs are usually pretty open to getting Research Assistants and are very open to getting thoughtful personalized cold emails (this applies to scientists you would consider famous as well).

I’m fairly confident that if someone capable of producing Nobel-level research decided to do a PhD in biology or neuroscience, they would be able to enter great programs with great advisers and funding in less than 2 years.

The problem here is that almost everybody for whom understanding the stuff I wrote above would be useful is misinformed. People outside of biology generally think that doing a PhD means spending 6 years at the bench performing your advisor’s experiments and is only possible with perfect undergrad GPA, not realizing that neither of these are true if you’re truly capable, and as a result form their decision to avoid grad school on misinformation.

Nobody cares if you’re a genius

This may sound like I’m contradicting the previous section, but note that there I only wrote “get in”, not “stay in”.

In order to get in, you only have to convince one PI that you’re worth taking a small chance on. In order to stay in, you have to convince many professors, many study sections who will assess your grant applications, various charitable foundations, hiring committee at the university, tenure panel, and so on, and so on.

Every one of these entities will be less open and less forgiving than the PI who might decide to take you in.

I know [of] many brilliant researchers.

A great many skills are required of a successful scientist in academia (a). This is what Eric Weinstein means when he talks about the perils of pursuit of excellence (a):

So genius and excellence are both worthwhile but they are distinct modalities, and not recognizing that it is a serious problem to take somebody in the genius idiom and to push them into a different idiom, which is to reduce their variance, is going to be very destructive and it’s going to keep us from founding the industries that will allow us to change paradigms and move forward.

Almost all biologists are solo founders. This is probably suboptimal

Many of the listed above ailments would be solved if the researcher in question had a champion or a co-founder who would complement them at whatever they’re bad at.

In Jessica Livingston’s (co-founder of Y Combinator) Founders at Work (a), Max Levchin (PayPal co-founder) says:

Try to have a good cofounder. I think it’s all about people, and, if you are doing it completely alone, it’s really hard. It’s not impossible, in particular if you are a loner and introverted type, but it’s still really hard … I had run a company before PayPal, alone, and I thought it was fine. I could deal with it. But, you only can count on energy sources and support sources from yourself. There’s really no one else who you can go to and say, “Hey, this thing is going to fall apart any minute now. What the hell are we going to do?”

The thing that kept us going in the early days was the fact that Peter and I always knew that both of us would not be in a funk together.

YC itself is famous for its preference for teams rather than individuals. There should be a CEO (visionary, salesman, manager) and a CTO (builder and designer). In light of this, it’s puzzling why universities seem to virtually never hire two people to run a lab jointly. There’s a single PI who has to both be excellent at being the CEO and at being the CTO and who moreover has to run the lab essentially alone – everybody else in the lab will be an employee with negligible amount of responsibility for the long-term future of the lab.

This is very weird and reminds me of Feynman and Dyson. Freeman Dyson:

[Feynman] was working on these problems of quantum electrodynamics, and he had done a great deal which was very beautiful, but which nobody else understood. And he loved to talk, and I loved to listen. So within six months I had pretty well mastered his language. And within one year, I actually was able to translate his ideas into mathematics, so it became more accessible to the world. And as a result, I became famous, but it all happened within about six months.

How many scientists never reach their potential because they fail to find their co-founder and because the way academia is structured today seems to be very unwelcoming of such arrangements?

There’s insufficient space for people who just want to be researchers and not managers

Many people who always wanted to become scientists do not pursue or leave academia because they see how PIs work and think that they do not want to just manage people and fundraise/write grants. This is a great tragedy. Very few labs have permanent Research Scientist positions and for some reason there’s a path “PhD–>Postdoc–>PI” that is almost impossible to avoid (there are institutions that experiment with permanent pure researcher positions (Broad, CZI, Wyss, Calico) but there seem to be very few of them).

There was a famous problem in large companies where if you were a really good software engineer, you’d have to become a manager to continue to advance your career but I hear that these days, companies have introduced “Individual Contributors” where you can grow while still being primarily a technical contributor. This is something that academia still has to figure out.

My intuition is that the biggest unnecessary attrition from academia happens due to the absence of an established “pure researcher” career track.

Peer review is a disaster

In The Double Helix (a), James Watson (co-discoverer of the structure of DNA along with Francis Crick) writes:

[I]t would take two or three years to set up a new research group primarily devoted to using X rays to look at the DNA structure. Moreover, such a decision would create an awkward personal situation. At this time molecular work on DNA in England was, for all practical purposes, the personal property of Maurice Wilkins, a bachelor who worked in London at King’s College.

Several people I talked to describe a similar dynamic and tell me that there are niches dominated by a particular research group that guards that niche almost as its fiefdom. It takes a lot of courage and determination to move in and try to upend the dominant methodologies and research directions of the field. See The maddening saga of how an Alzheimer’s ‘cabal’ thwarted progress toward a cure for decades (a) for a concrete example.

Peer review exacerbates this dynamics. Peer reviewers in your field are your competitors, who have not themselves solved the problem you claim to be able to solve. They have both personal and professional interest (especially so if funding is limited) in giving low scores to grant applications of competing teams and to recommend rejection of their journal submissions. Further, since they’re experts in the grant application topic, while rejecting your paper or grant application, they can lift your research ideas and then pursue them themselves. This happens more frequently than you would expect (a).

Further still, committees reviewing grants have all sorts of weird interpersonal dynamics that make funding anything unconventional even more difficult than it would be in their absence, because, while being on a committee people are usually averse to be publicly approving towards anything that seems weird.

An anonymous redditor writes (a):

In my field virtually 100% of the papers in “top” journals come from the same 5-10 senior authors, and they can just about get away with murder.

Almost everyone I talked to shared a broadly negative sentiment about peer review.

Nobody agrees on whether big labs are good or bad

Big labs tend to have

On the other hand, scientists are not at all always good managers. In a big lab, students frequently receive insufficient mentorship from the PI and have to be very independent and look for outside mentors and support (this is worsened by the fact that few people care to hire middle managers for big labs). The result is that often people are left to their own devices and are sometimes lost in research, meaning that strictly speaking the amount of waste is increased.

People who have big labs continue to fight for more funding and feel that what they create is unique and must be protected, If you have a big lab, you probably have a lot of people trying to get in, meaning that you feel there’s always plenty of opportunities to grow. while people who are barely scraping by suggest strict limits on lab size (and other ways to make distribution of money more equitable) and point out that sometimes big established labs continue to get funded almost by inertia. People have very strong feelings about this.

Many scientists seem to measure their professional success by the number and the size of their grants and do everything they can to maximize their funding and my impression is that simply giving them much more money (like HHMI does) won’t change how they spend their time too much – they’ll just continue writing grants.

Senior scientists are bound by their students' incentives

As I noted before, in biology, PIs mostly manage people – all the real work is done by grad students and postdocs. Grad students and postdocs have to graduate and to look for faculty jobs, meaning that however long the PI’s horizon is, the people who do the work will be bound by their short-term need to publish. And given that grad students and postdocs typically make about 25-50% of their market rate, however passionate they are about science, they want to graduate and move on as fast as possible.

(This also applies to those who have fellowships, as they always have a pre-determined maximum length, usually 3-5 years)

Universities seem to maximize their profits, with good research being a side-effect

When I started investigating how biology works, I believed that universities spend their own money to run labs and enable research, because this is how it works in economics, where I come from. I soon learned this is not the case.

In biology, it’s the scientist who brings money to the university. University provides affiliation/credibility, space, and administrative assistance. But then, to pay for its costs it takes a cut (“overhead”) from every NIH grant the scientist gets. University usually gets about 1/3 to 1/2 of the size of the grant.

Because university’s cut is linear but actual expenses are sublinear, pretty soon “overhead” turns into “profit” for the university.

As a result, in hiring decisions, the amount of money the researcher is able to bring sometimes effectively becomes the measure of quality of research.

One of the funnier side-effects of this is that now universities have not only reputational incentive to cover up fraudulent or questionable research by scientists but also a financial incentive to protect scientists who bring in a lot of money. Analogous to how journals these days are thinking much harder before retracting well-cited but “questionable” papers – their impact factor depends on them, after all.

The anonymous redditor from section about peer review further notes (a):

In my department, the main standards [for getting tenure] are to a) win two RO1 research grants, then b) renew them both. If you’re bringing in hundreds of thousands in indirects every year, the tenure committee doesn’t much care whether you publish in Nature or Diabetes.

Also see The Uncharity of College: The Big Business Nobody Understands. (a)

Large parts of modern scientific literature are wrong

I am confident that somewhere between 10% and 50% of papers published in good journals are wrong, meaningless or fraudulent. This figure is based on my expertise in parts of economics, psychology, neuroscience, and genetics. Unless you invested significant time studying the subject matter, you will have very bad intuition about which papers are good and which ones are bad. See Replication crisis (a).

Further, my guess is that papers published in top journals (Cell, Nature, Science) are more wrong on average than papers from top journals from specific fields of science. It seems that CNS chase hype and have rather lax standards on the methodology. Plus, chasing hype and being interested only in “large update” papers means that we would expect more of these papers to be wrong just because of regression to the mean.

People I talked to say that the papers CNS publish from their fields of expertise would never get published in more specialized journals and that CNS casually disregard reviewers' criticisms of methodology.

This matches my observations, where sometimes catchy but bad papers that would be rejected on the grounds of poor methodology from a field journal get published in CNS.

A paper (a) published in Frontiers in Human Neuroscience finds that:

methodological quality and, consequently, reliability of published research works in several fields may be decreasing with increasing journal rank

You should reflect on whether a typical study you hear about is selected more on the ability to propagate itself across researchers, news, and social media or on sound methodology.

Sometimes, virtually entire swaths of the literature turn out to be meaningless. Recent examples from life sciences are candidate gene studies (a) and small-n cognitive neuroscience (a). A recent example not from life science is small-n social psychology (a).

How is it possible that entire research fields turn out to be meaningless? Some people (a) blame the incentive structure of modern academia. I disagree.

Any area of study – however meaningless – can produce a body of established facts, practices, and experts. Newton studied (a) theology, alchemy, and physics. 25% of people in the US believe (a) in astrology and some of them go to astrology experts.

If one can convince other people to give them money to study something, they will gain a financial motive to tell everybody else about the importance of the subject and become an “expert” on it. If there’s a group of “experts” they will all amplify each other’s voices and will try to legitimatize whatever it is that they study. As a different redditor astutely notes (a):

[N]early every expert relies on the valuation of their expertise for money: therefore every expert has a strong case to oversell their expertise/the state of knowledge in their discipline

(as an aside – the interaction between personal rivalries in the field and between the common interest of its practitionaires to inflate its perceived importance ought to produce some truly fascinating interpersonal dynamics)

Finally, some papers are just fraudulent. Elisabeth Bik looked (a) at more than 20,000 papers that contained a particular type of easily examinable picture published in good biology journals between 1995 and 2014 and found that 3.8% – about 1/25 – contained “problematic figures” and at least half of these had evidence of deliberate manipulation.

Here’s what one of the VCs investing in life sciences I talked to told me:

If you see a picture, this is the best picture the authors had. If you see a statistic, there’s probably something wrong with it. 5% of the companies we do due diligence on – and we only do due diligence on excellent companies – edit their pictures.

Interlude: "if this is true, how can you be optimistic and write that science is not slowing down?"

It may seem like the section above opposes my previous “science is not slowing down” optimism. It does not. There were always large parts of science that were wrong or meaningless, so the present situation is not worse than the past in this respect.

Even if 10-50% of the studies I mentioned in the first section turn out to be wrong, the pace of correct and important results is still astounding.

Raising money is very difficult even for famous scientists

Collectively, scientists I talked to interacted with more than 20 billionaires but I’m not aware of any of them raising any significant amount of money as a result. I’m very surprised by this because if I were a billionaire, I would probably fund at least one Boyden-like scientist in perpetuity and it seems that there are many more billionaires interested in science than there are Boydens.

Perhaps this is the result of the widespread (and not entirely unfounded) perception that science is broken and that scientists are effectively conning the society into giving them money, resulting in people believing that

  1. an average scientist is not worth giving more money to
  2. while famous scientists like Church and Boyden don’t need financial support (even if in reality, the reason they’re well-funded is because they spend a ton of time fundraising and not having to write grants every year would greatly increase their research output)

In an interview to Tyler Cowen, Boyden says (a):

For me, it became personal because when we proposed this expansion microscopy technology, where we blow up brain specimens and other specimens a hundred times in volume to map them, people thought it was nonsense. People were skeptical. People hated it. Nine out of my first ten grants that I wrote on it were rejected.

If it weren’t for the Open Philanthropy Project that heard about our struggles to get this project funded — through, again, a set of links that were, as far as I can tell, largely luck driven — maybe our group would have been out of business. But they came through and gave us a major gift, and that kept us going.

Conclusion

If you take away one thing from this essay, let it be this: academia has a lot of problems but it’s less broken than it seems from the outside.

Everything is much more complicated than everyone thinks and it’s very easy to try to improve something only to have second-order effects mess everything up. Because of all of this, I’m wary of proposing any drastic changes to how grants are distributed, how researchers are trained, and so on.

Instead, I have several smaller-scale recommendations:

Finally, if you have upwards of a $50,000 available for charitable giving, consider reaching out to a scientist you like, and seeing if you can help them financially. They will be grateful for support and will probably be happy to grab an occasional dinner with you and tell you all about the latest developments in their field.

Previously, I wrote Reviving Patronage and Revolutionary Industrial Research

If you’re interested in making life sciences work better, get in touch – alexey@guzey.com. You can also subscribe to receive monthly updates from me here.

Acknowledgements

I would like to thank (in reverse alphabetic order) Andrew Zuckerman, Guy Wilson, Brian Timar, Sasha Targ, Panagiotis Symvoulidis, Ben Reinhardt, Nabeel Qureshi, Manjari Narayan, Adam Marblestone, Anastasia Kuptsova, Bonnie Kavoussi, William Eden.

Thanks to Emergent Ventures (a) for financial support.

Citation

In academic work, please cite this essay as:

Guzey, A. How Life Sciences Actually Work: Findings of a Year-Long Investigation. Guzey.com. 2019 August. Available from https://guzey.com/how-life-sciences-actually-work/.

Or download a BibTeX file here.

Appendix

What I am saying and what I am not saying in this essay

Corrections

A previous version of this essay quoted Cosma Shalizi on peer review (a). I didn’t read his post carefully and ended up taking his quote out of context and misrepresenting his view (he believes that peer review is good). Thanks to Daniel Weissman for pointing this out (a).

Risk-aversion might be overstated

I think that it might be the case that the degree of risk-aversion in academia is overstated. My research suggests that those who want to pursue high-risk research usually find ways to do it, while those who don’t want to, blame the system.

Grad students and postdocs in particular are usually able to pursue 2-3 projects simultaneously and can typically work on 1 project with high chance of success and 1-2 projects with low chance of success. It does definitely seem that you cannot pursue one project full-time and have this project be high-risk, because, if it fails completely, you’ll have no publications and, without publications, you’ll just not be able to get a job and stay in academia.

The alternative explanation of my observations about risk-taking is survivorship bias: I only see those for whom taking risks panned out. I can’t figure out how much survivorship bias is present here, which is why I do not make any firm conclusions about the degree of risk-aversion in academia.

Studying people and animals is really hard

My IRB Nightmare (a) by Scott Alexander:

I feel like a study that realistically could have been done by one person in a couple of hours got dragged out into hundreds of hours of paperwork hell for an entire team of miserable doctors. I think its scientific integrity was screwed up by stupid requirements like the one about breaking blinding, and the patients involved were put through unnecessary trouble by being forced to sign endless consent forms screaming to them about nonexistent risks.

I feel like I was dragged almost to the point of needing to be in a psychiatric hospital myself, while my colleagues who just used the bipolar screening test – without making the mistake of trying to check if it works – continue to do so without anybody questioning them or giving them the slightest bit of aggravation.

I feel like some scientists do amazingly crappy studies that couldn’t possibly prove anything, but get away with it because they have a well-funded team of clerks and secretaries who handle the paperwork for them. And that I, who was trying to do everything right, got ground down with so many pointless security-theater-style regulations that I’m never going to be able to do the research I would need to show they’re wrong.

See discussion (a) on Andrew Gelman’s blog + IRB nightmares (a). Several people I talked to say very similar things about IRB.

More funding ideas

Funding undergrads

I know several cases of brilliant undergrads who did not pursue scientific career because of external circumstances but for whom giving, say $50k would allow them to deal with these circumstances, fund exploring and pursuing science for a while and let them pursue the academic track.

High-quality clinical trials of generic medicines, nootropics, etc.

See An Iron Curtain Has Descended Upon Psychopharmacology (a) by Scott Alexander

Large-scale interventions

Mostly just fun thought experiments:

A section on medicine

Alzheimer’s

A Brief Note About Alzheimer’s (a)

I had no real expectations that aducanumab would work, despite all the attempts at positive spin over the years, and by golly, it doesn’t work. That doesn’t make me a prophet – I think any objective observer would have to have come to the same conclusion. Biogen and Eisai put themselves into this situation, although they definitely made it worse by trying to pretend that things were going differently than they have for every single other amyloid antibody program ever. They have all failed. One after another, again and again.

A lab tech/manager’s email about their experience in response to this essay

I just read your article on how research in life sciences work. You are spot-on in your treatment of the dearth of support research team members that are not students or post-docs. I have been a lab technician/lab manager for the last 13 years. My current job started 8 years ago when my boss told me he had 6 months of guaranteed funding. I worked for him full-time for 4 years, my salary cobbled together from a half-dozen grants over that time. Over time, my job went from being the lab expert in organic chemistry extraction techniques to administrative assistant who happened to be well-versed in research.

I have argued for many years about the need for PIs and funding agencies to include in their proposals and grants salary items for “supertechs.” Alternatively, faculty members could negotiate for at least partial hard-money funding for a supertech when hired by their institutions. It is very easy to get funding for students and post-docs, but virtually impossible to get funding for a full-time technician or researcher that can oversee or assist or manage research for multiple projects. In many cases, the students in charge of their projects have their own systems of organization and notation, making it incredibly difficult to parse anything out of their research after they leave. I have been able to salvage many lines of research and collaboration purely on the basis of the length of my tenure and my policy of never deleting any work-related email.

After I was unable to work full-time, my boss quickly discovered he needed a replacement or the productivity of the lab would be sharply curtailed. He was also in an awkward position where the school would not allow a student or part-time employee to make purchases, which meant he spent several hours a week manning supply requests and making purchases. It was definitely not an optimal use of his time.

Had there been a career path towards “Scientist Researcher” as you mention in your article, I most certainly would have taken it. As it is, I have worked for a half-dozen PIs and never once envied their job. I am quite good at implementation of research, but terrible at the networking and idea-generating aspects of a good PI. However, while my skills are undoubtedly valuable to a research lab, it is incredibly difficult for someone like me to find a stable job because of the funding issues and lack of recognition of the value of a supertech position. I think an alternate career path of Master’s/PhD–>lab tech–>lab manager would provide an outlet for talented research-minded students who may not have the aptitude set to make a productive PI, but could contribute significantly to research. It could alleviate the glut of PhDs looking for tenure-track positions as well as make labs more efficient and productive.

Peer review

“So who decides which people get the grants? It’s their peers, who are all working on exactly the same things that everybody is working on. And if you submit a proposal that says “I’m going to go off and work on this crazy idea, and maybe there’s a one in a thousand chance that I’ll discover some of the secrets of the universe, and a 99.9% chance that I’ll come up with bubkes,” you get turned down." (a)

“A researcher made up whole trials, his co-authors at the very least were complicit by putting their names on his papers without checking anything, peer review failed to spot numerous problems with the studies, journals failed to react to red flags raised after publication” (a)

“Many times, paper rejections are not due to weaknesses of the paper but misunderstandings of the goals. E.g. I often get “provides no mechanistic insight” when I write a normative paper or pure engineering papers. Who would be able to share examples of such reviewer confusions?" (a)

“Is the NIH grant system broken? Shockingly, I have thoughts. Thoughts from both sides of the process, grantee and reviewer. Thoughts from a mid-career scientist. Thoughts as an employer. Thoughts as a mentor. These are MY thoughts, attributable to no one else. Still with me?/1 … “ (a) …And therein lies the problem. With an 8% pay line, all it takes is one reviewer. One reviewer who doesn’t have the right knowledge base. One reviewer who simply isn’t excited about your proposal. One reviewer who says “it’s not translational enough”. /36 (a) “writing an A0 makes me smarter. Writing an A1 makes me especially smarter. Taking smart criticism into account and finding solutions to address them is almost as good as having a great collaborator. /9” (a)

“After a certain point, you just keep submitting the same damn grant, maybe shopping study sections, until its funded. My record is 9 submissions. Just had another that took 5." (a)

Thinking again this morning that so much scientific writing is bad because it is written for hostile reviewers rather than for interested readers. Not sure there is a way out of this trap. (a)

“I can live with papers being rejected. But not when the main reason for being rejected is that did not find what we had expected. Really rather demotivating when I’m trying to train my PhD students the values of honest reporting, preregistration etc…" (a)

In the current system, no responses (regardless of their length, veracity and clarity) can save a paper from a senior reviewer determined to sink it. (a)

“On clarity. Work that is properly presented is often mistaken to be ‘trivial’, when instead it was simply easy to digest because it was communicated effectively. Poor writing, obfuscation, and needless complexity are not virtues." (a) (this actually happened to me! The reviewer of my Bachelor’s thesis wrote that the paper is well written but all the conclusions are trivial. Fortunately, I was able to convince the panel during my oral defense that this is not the case.)

“My three most cited papers were either rejected under peer review or accepted with no peer review." (a)

Administrators maximize legibility

“Recently learned that faculty applications for most Chinese uni’s are screened solely based on impact factors." (a)

In US it can be scary how much it is about business. Chairs of departments are measured in metrics such as how much funding $ /m^2 they can get. So they need to hire faculty that can bring $$$. Candidate with IF alone will not guarantee that though. (a)

“I’ve worked places where promotions and hiring were based exclusively on the types of grants you have - if they don’t pay overhead rates >40%, they don’t count towards your tenure / promotion / resource allocation package. W/ chasing IF and 52+% overhead, few options for new PIs." (a)

Also see Seeing Like A State (a)

Miscellanea

“while switching topics may be easy in theory, certain fields are jealously guarded by more established Pis” (a)

“A European Research Council report suggests 79% of projects they fund “achieved a major scientific advance”, & only 1% make no contribution. Also, that they fund mostly “high risk” work” (a)

Chargaff (a famous biochemist) about meeting future discoverers of DNA structure:

“They impressed me by their extreme ignorance. Watson makes that clear! I never met two men who knew so little—and aspired to so much. They were going about it in a roguish, jocular manner, very bright young people who didn’t know much. They didn’t seem to know of my work, not even the structure and chemistry of the purines and pyrimidines. But they wanted to construct a helix’ a polynucleotide to rival Pauling’s alpha helix. They talked so much about ‘pitch’ that I wrote down afterwards, ‘Two pitchmen in search of a helix’. I explained to them the observations on the regularities in DNA, and told them adenine is complementary to thymine, guanine to cytosine, purines to pyrimidines; that any structure would have to take account of our complementary ratios. It struck me as a typical British intellectual atmosphere, little work and lots of talk.”

“Once upon a time I ran across an interesting RFA from the NIH that seemed highly targeted for one particular lab. Oh, don’t get me wrong, many of them do seem this way. But this one was particularly…..specific. The funding was generous, we’re talking BigMech territory here." (a)

“I suspect that peer review actually causes rather than mitigates many of the “troubling trends” recently identified by @zacharylipton and Jacob Steinhardt … It’s very common for reviewers to read empirical papers and complain that there is no “theory”. But they don’t ask for theory to address any specific question. I think they are just looking for an easy reason to reject—-they skim and don’t see scary equations. … Reviewers seem to hate “science” papers, but it’s possible to sneak science in the door if add some token amount of new method engineering” (a)

“I spoke recently to a member of the panel that awarded me my first research grant on quantum computers (1985). He said it was a close-run thing. I asked if I’d have got it under today’s criteria. He said: no chance; basically I could tick none of the boxes.” (a)

“Ioannidis and Trepanowski say that without RCTs, nutrition science can never credibly answer important questions about nutrition. That Harvard department’s response is that RCTs are essentially impossible because you cannot get people to meaningfully change their diets over long periods of time. It seems clear to me that both sides are correct." (a)

90% of all claims about the problems with medical studies are wrong (a)

Beware Mass-Produced Medical Recommendations (a)

Endless citations to already-retracted articles (a)

“It breaks my heart to see a postdoc’s NRSA grant rejection summary statement citing mixed undergraduate grades from a decade ago as a weakness. Reviewers: you have a lot of power to empower or discourage young people, use it wisely." (a)

Early Career Funding Opportunities (a) (xlsx a)

'“Everything in my career has led up to this. All of that hard work in undergrad, grad school, my postdoc…that has prepared me to do this job.” This is what I was thinking as I started my faculty job. I was so wrong.(1/n)' (a)

“Do other international PhD students in the US feel like their careers are negatively impacted by the lack of fellowships that we can apply to/are even eligible for?" (a)

“Putting together a talk, and once again floored by the fact the average cancer drug coming to the US market improves survival just 2.1 months And that is in the minority of times we have actually measured survival, rather than surrogates like RR and PFS!! (Fojo, et al.)" (a)

Efficiency of grantmaking, is peer-review legit, etc. (can’t archive notion…)

“At a top School of #PublicHealth I once saw a presentation where after consistently null findings were shown the speaker said “but in my heart I know it works,” the audience nodded in agreement, and now years later the NIH is still funding this ineffective intervention." (a)

“The findings of this study are wild: “Almost every trial” pubbed in 5 high-impact med journals over a 6-week period had engaged in outcome switching. The authors sent correction letters to the journals & most were rejected” (a)

“Five years ago, I reported 800 scientific papers with image duplications to journals. As of today, only one third of these have been corrected/retracted. Journals/institutions are very reluctant to take action." (a)

“Tenured profs wouldn’t be given nearly the freedom and discretion that they have if any substantial fraction of them actually used it to pursue lines of research that disturbed & disrupted powerful others.

“Hot take: I used to be suspicious of industry labs due to profit motivation. Now I feel the opposite: for many types of commercial products, objectively-demonstrated performance is everything. That can add a layer of accountability which we sorely miss in academia."

Specific examples of important, recent medical progress

References

Bik EM, Casadevall A, Fang FC. The prevalence of inappropriate image duplication in biomedical research publications. MBio. 2016 Jul 6;7(3).