How to read a nutrition paper (Matt Laye & Mike Haney)

The world of research papers may not excite the common individual, but they are critical to the development of public policy. In the realm of personal health, they can also provide important guidance on maximizing fitness and wellness. In this episode, Levels Head of Content, Mike Haney, sat down with Dr. Matt Laye to discuss what it means to actually digest a scientific article. They chatted through a recent article Matt wrote about the 10 most important things to keep in mind when you are reading a scientific study firsthand.

8:19 – The importance of scientific literacy

An ability to read and interpret scientific studies is an important skill when it comes to improving our public policies and civilization as a whole.

I came across a quote in a paper that I was reading that scientific literacy is fundamental to both the functioning of modern civilization and its advancement. And I just think that having the public be able to have a level of scientific literacy is absolutely critical to how public policy gets determined. And while this stuff is really hard to read and there is a case that maybe we should lean on experts a little bit, I think that at least having an idea of, okay, what is the process that is going into doing these studies? How do we actually look at it? And given that nutritional scientist has had this dip in public trust given all the swinging back and forth between this is good for us, this is bad for us, this diet is best, that diet is best, I think it’s an area of science that really needs to have individuals who are not involved in the science to be able to understand and read it and be able to interpret it on their own.

11:29 – Determine the relevance of a study

Not every study is applicable to every person. Teasing out what is relevant to who is a major challenge.

The challenge for public policymakers is to somehow make some recommendations that are based on the entirety of the literature that applies to the entirety of the population, which is really difficult when you have a lot of things that have very high degrees of nuance are very dependent upon the situation, dependent upon the study. And being able to pick out those little variations can maybe help give you a little bit of an insight into, well, how much does this apply to me or how much does this apply to the general public or how much faith should I be putting on in this specific article. And not faith as in was it done well or not, but just faith as in how relevant is it to me, how relevant is it to the population.

13:00 – Identify the question the study is exploring

One of your first goals as a reader is to look at what specific question the study is designed to answer.

Looking at the specific question is really important. And one way that I actually didn’t lay out in the article, but I think is useful is studies when they’re designed, they’re often designed to answer a specific question, a specific aim. And we call that the primary focus of the study. So the population they select, the intervention they select, the outcomes that they select is all based on that one question. And that will be the strongest evidence will be to try to answer that question, but there could be secondary questions or secondary analysis in that paper that might not answer that specific question, but a slightly different question that’s related or something to that degree. And so when you look at a paper and you see, okay, was this study actually designed to answer this specific question?

16:25 – Consider if the study is blind

Nutrition studies are unique in that blinding is difficult; diet cannot be kept secret from study subjects, and randomized control trials are tricky.

Nutrition studies are just always going to have an issue with blinding, because the subjects are going to know what they’re eating most of the case. So you’re not going to be blinded at least from the subject standpoint. And when you do these randomized control trials, so this is trying to put one group doing one diet and one group doing another diet. And so one example that I’ve read about is the women’s health initiative, which was comparing a low fact to a regular diet versus a low-fat diet. And they were just told to eat these things. And they wanted to make this study long study so that they could try to find an effect. So it was, I think over a year, for years, actually, they’re supposed to follow the diet. And so they randomized them, they have them into these two different groups and then they look at the data at the end, and it turns out that they end up both eating the same diet. And so it’s really hard to force people to eat different diets. That’s one of the issues with these randomized control trials is that changing people’s entire dietary habits for extended periods of time, which is necessary to really get a question of does this diet improve this chronic metabolic disease.

19:17 – Controlled versus observational studies

There are two main types of studies, and both are valuable in different ways.

I look at those randomized controlled trials as often the best things you can measure is not the main outcome that you’re interested in, which is the disease. So you’re looking at some biomarker of that disease, something that we know is correlated to the disease process. And that combined with an observational study that maybe can look at the actual prevalence of the disease, gives us a guiding light towards the answer. And so they can work together. And I think you’ve described it really nicely of how they answer different questions. One maybe is a little bit it more of the physiology and basic mechanisms and the randomized control trials. And then the longevity, the observational studies, the longitudinal data can then get at, okay, what’s the prevalence of these diseases happening? And when you start to get consistent data among the many different types of study designs, then that gives us a much stronger evidence-based to make recommendations upon.

22:21 – Take questionnaire data with a grain of salt

Most data gathered from questionnaires is suspect. People typically ballpark their answers, and they can in fact be wildly inaccurate.

There was a paper that I always go over in my nutrition class about how the NHANES data, which is this big data set that is basically a bunch of surveys uses that data to make dietary record recommendations when about 60, I think it’s like 67% of the respondents would not even be consuming enough calories to live. So the records are very off. They’re very off. So you can get around that a little bit by collecting a ton of people’s data and hoping that the averages lead you to somewhere that’s closer to the truth, but they are just inherently not a great tool for collecting nutritional data. Now, I think technology can solve some of these issues. Having these computers with us all the time with a camera attached, being able to take pictures of food is going to change that a little bit. I think it’s going to make the observational studies once we can get the AI and machine learning down to actually pull out the nutrient information from those meals, I think will help. But right now, those old school questionnaires, my students take them and they’re like, “What? This is how we learned about nutritional sciences? I don’t remember the last two months or month how many times I’ve had an apple or nonfat ice cream or something like that.”

28:43 – Consider funding sources

Most scientific research is not funded by the government, which means you must consider the interests of the groups providing the financial backing.

A lot of this research doesn’t necessarily get funding from the federal government. And so researchers who are interested in really doing the research have to find other funding sources and industries are interested in funding it. And I think from a good spot in general, they’re interested in seeing whether or not knowing that there are specific health-related questions that need to be answered. And I guess I look at it as I’m just more inherently skeptical if I see a funding organization that’s called out in the conflict of interest at the end of the paper. That’s often where you’ll see it, but I think it’s worthwhile looking sometimes at the authors and seeing if they’re funded by organizations elsewhere if you’re questioning it, because I’ve also seen multiple times where it’s not been disclosed. And I think that, that can really erode your trust in a paper or trust in the science that’s being done, and when people aren’t upfront about the biases that they might be having. And I just think that the data is actually pretty strong to say that industry-funded work is much more likely to come out on industry side of things than government-funded work. And so I’m inherently skeptical towards that reason.

34:21 – Take biases into account

Matt describes one of his own biases and how it’s changed over the years. He used to believe that only physical activity mattered, and not diet.

My PhD advisor was fanatical about physical inactivity being the causal effect of so many different chronic diseases. And we’ve written some really massive review papers with 400 or 500 cited articles and then looking at 30 diseases that are caused. So my bias is certainly that being inactive is one of the worst things you can possibly do for yourself. And that was one that I took and I actually took nutrition out of it. And I said, “Okay, it doesn’t matter, it’s all about the inactivity.” And even to the point where Gary Taubes came and talked at our school when I was a graduate student and I was almost infuriated that he was saying diet was the thing that was important. And that’s one of the things that I have had to change my mind on to really recognize that, no, actually, I think these are two maybe equally important. And specifically with weight, I think he was writing about weight loss at the time how exercise was worthless for it. At the time, I was just completely against that idea and I wasn’t necessarily even doing work in that area, it was my own bias from just the experience I was having.

47:34 – The meaning of statistical significance

When results don’t differ by random chance, they are said to have statistical significance.

That word significance is like a quirk of statistics. And what it essentially means is that the difference between say weight loss in group A and weight loss in group B doesn’t look like it was due to just random chance, that there was actually something at play, but it doesn’t tell you anything about the degree of that weight loss, which is really what we’re interested in from the clinical standpoint is how much actual weight loss occurred in group A versus group B. And that’s the clinical significance. So example would be, okay, you have a really big study and you’re able to get a lot of what we call statistical power, so you’re able to detect small differences in measures that are not just due to random chance, but are actually due to something else. And so maybe you have diet A, diet B, weight loss of one pound difference and it’s statistically significant, highly significant, very unlikely due to just the chance of probability, but it’s not really significant from a clinical standpoint. One pound, probably not that big of a deal. And so that’s the key thing is to pay attention to both the statistical and clinical significance.

54:17 – Studies provide a starting point for individual experimentation

Scientific studies can help inspire individuals to begin their own biohacking experiments.

When I think of sciences, okay, even if we have some public policy truth, there’s going to be variation in how individuals respond to that. And so when we read a study and we look at the results, the only way we’ll really know if that works for us is to replicate that study and have one and see if we can replicate it, knowing that we can’t do a controlled version and a non-controlled version of ourselves. And so there’s always a little bit of problems with that. There could be other things that we change inherently when we’re making these interventions on ourselves. But that is how I view these studies and their utility is that, okay, this gives me a place to start experimenting in my own life, regardless of the public policy implications of this work. And they’re not the same things like I write, one is science, the public policy trying to gather all of the evidence and this one study will contribute a little bit to what we know a little bit more, so we’re a little more confident or a little less confident in these are things that we thought we knew. And the other is, okay, this is a basis for me to try something.

Matt Laye (00:06):

Having the public be able to have a level of scientific literacy is absolutely critical to how public policy gets determined. And while this stuff is really hard to read and there is a case that maybe we should lean on experts a little bit, I think that at least having an idea of what is the process that is going on into doing these studies? How do we actually look at it? And given that nutritional sciences has had this dip in public trust given all the swinging back and forth between this is good for us, this is bad for us, this diet is best, that is best, I think it’s an area of science that really needs to have individuals who are not involved in the science to be able to understand and read it and be able to interpret it on their own.

Ben Grynol (00:58):

I’m Ben Grynol, part of the early startup team here at Levels. We’re building tech that helps people to understand their metabolic health, and this is your front row seat to everything we do. This is A Whole New Level.

Ben Grynol (01:24):

You can read for pleasure or you can read for education. And sometimes there’s an overlap in the Venn diagram, the way that you look at academic or maybe more educational information. But when digging into this type of content, there’s really a way a structure of going through the information to see if it actually makes sense, a way of deciphering everything that’s been listed. When it comes to scientific based content, that being things pertaining to metabolic health or health and wellness in general, there’s really a format that you can follow. The key is always questioning things.

Ben Grynol (02:01):

But when it comes to reading articles about research based content, there’s no one better than Dr. Matt Laye. Matt focuses very much on deconstructing research based articles every single day. That’s what he does. And he’s been a big part of what we’ve been doing at Levels in helping to vet some of the blog content that we’ve produce. And when producing science back content, you’re not trying to persuade people to take one position over another. The key is to present the information as is, here are the findings and people can make their own takeaways, come up with their own insight or opinions based on presenting the material as is. That’s what our job is to present the facts, lay them out and let people decide what they want to think about it.

Ben Grynol (02:46):

Now, when it comes to insight, sure, we might have some takeaways, but we never really want to have a POV, that’s a point of view on exactly what to think. It’s like a good Gladwell book, presents both sides of the argument and lets a reader decide what to take away. So Matt Laye sat down with Mike Haney, head of content and our editorial director. And the two of them chatted about what it means to actually read and digest a scientific fact article. And really changed the lens and the way that you start to look at some of this. Here’s where they kick things off.

Mike Haney (03:27):

We don’t like to do a lot of traditional where are you from and what were your parents like as bio set up, but I do think it would be helpful to go over a little bit just what you do and the research that you look at, because I think what’s been so useful about all the articles you’ve been contributing for us is that you are an actual metabolic researcher. We write about your folks all the time, but you are a real one. You do this for a living. So maybe just start by telling us a little bit about the research that you do, some of the current projects you’re working on, what your focus is.

Matt Laye (03:57):

So I’ve had a long real interest in exercise actually. That’s been my main interest in how exercise can prevent many different chronic diseases. And in my research past, I’ve worked in exercise focus labs as well as aging and dietary restriction labs. So a little bit more on the dietary side as well. And about six years ago, I moved to the College of Idaho, which is a small liberal arts school here in the Treasure Valley, near Boise. And my research here has taken a little bit backseat to my main job as a teacher, but the interest that I’ve really developed is how do we tease out in humans the impact of exercise and metabolism.

Matt Laye (04:40):

And so specifically working with a colleague from a previous research position I had, we’ve been interested in, okay, how do you figure out the right time to exercise to minimize postprandial glucose levels or that glucose spike after you eat? And this is important because it’s independent risk factor for cardiovascular disease and other chronic diseases. And so that’s what I’ve been interested in for the last few years and what I’m currently pursuing during my sabbatical year doing mostly research and taking a step out of the classroom for a year, which has been nice.

Mike Haney (05:14):

And you are a former or current ultra runner as well, right?

Matt Laye (05:18):

I would consider myself mostly former in the competitive aspects. I took a bad fall and have had some knee issues since then, but I do coach a number of ultra runners as one of my side hustles as well. And I’m still involved in the sport for sure.

Mike Haney (05:34):

Okay. And your research has spanned both that side of things or at least I know you’ve written a lot about that side of things, how do ultra runners in that extreme endurance athlete fuel and how does their metabolism work, but also those of us who are not ultra running simple things like walking after meals.

Matt Laye (05:51):

Yeah. So from the aspects of high performance nutrition, which looks almost completely opposite of what I would say day to day nutrition should look like and day to day nutrition, timed in relation to physical activity bouts. One of the things I find most interesting is that we have these exercise recommendations for health, but we never really put those exercise recommendations in the context of when we’re eating, which does make a really big difference. And so we have recommendations for diabetics that look the exact same as those for healthy people, even though their metabolism is very different.

Mike Haney (06:29):

Right. Well, let’s dive into the paper or the article rather. I keep calling it a paper, but it’s an article about reading papers. And the idea behind this really came up, because I think you’ve done a number of articles for us in breaking down studies. And a lot of what we try to do in our overall content operation here is take the research, the deep studies, and try to synthesize what’s going on in them and present the findings in a way that makes sense and is actionable for your average reader. And that’s a lot of what you’ve done for us as well is taking a study and breaking it down. But this article goes another step, which is to say, if you’re going to read a paper, a nutrition study yourself, Mr. Average person, here’s some things to keep in mind when you do it.

Mike Haney (07:18):

And where I wanted to start before we get into the 10 heuristics that you laid out, 10 things to keep in mind when you’re reading these studies, is this article presumes something which is that there’s value in your average person, somebody who maybe didn’t come up with a science education in college where they got experience doing this or isn’t a doctor that there’s value in those people actually reading original research, reading studies. And I feel like we could make the case for that, but there might also be an argument against that, that maybe really they’ll get the most value out of reading carefully written articles that are doing that interpretation for them. So I’m curious, just your thought on what is the value of or do you recommend that in an area that interests somebody, whether that’s exercise or nutrition or a particular disease that they go look at the original research, that they go out to a PubMed or they go to the journals and they try to read these papers in the first place?

Matt Laye (08:15):

Yeah. So I was thinking about this actually earlier. And I came across a quote in a paper that I was reading that scientific literacy is fundamental to both the functioning of modern civilization and its advancement. And I just think that having the public be able to have a level of scientific literacy is absolutely critical to how public policy gets determined. And while this stuff is really hard to read and there is a case that maybe we should lean on experts a little bit, I think that at least having an idea of, okay, what is the process that is going into doing these studies? How do we actually look at it? And given that nutritional scientist has had this dip in public trust given all the swinging back and forth between this is good for us, this is bad for us, this diet is best, that diet is best, I think it’s an area of science that really needs to have individuals who are not involved in the science to be able to understand and read it and be able to interpret it on their own.

Matt Laye (09:21):

And there’s just a lot of underlying difficulties in making sense of a lot of the stuff. Part of it is just because there is so much of it now. If you look at the number of papers that have been published year to year and this decade versus previous decades, I’m going to say it’s multiple times more. It’s just not possible to read all of it. So having that skillset yourself, you’ll be able to find the things and interpret the papers that may apply more directly to your own life situation as well.

Mike Haney (09:54):

That’s a really useful point, I think both about the notion of scientific literacy, which is to say that understanding some of these questions you might ask about a paper doesn’t even necessitate that you go read the paper, but it might give you more tools for understanding the article you’re reading about the paper. But also your point about the increase in the number of papers that are available and the of studies, I feel like that’s one of the areas where this larger point about understanding the context and nuance of any given study is really important, because I know I found this happens with me as well.

Mike Haney (10:27):

Anytime we write a new article, we’re going to dive into fiber or cancer or working on one now about how circadian rhythms affect metabolic health. You go start reading, start researching the topic and you read a paper and it’s a fairly conclusive paper. Maybe it’s even a meta-analysis. So it’s a bunch of papers. And you think, “Uh, I found the answer, I know the definitive word on this.” It turns out X is the answer. But then you maybe you click off to some other paper that’s referenced, and it’s a completely opposite finding or it totally questions the first one. And it’s really easy, I think, to read a paper and then go to the cocktail party that night or go to the bar and go like, “Oh, I found the answer to the thing, and I know it’s true because I’m not even just trusting a journalist interpretation, I’ve read the actual science.”

Mike Haney (11:10):

But it seems to me it is almost always the case that there are lots of studies about any given topic, the studies are looking at different things, which is one of the things you get into in the points here. And one of the great points you make at the end of the piece, which we’ll get to later is science is incremental. It’s rare that we say, “All right, we’ve done the study, we now have the answer.”

Matt Laye (11:29):

Yeah. And that’s the challenge for public policy makers is to somehow make some recommendations that are based on the entirety of the literature that applies to the entirety of the population, which is really difficult when you have a lot of things that have very high degrees of nuance are very dependent upon the situation, dependent upon the study. And being able to pick out those little variations can maybe help give you a little bit of an insight into, well, how much does this apply to me or how much does this apply to the general public or how much faith should I be putting on in this specific article. And not faith as in was it done well or not, but just faith as in how relevant is it to me, how relevant is it to the population.

Mike Haney (12:17):

Yeah. And I think that’s a good lead into the question. So we have 10 heuristics and I thought we’d just go through them and you can talk a little bit about what you find valuable with each one. And the first one, I think speaks to what we were just talking about, which is what is the big question addressed in this study? And to me, that’s getting at this point of each study is trying to answer something relatively specific and not maybe taking on the entire topic of cancer or the entire topic of a given nutrition subject. But it’s probably asking one very specific question, right?

Matt Laye (12:48):

Yeah. And it could be comparing one specific diet to another specific diet. It could be comparing that in a specific population. So say cancer patients postoperative or in remission or something like that. And so looking at the specific question is really important. And one way that I actually didn’t lay out in the article, but I think is useful is studies when they’re designed, they’re often designed to answer a specific question, a specific aim. And we call that the primary focus of the study. So the population they select, the intervention they select, the outcomes that they select is all based on that one question. And that will be the strongest evidence will be to try to answer that question, but there could be secondary questions or secondary analysis in that paper that might not answer that specific question, but a slightly different question that’s related or something to that degree.

Matt Laye (13:41):

And so when you look at a paper and you see, okay, was this study actually designed to answer this specific question? Let say, is that a hypothesis of this study or is it more of an exploratory study that’s looking at something secondary that might give those insights into how to conduct a more rigorous study in the future? And you can often find these when studies are designed, they have to get registered with a clinical trials registration and you can actually put that in the paper themselves. And so you could go see and see was the study when they originally registered it designed to answer the specific question?

Mike Haney (14:15):

That’s right. And often that is also found sometimes in the abstract, which is the short description about this study, which you can always access for free even if the study itself is behind a pay wall. But if you have access to the full study, there’s usually an introduction section that will also get at, in some place describe what that primary hypothesis is that’s being addressed here.

Matt Laye (14:35):

Yeah. And even the best studies will say an exploratory analysis, like in the title, which will give you a nice setup of, okay, this wasn’t the primary question, they were asking something secondary to that main question.

Mike Haney (14:48):

And I think this cuts the other way as well which is if a study’s designed to look at one particular aspect of say a diet, that it’s not addressing something else does not mean that other points can’t be true, sort of the absence of evidence is not evidence of absence as Peter T says. And so it’s important not to draw conclusions from what’s not in the study.

Matt Laye (15:09):

Yeah. And it’s often the things that you didn’t know you were going into the study to look at that end up being the most interesting and lead on again that incremental part of science, lead on to the next question and the next study that maybe gets at that.

Mike Haney (15:23):

Yeah, that’s true. I feel like it’s a joke that almost every paper I read if you get down to the conclusion ends with. And that’s why more stuff is needed in X and it just speaks to that point of science is incremental.

Matt Laye (15:35):

It’s required. You actually learn that as a PhD student you have to put that in. Not that really, but it does seem like that, that’s part of the template of writing a paper.

Mike Haney (15:44):

Yeah. You mentioned the design of the study. And the second question is what is the design of the study? And why is it designed that way? And a point I really like that you get at here is you say, well, randomized controlled trials or RCTs with a double blind placebo controlled crossover approach might be the “gold standard.” There’s plenty of reasons not to do those particular studies for any given topic. And I’d love to hear you talk about that more, because I think to the extent that folks are literate in the idea of study design that’s a very quick space to go like, well, was it placebo controlled? Was it double blind? Was it randomized controlled? So talk a little bit about why studies are designed the way they are.

Matt Laye (16:25):

Well, first off, nutrition studies are just always going to have an issue with blinding, because the subjects are going to know what they’re eating most of the case. So you’re not going to be blinded at least from the subject standpoint. And when you do these randomized control trials, so this is trying to put one group doing one diet and one group doing another diet. And so one example that I’ve read about is the women’s health initiative, which was comparing a low fact to a… Let’s see. It was a regular diet versus a low fat diet. And they were just told to eat these things. And they wanted to make this study long study so that they could try to find an effect. So it was, I think over a year, for years, actually, they’re supposed to follow the diet. And so they randomized them, they have them into these two different groups and then they look at the data at the end, and it turns out that they end up both eating the same diet.

Matt Laye (17:19):

And so it’s really hard to force people to eat different diets. That’s one of the issues with these randomized control trials is that changing people’s entire dietary habits for extended periods of time, which is necessary to really get a question of does this diet improve this chronic metabolic disease, which takes a decade to develop? Well, then you have to be on that diet for a decade to have a definitive answer. It’s just not necessarily feasible to randomize people and to put them into those conditions. So sometimes we have to rely on observational studies, which is just finding the people that already naturally eat, say, a regular diet and a low fat diet and follow them for years. And then we might have a better understanding of, okay, well, who develops metabolic disease and who does not. And that’s just a couple of the issues related to nutritional epidemiology, nutritional science, as far as the study design.

Mike Haney (18:15):

And how do you think about those two inter-playing, because I find myself writing this a lot in assignment briefs to writers when we’re looking at a particular topic. Like we just did a big piece on eggs, and I said, “Boy, I’d love to see both the epidemiologic research here, what you just described, hey, we followed people for 10 years who say they eat an egg every day to see what the outcomes were, but also interventionist designs where we give people a specific diet, we tell people that need an egg every day and we’ve been for three weeks or three months or whatever it might be,” because it feels to me like both are you useful that you get trends from the population epidemiological thing, you get indications.

Mike Haney (18:55):

But for reasons we’re going to talk about, about all the confounding factors, it’s hard to be definitive there. And the intervention studies might tell you a little bit more about specific mechanisms involved, because you have more ability to test people and to get a fuller picture of who those folks are, what effects this intervention is having and to control that they’re actually sticking with it.

Matt Laye (19:16):

Yeah. I look at those randomized controlled trials as often the best things you can measure is not the main outcome that you’re interested in, which is the disease. So you’re looking at some biomarker of that disease, something that we know is correlated to the disease process. And that combined with an observational study that maybe can look at the actual prevalence of the disease, gives us a guiding light towards the answer. And so they can work together. And I think you’ve described it really nicely of how they answer different questions. One maybe is a little bit it more of the physiology and basic mechanisms and the randomized control trials. And then the longevity, the observational studies, the longitudinal data can then get at, okay, what’s the prevalence of these diseases happening?

Matt Laye (20:04):

And when you start to get consistent data among the many different types of study designs, then that gives us a much stronger evidence based to make recommendations upon. It’s when those things contradict each other that it gets really messy and really hard to make firm conclusions. But when they agree, then it’s great. It’s just nice supporting evidence in a different way, exactly what we’d want to see.

Mike Haney (20:31):

Right. And I think this relates to our third question, which is how is the nutrition data collected? And you point out here that, for example, a survey that asked you to recall the number of times you ate a specific food in the last few weeks is not as accurate as a laboratory providing every meal throughout the study. So talk about how nutrition data gets collected in these studies.

Matt Laye (20:49):

Yeah. So I’ve been part of studies in Copenhagen during my postdoc. They ran studies that were looking at inactivity, but they wanted to control for diet. So the participants basically got a bag of groceries and then they would bring back what they didn’t eat. And by deduction you’d say exactly what they ate. And so you had a good idea of what their nutrition was, whereas a lot of these observational studies are looking at these dietary recall measures. So the randomized controlled trials often have a better controlled diet. So it could be providing the diet exactly is going to be the best option, even internal metabolic ward. So it’s like having people come to, say, a research site and stay on site and have all the food provided again, then we have really good knowledge of the nutritional data, but not super feasible for long term studies.

Matt Laye (21:41):

And those observational data are often collected from, say, a single dietary record, like recall maybe a three day dietary record that’s extrapolated out to what they eat always or a seven day, something like that. And those dietary recalls or some of the surveys are called food frequency questionnaires, how many times in the last month have you eaten eggs? How many times have you eaten red meat? Those are then sometimes correlated with the disease at the end. And the problem with those is they’re quite inaccurate and they’re actually the basis for the food recommendations that the USDA makes as these food frequency questionnaires.

Matt Laye (22:21):

And there was a paper that I always go over in my nutrition class about how the NHANES data, which is this big data set that is basically a bunch of surveys uses that data to make dietary record recommendations when about 60, I think it’s like 67% of the respondents would not even be consuming enough calories to live. So the records are very off. They’re very off. So you can get around that a little bit by collecting a ton of people’s data and hoping that the averages lead you to somewhere that’s closer to the truth, but they are just inherently not a great tool for collecting nutritional data.

Matt Laye (23:02):

Now, I think technology can solve some of these issues. Having these computers with us all the time with a camera attached, being able to take pictures of food is going to change that a little bit. I think it’s going to make the observational studies once we can get the AI and machine learning down to actually pull out the nutrient information from those meals, I think will help. But right now, those old school questionnaires, my students take them and they’re like, “What? This is how we learned about nutritional sciences? I don’t remember the last two months or month how many times I’ve had an apple or nonfat ice cream or something like that.” Yeah.

Mike Haney (23:44):

This is maybe a dumb question, but I didn’t take statistics in college. So are there ways that researchers statistically control for that poor memory? Is there something you can do to discount or contextualize people’s answer because you know that they tend to misremember X% of time?

Matt Laye (24:04):

I’ve seen different approaches. Recalling them off the top of my head, I can’t really recall. I imagine some a correction factor can be applied if, say, you have a large cohort and you have a smaller aspect to that cohort that you collect really good data on, and you might be able to then induce what the larger cohort should be like. But I’m not skilled enough as a statistician either to actually do that. But I do know that such approaches have been taken in some studies, but not all studies. And that’s one of the things that people should be aware of, what are they doing to really ensure that the nutritional data is valid? Is there stuff in methods of the paper that says, okay, this particular instrument that they’re using to measure nutrition has been validated somewhere else? And then how well was that validation done and how valid actually it is. The secondary questions which could lead you down the rabbit hole looking at paper after paper, after paper, and then finding that the correlation is not that strong or it’s not that great. It’s valid, but it’s not that great.

Mike Haney (25:15):

Right. So our fourth question here is what were the significant outcomes, primary and secondary of the study? So talk about what a significant outcome means in the context of a paper?

Matt Laye (25:26):

So that’s what question are they trying to answer? So this gets at point number one too like what is the big question? So is the big question related to weight loss? Is it related to metabolic health? So is it related to one’s… And it’ll often be related to one specific aspect of that. So let’s take weight loss, for instance. So the primary outcome might be how much weight did they lose? So that would be the major goal to study to compare a couple different groups and see how much weight that they lost. A secondary outcome for that might be, was that weight fat or was that muscle? So that could be an aspect related to the primary outcome that’s a little bit different and maybe not as important, but you might want to know whether that weight that’s coming off is metabolically active lean tissue or is it hopefully at a post tissue and fat tissue?

Matt Laye (26:21):

And so the good studies will tell you, again, in the introduction that the primary outcome of this study was X and some secondary aspects were Y and C, and this is mostly going to be done in clinical human trials, reviewing a paper for you guys right now that’s a more of a mouse study. You don’t have that same primary secondary aspect when it comes to animal work or cell culture work. That’s mostly related to human experimental trials.

Mike Haney (26:51):

The next question is one I’m really interested to talk about, which is are there biases at play? And I think this comes up a lot in nutrition. We did a piece on fructose a while ago, a big ultimate guide to fructose. And it started as is fructose good or bad for you? And it started that way because as we were doing the research, we kept coming across these studies, we came across a whole bunch of studies that pointed all the reasons it’s really not good for you and all the negative effects that it can have both in animal and in human studies. Then we would find these studies that seemed to be really big reviews that really questioned those aspects, and that pointed out a whole bunch of reasons why, in fact, those findings were not quite right and they really weren’t contextualized well. And only when we dug in a little bit further, did we realize that pretty much all the studies questioning the fructose were funded by industry food groups that when you took those out that their findings were a little bit more clear.

Mike Haney (27:46):

At the same time, I feel like this can be a bit of a boogeyman that we assume there’s some nefarious evil cabal of industry food groups that is funding research to question everything that’s out there. And I try to avoid this conspiratorial urge and think, well, if everybody is operating from a place of genuine good intent or at least what are their incentives that maybe the truth will emerge here or that we shouldn’t necessarily discount things. So I know there’s other kinds of biases that we can talk about, but I’d love to start with just the idea of, particularly with the nutrition research, how do you think about these industry funded food studies or nutrition studies or other sources of funding? An orange growers’ trade group, for instance, is not an industry, but it’s not like Nestle funding a study. How do you think about these funding sources and how seriously do you discount or not discount the findings based on the funding source?

Matt Laye (28:41):

Yeah. A lot of this research doesn’t necessarily get funding from the federal government. And so researchers who are interested in really doing the research have to find other funding sources and industries are interested in funding it. And I think from a good spot in general, they’re interested in seeing whether or not knowing that there are specific health related questions that need to be answered. And I guess I look at it as I’m just more inherently skeptical if I see a funding organization that’s called out in the conflict of interest at the end of the paper. That’s often where you’ll see it, but I think it’s worthwhile looking sometimes at the authors and seeing if they’re funded by organizations elsewhere if you’re questioning it, because I’ve also seen multiple times where it’s not been disclosed. And I think that, that can really erode your trust in a paper or trust in the science that’s being done, and when people aren’t upfront about the biases that they might be having.

Matt Laye (29:45):

And I just think that the data is actually pretty strong to say that industry funded work is much more likely to come out on industry side of things than government funded work. And so I’m inherently skeptical towards that reason. And I don’t necessarily think it’s nefarious. Like you, I don’t think that they’re running some massive scam that’s requiring everybody to be on the same page. Again, it’s like that would come out some other way. People would find out about that. And I think it’s just much more innocent than that and just that we have these inherent non conscious biases that can come out of this stuff. And it might be that the data is perfectly fine, but the way that they set up the study just was going to favor the industry point of view. And that may not have been conscious, it was just, well, how do we design this? And yeah, it could have been a completely non conscious decision.

Mike Haney (30:38):

Yeah. I think that’s a really interesting point. And what’s useful about all of these different questions and all of these different ways to look at a study is to start to understand when we talk about study design and we talk about what’s the primary outcome and what’s the population, how’s the data collected, that when you start to understand all the different factors, I think you say in the intro to this piece that doing a study is basically making a series of compromises or at least a series of decisions about what you’re specifically going to do here, which just means that inherently there is a whole bunch of places that this bias, whether it’s conscious or not can creep into the study design and not necessarily predetermined, but at least influence a particular outcome.

Mike Haney (31:15):

And I’m curious as somebody who does research, who has to find funding for the research that you do, how conscious are you? Is your working on designing a study coming up with it and then actually executing it of who funded it? How much do you think about that? How independent are most studies from their actual funding source?

Matt Laye (31:37):

You’re not going to take money from an organization that says they want the specific answer for sure. That’s a pretty big picture, but the big compromise at the beginning is, do I want to do this research or do I not want to do this research? And if I want to do this research, I need to find the money. And so I’m working with some colleagues and we’re working on a probiotic study and we have money from a probiotics organization. And they don’t have any hand in the study design, but we have to consciously remind ourselves when we’re setting up the experiment, okay, are we doing it in a way that maybe favors that outcome more unfairly? Should we be looking at a different way of setting up this experiment and looking at the outcomes based on the fact that we are getting money from a probiotic company in this instance.

Matt Laye (32:26):

And it is hard to just remind yourself over and over again that, that is the case. So I think of it as sometimes necessary depending on your situation and me being at a smaller school without a lot of internal resources and without my job description is not to do research all the time. So I don’t spend a lot of time writing government grants. We don’t have the resources on campus to really carry out the level of research that would be expected from such grants. So I have to figure out other ways around it. And that does make it difficult. It is certainly difficult. And the type of work we do is going to probably require less money. And therefore, I’m making compromises on the outcomes that I can measure and the things that I can collect, the approaches that I take. So that’s how I think about the conflicts. I haven’t worked with a ton of different pharmaceuticals or nutritional companies in the past. So this is a new experience for me as we’re working through our design for this particular study.

Mike Haney (33:27):

And you make a good point in the article that it’s not just funding as a bias, you have a line in here that says the integrity of the data might also be subject to the researcher’s bias to promote their research or personal agenda with a goal of future funding. How much do you think about and have to consciously guard against that as well that when you go into a study project with a hypothesis, obviously our goal in science is always to have a question, prove ourselves right or wrong, that it’s great if we end up being wrong, that’s fine, we’ve learned something. But at the same time, I assume as a researcher, you don’t want to constantly be wrong. You go and do it with hypotheses, you start papers and you probably prefer that your hypotheses has proven true or all of the other career aspects or respect within your school or within your industry, etc, how much do you have to consciously guard against other kinds of biases independent to funding?

Matt Laye (34:19):

Yeah. This is one that I’ve struggled with actually, because my PhD advisor was fanatical about physical inactivity being the causal effect of so many different chronic diseases. And we’ve written some really massive review papers with 400 or 500 cited articles and then looking at 30 diseases that are caused. So my bias is certainly that being inactive is one of the worst things you can possibly do for yourself. And that was one that I took and I actually took nutrition out of it. And I said, “Okay, it doesn’t matter, it’s all about the inactivity.”

Matt Laye (34:58):

And even to the point where Gary Taubes came and talked at our school when I was a graduate student and I was almost infuriated that he was saying diet was the thing that was important. And that’s one of the things that I have had to change my mind on to really recognize that, no, actually, I think these are two maybe equally important. And specifically with weight, I think he was writing about weight loss at the time how exercise was worthless for it. At the time, I was just completely against that idea and I wasn’t necessarily even doing work in that area, it was my own bias from just the experience I was having.

Matt Laye (35:35):

And since I’ve pretty much done a 180 where I’m like, yeah, exercise is actually not that helpful for weight loss, maybe weight maintenance, but nutrition is really the key there. And that process, I think you constantly have to do. And it’s not a flip that you switch and can all of a sudden be like, “Oh yeah, that idea I had is completely wrong.” It is often much more gradual and it takes time. And luckily I wasn’t entrenched in that research and I wasn’t making a career out of that specific question of weight loss and diet versus exercise. But others who have that entrenched position and are known a specific hypothesis, I can just imagine it’s going to be so much more difficult to see the other side to even be convinced of the other side. And you see this play out in society in all different areas, not just science, that being convinced of the other thing is really difficult.

Matt Laye (36:34):

And as scientists, we’re very good at finding the data that we want to support our hypotheses. And it’s one of the dangerous things about having so much data out there in the world is that we can find a paper that supports our hypothesis and cherry pick it every time we need to prove a point. But having that self integrity that knowing that you’re doing that is really hard. It’s struggle for sure. Yeah. And it’s-

Mike Haney (37:00):

One of the things that gives me hope is that I think you’re right, I think this does extrapolate to so many other parts of society, politics, social issues, etc. One of the things that always gives me hope when I talk about science or I’m trying to teach my eight year old about science is at least the field of science as we think about it now is at least structurally set up to fight against that, that at least there are the mantras of science that we are testing hypothesis, we are 100% okay being right or wrong. I’m trying to teach my son that it’s okay to be wrong. So I keep telling him and pointing out papers where I’m like, “Look, scientists are wrong 95% of the time, because that’s just the act of seeking knowledge and trying things. You’re just going to continue to be wrong.”

Mike Haney (37:38):

So I feel like that’s maybe a bit of a counterweight to this point is that at least coming up and training as a scientist is that at least you were working within a system that reminds you, hey, you have these biases try to work against that. We are trying to be neutral. We are trying to always be open to new ideas, etc.

Matt Laye (37:56):

Yeah. The problem comes when funding is involved and then that idea is funded and funded and funded again. And those people who get funding get funded in the future. And that is, I think, one of the systematic issues that people are aware of in the scientific community that we need to diversify some of these funding sources eventually, because if you just keep funding the same people, you’re going to be getting one point of view and their hypothesis is going to be driven over and over into the public view. And I think that, that’s a well recognized problem. And there are ways about going about fixing that specific problem as well.

Mike Haney (38:36):

Well, if we haven’t already completely boggled your brain with all the ways that studies can be tricky, the next one is really a big one, which is what are the confounding factors? And I found this point to be really illustrative as it pertains to nutrition science, even as somebody who spends all every day reading these papers. Some of the examples that you give about the confounding factors that might be at play in any given nutrition study is diet composition, age, physical activity level, socioeconomic factors, sleep, stress, time of the year the study was done. So it feels like nutrition in particular, there’s just all kinds of confounding factors that can play a role in how the study bears out.

Matt Laye (39:17):

Yeah. Life is a confounding factor, because you have to eat to live and then everything else around your life is related to also what you’re eating. And so food is such a simple thing, but it’s also very complex from the neurobiology and the idea of satiety and hunger and the emotions associated with it, and all of the other things that can play a role in what we eat and when we eat it. And yeah, I don’t even have time of day that people are consuming food is something that could be a confounder here. And so the basic idea for confounding factors is that when you look at a relationship between, say, a diet and an outcome, is there something other than that diet that could have caused that outcome?

Matt Laye (40:02):

And like the examples in the piece, there’s lots of other things that could have caused it. So you have to be aware of that as a researcher and figure out ways to control it. So if you’re two different groups, are very different ages, have very different body compositions, then you may have to control that statistically or at least be aware of that. Ideally you want to have those two groups randomized and that when you randomize them, they’re exactly the same and they look perfectly the same so you don’t have confounders. If you get a large enough sample size, then perhaps all that other stuff maybe goes away in the wash, but it’s something you they’ll have to be aware of.

Mike Haney (40:38):

Yeah. And you point out in the piece, which I think is helpful, that you can often find these constraints in a couple of places in the study. One is toward the bottom it’ll often say, here are some limiting factors. There’s usually a section at the bottom of the paper that’s like, “Here’s some potential problems with this research or here’s some things we want to acknowledge that might be shortcomings of it.” And sometimes they’ll even be explained in the introduction as well.

Matt Laye (41:01):

Yeah. And that’s a great first place to look for those things. And it’s where when I read a paper and I’m like, “I don’t actually see anything wrong with this,” and I’ll go to the bottom and then be like, “Oh, I missed this, this and this, that’s really nice that they’re telling me.” But sometimes there are things that maybe they haven’t thought of or just their point of view as in the field that they’re in, they may not see that as a confounder. I may not have seen diet as a confounder for weight loss and they might not see the physical activity as being a confounder for certain nutritional things or metabolic outcomes. And so the point of view that we have, our biases will affect what we see as confounders and not confounders as well.

Mike Haney (41:41):

The next two questions I’m going to group together, because they’re both about the concept of extrapolating the finding. And you point out that it’s really easy to read a headline that says, “Piece of dark chocolate a day will keep the heart attacks away,” and think, great, we’ve solved it, we know now that dark chocolate is good for us and I should therefore eat dark chocolate every day. But you make the point, the two questions you ask here is do these findings extrapolate to other populations? And do the finding extrapolate to other clinical measures?

Matt Laye (42:08):

Yeah, so other populations. If you have a narrow population, say, type II diabetic that have recently been diagnosed in the last five years, but have controlled, and then you do a study and you look at glucose control and then you say, “Well, I’m a healthy individual, that should work for me,” probably not, very different underlying pathologies between somebody who’s healthy and someone who has type II diabetes. Some are young and old people, you’re not be able to extrapolate those findings between those different populations. And so it is important to look at, okay, who is being studied.

Matt Laye (42:41):

And then the second idea there of different clinical measures goes back to the idea of the randomized controlled trials might be looking at a surrogate measure or biomarker for some chronic disease. And we just want to be careful that we’re not extrapolating too far out with that and we’re not saying, “Okay, well, this lowered HbA1c, therefore, longevity has been increased.” Well, we know those are associated, but we can’t necessarily say that in this case. Yeah, it suggests it and you probably would want to do the thing that lowers HbA1c levels, but that doesn’t necessarily mean that there’s going to be some massive bump in, in longevity.

Matt Laye (43:21):

And so it’s very easy because there’s so much literature to find, okay, some clinical marker that they measured in this paper that you’re reading and they’re related to something that you are really wanting to try to improve or really interested in another paper. You can find those papers all the time that connect those dots for you. And it’s part of the way we make up hypotheses and we design studies is because we see these connections in different papers. And we now want to have a study that answers does that diet specifically affect the clinical measure that we’re really interested or able to measure? Yeah.

Mike Haney (43:57):

Yeah. I think we’ve all learned the phrase over the past five to 10 years. Probably it is not correlation and it gets touted out on Twitter all the time to argue for against any particular thing. But I do think about this a lot in the articles we write, and a very common change we get from our fact checkers, because we have fact checkers go through all of our pieces and they read the studies. And one of the things they’re really looking at is are we interpreting the study correctly? Did the writer and the editor draw the right conclusion? Did we catch the right nuance? And it’s often changing some phrase to is associated with. It’s that this particular exercise is associated with this outcome.

Mike Haney (44:37):

And I think it’s useful for folks to remember to look for that phrase, because it’s really easy, particularly for journalists or those of us trying to interpret these to take it a step further without even realizing it to say this leads to this or this causes this. And in fact, what we really see is what you said, the dots are connected, but we don’t quite know yet why or how or if even that association might go away if some other study takes a look at it and maybe there was some confounding factors or something else that made that association appear to be there when it really wasn’t.

Matt Laye (45:09):

Yeah. We can be really buzzed kills when we’re with writing these things that’s associated with or linked to, but never use that word causal. And as a counter example is just the idea of smoking and lung cancer. I think that this is important, because sometimes we almost overly discount the associations, but some studies you will never be able to do to show the causal link, you will never be able to randomize people into the smoking group and the non-smoking group and look at lung cancer. So we came up with that evidence and that link between cancer and smoking all through associative studies. And maybe it took longer than it should have to make that link, but eventually we were confident enough to say smoking causes cancer. And without a single randomized control trial to lead us to that point.

Matt Laye (46:01):

So there are ways to do it. And there are actually some standards that have been laid out about how to get epidemiology to be causal. And I can’t remember all of the specific things that have to be proven. Maybe that’s another article at some point to show how epidemiology can be causal, but it can be done, but we still always want to be cautious and say associated not caused by.

Mike Haney (46:28):

Yeah. I’m glad you made that point. And I think that the smoking and lung cancer is a great example for a number of these things that we’re talking about, including one in which there was some pretty nefarious intent, I think on the point of the industry funding to prove otherwise. But yeah, I think that’s a really good point to make that when you see the phrase is associated with, it doesn’t mean you delete that sentence entirely, it just means that you remember that this is an association and there may be the mechanistic cause is not yet explained. But if you see that association over and over again, it does continue to point towards something that’s meaningful or that might make up a recommendation.

Mike Haney (47:02):

And the next point, the question number nine in here is related to this, which I love, because this is another one people get confused on. Is it statistical significance or clinical significance? This is something that often gets caught in our editing as well, where a writer will write researcher has found a significant effect of such and such. And the fact checker will point out, well, it was significant from a statistical point of view, but the actual effect was really, really tiny. And when a regular person reads significant, we think it was a huge effect, but statistical significance actually means something else for you as a researcher?

Matt Laye (47:34):

Yeah. That word significance is like a quirk of statistics. And what it essentially means is that the difference between say weight loss in group A and weight loss in group B doesn’t look like it was due to just random chance, that there was actually something at play, but it doesn’t tell you anything about the degree of that weight loss, which is really what we’re interested in from the clinical standpoint is how much actual weight loss occurred in group A versus group B. And that’s the clinical significance. So example would be, okay, you have a really big study and you’re able to get a lot of what we call statistical power, so you’re able to detect small differences in measures that are not just due to random chance, but are actually due to something else.

Matt Laye (48:21):

And so maybe you have diet A, diet B, weight loss of one pound difference and it’s statistically significant, highly significant, very unlikely due to just the chance of probability, but it’s not really significant from a clinical standpoint. One pound, probably not that big of a deal. And so that’s the key thing is to pay attention to both the statistical and clinical significance. And hopefully studies are reporting in addition to the P value, which is the measure of statistical significance, the effect size, which is the measure of clinical significance.

Matt Laye (48:54):

And this is something I teach in my research methods class and it’s not intuitive and it’s hard even going over and over and over. And many times as I tell the students this, they will still mix it up on the exam. And it’s just a very counterintuitive way of phrasing these things. So the effect size, we see that that’s the one we’re more interested in to tell us is it a small effect, a moderate effect, a large or a very large effect. And, of course, there’s just arbitrary cutoffs for what those words mean in the statistical world, but at least we have some way of gauging how strong of an effect it is or how clinically significant it might be.

Mike Haney (49:29):

Yeah. This is the area that find as somebody who didn’t come up with a statistics education or even through a science education. As an undergrad, I came up as a journalist, and we should be taught as journalists more about this or maybe I just missed that class. But I feel like when you get into all these statistical language around studies, like the P value, how powered your study is, effect size, is where studies can start to feel a little Greek to most people. And it’s really difficult to know how to account for those things. I see a P value and I go, “Well, there’s a P value there, I guess it’s probably good that there is a P value, but the P value between this and that I don’t quite know.” And I confess that I essentially gloss over those parts of the paper.

Mike Haney (50:16):

Is there any quick guidance or anything that you can tell people, your average person who doesn’t know statistics, to look at around that language that would be a red flag or the opposite of a red flag, and then you go, “Okay, this is a pretty good P value or this is whatever?”

Matt Laye (50:33):

Yeah. I think the common one you see in epidemiology studies is when you’re correlating two variables. So say, intake of a specific nutrient and some clinical measure like glucose area in the curve or weight loss or aerobic capacity or something. And you’ll get just such a huge number of people that it’s very easy to get a significant correlation and you’ll get from that. So you’ll have a P value of your correlation of 0.0000001 or something and you’ll be like, “Wow, that was highly unlikely that those two variables would move in the same direction.”

Matt Laye (51:10):

And then you look at the R-value, which is a measure of the correlation strength, which is also another way of saying how clinically significant is this. And you’ll see an R-value of 0.2, which is pretty low. It should range from zero to one. And a 0.2 is pretty low showing a pretty weak correlation or association. And so you say, “Well, it’s highly significant, but it’s also weekly correlated.” And this is very commonly mistaken. And it’s very common in, like I said, these large epidemiology studies where you have so many people you can detect any small association. And that’s the one that I think is most difficult to interpret.

Matt Laye (51:53):

If you see the scatter plot, sometimes you can tell and you can see that, that’s not like all the dots on a line, which would be a correlation of one instead it’s dots scattered way above and below the line and R-value of 0.2 and you’re like, “Okay, well, it doesn’t even look like they’re that related. And that’s what the R-value’s telling us, even though the P value might look really impressive, being very, very low.” Yeah. This is another thing I teach in my class that I spend a lot of time actually talking about statistics in a research methods class, because to interpret the science you have to have, at least it doesn’t have to be a degree in statistics, but you have to have a familiarity with it.

Mike Haney (52:32):

Well, I think we’ve just uncovered another article that we can write where you’re going to make statistics understandable to us normals.

Matt Laye (52:41):

That’s a challenge.

Mike Haney (52:41):

So the last point we have in here is something we’ve been coming to again and again throughout this, but I’d love just your closing thoughts on it, which is, and I love the way you phrase this, science is noisy and iterative, but it bends towards innovation and truth. And we’ve been phrasing it here as science is incremental. And I often think the correlation or the phrase that goes along with that is science is done by humans and with the help of machines, but with all that, that inherently comes with that. So I wonder if you could just speak on that a little bit and how folks who are consuming science through the studies or through the articles about studies should use that truth as a way to interpret and use what they’re reading.

Matt Laye (53:21):

Yeah. I think one thing is a single study in a vacuum is not really science, actually, either. And that you have to really look at the full context around that question. And that becomes really difficult in novel areas, because there’s not that much data. So we can easily generate the hype machine to get to think that, okay, this new truth is somehow going to be groundbreaking in some ways, but we don’t have the backbone and the depth of studies to really show that as being the truth. And the goal of a lot of this nutritional science is to really improve public health. And so the public is made up of so many different individuals and individuals are going to it very massively in how they respond to the same nutritional stimulus, exercise stimulus, whatever the intervention is, our physiologies can be dramatically different.

Matt Laye (54:16):

And so when I think of sciences, okay, even if we have some public policy truth, there’s going to be variation in how individuals respond to that. And so when we read a study and we look at the results, the only way we’ll really know if that works for us is to replicate that study and have one and see if we can replicate it, knowing that we can’t do a controlled version and a non-controlled version of ourselves. And so there’s always a little bit of problems with that. There could be other things that we change inherently when we’re making these interventions on ourselves.

Matt Laye (54:49):

But that is how I view these studies and their utility is that, okay, this gives me a place to start experimenting in my own life, regardless of the public policy implications of this work. And they’re not the same things like I write, one is science, the public policy trying to gather all of the evidence and this one study will contribute a little bit to what we know a little bit more, so we’re a little more confident or a little less confident in these are things that we thought we knew. And the other is, okay, this is a basis for me to try something. And that latter one, me trying something is not necessarily science. And so we have different goals there and how we use the science as individuals versus how scientists look at the data, how policy makers should be looking at the data.

Mike Haney (55:36):

I love that point. It underlines for me that the point of this whole exercise we’ve been talking about today of trying to have better science literacy, and in particular around nutrition science, being able to read a study, should ultimately be empowering, it shouldn’t be confusing and it shouldn’t be disheartening. It should be empowering, and it should be empowering to help you find the things that work best for you guided by these findings, this work that other folks have done.

Matt Laye (56:03):

Yeah, data informed, not data driven. I think I’ve heard that on your guys’ podcast before.

Mike Haney (56:10):

Exactly.

Matt Laye (56:11):

Yeah.

Mike Haney (56:11):

Well, thanks so much, Matt. This has been incredibly helpful. If folks want to actually read the article, it’s out at levels.com/blog, where Matt’s gone through and said a little bit more about each one of these points and linked off to some other really helpful things as well. And continue to look for Matt’s byline on the blog, because his study breakdowns are always super interesting and are a great middle step of not having to read the paper yourself, but getting his expert take on what’s important about that paper. So thanks for your time today, Matt.

Matt Laye (56:40):

Yeah. Thanks for having me on the podcast.