Josh Clemente (00:00):
Let’s go ahead and do it. First Friday Forum of May 2023, somehow. As a reminder of what Friday Forum’s all about, we’re a fully remote, async team, we’ve got a lot of different functions, doing a lot of different things, and so this is an opportunity for us all to catch up and celebrate achievements across the business, and see into the other parts of the week. We also get to hear from our members and partners directly, and generally align on culture and the evolution of this experiment that is the Levels company. This isn’t the deep dive into business metrics, this isn’t the primary channel for social connection. We’ve got other opportunities for that.
(00:40):
This week, it was a big one. We kicked off Beta 2 with our hybridized new experience with CGM optional. This has been a huge effort. A ton of work has gone into this, as you all know, over not just the push into Beta 2, but also the work that built into Beta 1 and all the product work even prior to that with the alphas. Just a ton of work, culminating with the launch this week on-target, which is another big win.
(01:09):
We’re now working on fast follows, so adding additional analytics to things like multi-part stories, working on some bugs that have popped up, and generally starting immediately to scope Beta 3, which will have new features like a community feed. We’ll also be focusing on app performance. Just the amount of volume involved in all of this is really high, so the engineering team had the highest week, I think ever, of pull requests merged and released as that sort of final press to hit the date. I just want to celebrate the team for really crushing it, and I’m very excited to follow the Beta 2 cohort as they work through this brand-new product experience and we all evolve our understanding of where we have to build next.
(01:51):
Alongside that, we’re working on logging improvements. We’ve got a new UI, which Sean has been working on for a while, ready for testing. That will be just the updated frontend, and then on the backend, we’ve got a v2 in work with smart suggestions. This will be kind of the first foray into, or at least we’re evaluating the use of, GPT tools to generate good suggestions to help support the logging experience, make it more effortless. We’re also working on tap to highlight zone, which will be like you’ll be able to be kind of scanning across the glucose curve and be able to highlight a zone. User referral’s also in work, so a lot of stuff going on there which kind of touches growth and product experience.
(02:26):
Activation plan. Ben is pushing forward with the growth team on the plan for how we’re going to move into continuous signups. Instead of this cohort-based, Beta 1, Beta 2 structured situation, we’re going to be moving into ongoing, continuous signups where people can buy the product and get started. This is going to be obviously necessary for general availability. We had a successful member test of our walk-ins for labs. Labs 2.0 went live a while back. That’s our VIP sort of in-home phlebotomy experience, and the labs walk-in is going to dramatically improve accessibility there. We don’t have very high coverage in terms of the zip codes that can get an in-home phlebotomy, so with walk-ins, people can now schedule an appointment or just walk in at a local clinic, and they’ll be able to get that blood draw and get the results in-app.
(03:18):
We also had a successful quarterly business review with Truepill, just continuing to really work on optimization of our fulfillment operations. There’s a lot on the backend that goes into producing a really effortless experience for our members, but also reducing just errors and returns and frustration for our members means the same thing for our support team, so trying to constantly improve operations, and so I appreciate the ops team for continuing to push on our partners there.
(03:49):
The really cool result here with Foodguide, introducing a huge increase in event-based insights. You can see this massive spike starting on April 24th in event-based insights. This is being viewed, these are page views, and it was up 90% week-over-week, so continuing to have some really cool opportunities to surface these important insights, but also we want to see how these trend over time, how much people return to them and how consistent this trend stays.
(04:17):
Smoke Test #1. I think everyone’s kind of familiar, we’ve been running smoke tests. We deliberately split some of our traffic to our website to essentially test some assumptions on pricing and positioning of our new product. We got a lot of information there. A full analysis is ongoing, so no takeaways just yet, but Karin and Tom are working on that, and we’re going to push straight into Smoke Test #2, so continuing to take real traffic, real sort of A/B testing, with an intent to purchase, and going live with that on our website such that we can start to really hone in on a defined price for general availability. We’re also planning a soft general availability in June, so as you know the target is July for our full GA. We’re going to be potentially combining Beta 3 and a soft GA in June, so a lot more to come. We’re just constantly trying to descope and move faster.
(05:10):
We, on the content side, hit four million views on YouTube. We’re now over a million podcast plays, as well, so the numbers are getting very large. This is really awesome. The YouTube press is not that old. I don’t exactly remember when Ben said, “Hey, we should focus on YouTube,” but we certainly had like 10,000 views at most at that time, so maybe a year ago at most, pretty awesome to see this rate of return.
(05:32):
And then Whole New Level, we’re going to be resurfacing some of our… As the total number of place increases, our total audience increases, a lot of the older content, which are really high-quality episodes kind of fall off the map as people don’t necessarily go back in the catalog, so we’re going to be resurfacing some of that to get some more return on investment there. And then three new pieces up on the blog, Beth Bollinger, who makes a ton of our great recipes, MCT oil, sneaky spikers, which is a great one. And then we have a retro coming from our LA shoot that the team made a ton of guidance content last week, and the scale and 24/7 nature of the content production is, from what I can tell, pretty shocking. I am really excited to read the retro to see how the team has just been able to continuously turn out content that is member-quality, so I’m stoked for that.
(06:16):
And then we added a right rail to all member stories to the Levels blog. We’ve got an activation going with Molly Maloof. This is some user referrals imagery right here, which is being built out now. Made it into The Telegraph with a really nice piece. What else? Smoke test imagery, Ben, and based insights. I think we touched it all. I know we’ve got some love for, Sam actually gets this frequently, but just some love for how we share the real transparent insights into how we run our business. There are people who are legitimately building, in the image of Levels, their own companies in entirely different industries, but they just really appreciate us being willing to share how we do stuff.
(06:57):
With that, I want to welcome Michaela Kilander, a member here at Levels, senior scientist at a biotech firm. She’s working on stem cell differentiation, and also just generally into health and wellness, running, climbing, breathwork, a lot of stuff we all love. I would love to hear from you directly, Michaela, thanks for taking some time to join us this Friday morning, would love to just hear about your experience with Metabolic health, and specifically with Levels.
Michaela Kilander (07:23):
Hi, Josh. Thank you so much for inviting me to this forum. I feel like a lot of the information that we get from studies around health and wellness is very confusing at times, so just having something that can directly input, give me hard data on what’s going on in my body, which I can do with Levels, has been really helpful for me to just demystify the preconceptions that I might have around food and nutrients, so that’s been very helpful to me, and also just gauging my general status from day-to-day. For example, if I slept poorly, I can definitely see that impacting how I respond to, and my glucose levels after eating and so on, so that’s been really great for me.
(08:32):
Something that I would love to dig into with Levels also, connecting it to my microbiome and see whether a specific microbiome would lead to, supposedly it would do just that, but with the data with Levels, that would be something that I would really like to see. Do you know if anything has been done on that part, gathering the information around-
Josh Clemente (09:06):
It’s a great question, and there’s definitely a lot of exciting information out there that certain microbiome could contribute to better blood sugar balance and specific gut bugs could do a better job breaking down fiber and this sort of stuff. At this point, we haven’t yet been able to, although, yeah, exactly, Azure, we’ve basically had this on our minds for a very long time, and the difficult thing is making it super actionable and having an approach where people can collect that information in a consistent manner and then we can then surface it in the product.
(09:38):
It’s something that we’re very interested in, and certainly, we’ve explored a couple of the products out there that do some microbiome testing, so it’s really good to hear directly from you that that’s something you would be interested in. Our objective is to expand the amount of information you have that is accessible and actionable to make better choices, so it’s definitely an area of interest for us
Michaela Kilander (09:59):
Also, because not much is known about the microbiome, what is considered a good microbiome, a healthy microbiome, so having something that could potentially put some data or some metrics on differences would be super interesting.
Josh Clemente (10:17):
Absolutely.
Michaela Kilander (10:19):
Something that I also, with my breathwork practice, that I kind of recently started, around a year ago, so that was kind of at the same time that I’ve started using Levels also, so something that I would love to do, get some more data from Levels. I haven’t had a CGM in quite a while now, but knowing if I had a poor sleep, if I do breathwork practice, would that help me go back to a base level or something? That is something that I am also interested in investigating a little bit more.
Josh Clemente (11:00):
It’s very helpful, because I think what you’re probably getting at there is the relationship between stress management and blood sugar control, as well, so that’s a piece that is top of mind. We consider stress one of the top four or five pillars of metabolic health, and breathwork being a huge tool to be able to manage it. We have anecdotal input from people that breathwork helps with blood sugar control, but I would absolutely love to build features to be able to reinforce that and show, and provide a tool for people, when blood sugar’s kind of rising due to things like presentations, you can step in with some breathwork quickly.
(11:39):
You’re kind of getting exactly at what I wanted to ask, which was, based on your experience so far, where Levels should really focus, and you offered two really good suggestions here on providing more data around microbiome and potentially stress management or the relationship between actions like breathwork. In our new product, we are building checklists of healthy metabolic actions or metabolically-friendly actions like exercise, like healthy whole foods. I’d love to hear, beyond these two, what would be the number one thing that you would have Levels improve on, build, work towards from your experience so far?
Michaela Kilander (12:20):
One thing that I noticed when I used the CGM was perhaps I, when putting the CGM on, I kind of got some slightly different data from one to the other, like my baseline glucose levels changed a little bit from when I had to shift the CGMs after 14 days. That was kind of something that I was a little bit curious on why that was.
Josh Clemente (13:02):
It’s a good question, and it’s one that some comes down to variation week-to-week for the individual, and others come down to the calibration of the sensors as they’re sort of running in. I think there’s a lot that can be done to basically build a better blueprint of the individual so that a sensor can calibrate to your historic data to have a more consistent starting point, but that’s a little bit further in the future. I always love the reminder that we really should index on relative motion above and beyond everything else.
(13:39):
We try to remind folks that the absolute values can fluctuate here and there, even relative to reference measurements as those are all somewhat different, but really, it’s the trends that are going to ultimately define how your body’s responding to the choices we’re making every day. But really important reminders, and I really appreciate you coming on to the forum and sharing with us directly. Michaela, if you’d like to stick around, we have a meeting with a bunch of function deep dives, and would love to have you do so. If not, from the whole team, really appreciate you being a member of Levels and continuing to help us build the right product for your goals.
Michaela Kilander (14:19):
Thank you so much for inviting me, again.
Josh Clemente (14:22):
Of course. Thank you.
Michaela Kilander (14:22):
This was great. Thank you.
Josh Clemente (14:26):
Jumping into some quick culture and kudos here. We’ve got a ton here, which is awesome. Firstly, Sonja has been running this process in the background, which is a little bit out of sight for a lot of us, but closing the loop and celebrating guests who hit certain milestones with us. On this golden mic or this bronze mic, you can’t really see what’s written here, but I think this is Dr. Perlmutter’s one million Club, and Dr. Bickham hitting 100,000, so this is plays on podcasts and YouTube videos. The point is just celebrating the achievement.
(14:56):
Our advisors, our members, our guests are coming onto our platform and sharing a really important message, which our members love and really appreciate, and it’s not always clear what’s happening once that… They can just record and kind of drop off, they don’t get to see these analytics. This sort of thing, closing that loop, is such a huge piece of the Levels culture, just sharing some of that celebration with our members and with our guests, and so I really want to appreciate Sonja for just having this process running. We didn’t even know it was going on, and it’s really important. John jumped in and kind of very quickly packaged it up and handed off a project to Geniusee to handle the tappable zones on glucose charts. This is kind of an example of another piece of our culture of delegation and scaling oneself, so I just want to surface that.
(15:44):
Another piece of Levels culture is just continuously improving our documentation, and this is not easy to do. I know exactly how hard it is to set aside forward progress feeling work for making sure that documentation is being improved constantly, but Ian really sets a standard here for his team and for all of us, so I appreciate that. Justin cranking out guidance-related metrics. Crazy week for the whole team on product and engineering, and so just seeing the time to pause, again, make some adjustments to metrics while we’re trying to get work done is just setting the standard once again. And then always love seeing the team hanging out here in California, with Mike and Sonja getting dinner, with the significant others there.
(16:28):
And then a big, huge shout-out to the support team. Again, this whole process of iterating through on a consistent basis, breaking things, changing things as we go from alphas to betas to future betas and into GA. Each time that happens, the amount of change to our standard systems is hard to predict, and hard for any of us not working in that function to understand, so all of the members that we’re trying to keep super happy, despite unpredictability on our systems, it’s a really tough job, and I just really appreciate the support team for continuing to crush it. A lot of late nights. I think we had a queue that was above 250 cases, and they were down to 10 in the queue this morning. That’s 250 unique examples of keeping a high happiness standard and on making sure that we are the best in the business for what we do. Thanks again to the whole team. You’re the tip of the spear for Levels. Over to Sam.
Sam Corcos (17:33):
Hopefully there’s not too much background noise, and hopefully my internet works. The joys of Manhattan. The theme for today is we treat one another like adults. Many of you who were with us in the early days when we read No Rules Rules, it was a book on Netflix culture. We also read one on Amazon culture. But one of the things that we really took from Netflix culture is this concept of treat people like adults. The idea here is that when there is information that’s uncomfortable, there are two paths that most companies choose to go down. There’s either you hide it from people and then you spring it on them last minute, or you share it openly and transparently.
(18:20):
A lot of people have contributed to this. One of the early advocates for this was Andrew, who, when we started the company, was really adamant that we share our cash position and our burn with the whole team, because most startups don’t. Most startups keep that as a top-secret piece of information that only a handful of people know, and it can create stress and anxiety not knowing where the company stands, and the CEOs are usually telling everyone how great things are, and then out of nowhere there’s a round of layoffs or the company disappears. Part of the Netflix culture of treat people like adults is it also implies that people will act like adults and know how to handle this sort of information, which is why we put so much effort into making sure that people understand what these numbers mean.
(19:11):
It can feel more stressful, but these sorts of conversations, I was talking to Nicole about this recently, where these sort of conversations happen at every company, but they’re just usually not visible. I think somebody was mentioning to Nicole, “Is this level of chaos normal?” She said, “Yes. This is just how every company works, it’s just usually you don’t have access to that information.” A major part of treating people like adults is knowing that people will receive the information graciously and understand that we’re sharing a lot more information than would be typical. Knowing that, if there’s any information that you feel like you’re not getting from the company, definitely reach out to us, and we can try to be as transparent as we can.
(20:01):
I think one of the major things about this value is that when you’re able to treat people like adults, you don’t have to create policies that are paternalistic, that treat people like they are the lowest common denominator. One of the major failure modes in so many companies, Netflix has a similar concept around hire people with good judgment, which is very much related to this concept, which is if you hire people with good judgment and you help people understand what that means, you don’t have to create strict policies for every single thing that could possibly go wrong in an organization because you can trust people to make the right decision. It may end up costing a little bit more in different places, but the amount of overhead that you don’t have to manage, it massively outweighs the cost that comes from some added anxiety around having access to information that you wouldn’t order ordinarily know about.
(21:04):
The Netflix book also gives this example of, there was a conversation about whether a certain person was going to be let go because the position might have been eliminated. In the example in the book, they asked the question, “Should this person know?” and they play out the scenario. Scenario one is they don’t know and they think their job is safe, they buy a house, they commit to living in this location for a long time, and then suddenly, they find out that their role was eliminated, and they didn’t feel any stress until this moment. The other path is they’re told super early on, “We’re considering eliminating this role because we don’t know if this is going to be something that we can support in the long-term,” and while it might create some stress, that person decided not to buy the house, not to commit, and it allowed them to be able to make smarter life choices.
(22:00):
It turns out in this specific case, that role was not eliminated and so that person did stay at the company. But just knowing that you can treat people like adults and that you can trust them with this information, and that we also have an obligation as a company to share this information with you, and so making sure that we have a really open, two-way conversation about this, and knowing that there are oftentimes when people try to read between the lines on company communication, and we try our absolute best not to hide anything between or put anything in the fine print that people won’t see. We try to be as open and transparent as possible, and if you ever feel like you are missing some important piece of information or if there’s something that we could do a better job at in terms of sharing, please let us know.
Josh Clemente (22:54):
This is one of those values that Sam initially really pressed, and I felt even just the name of it was like, well, of course we’re going to treat people like adults, but when actually getting into these moments where there are these difficult transparency decisions, it’s really nice to be able to just default into understanding that we have only one policy, and that’s to share and not to build policies that limit information, because our main goal is to have people here who want to be here, who understand the circumstances that we’re operating under. Anyway, I just want to appreciate everyone for stepping into the other side of this decision, which is it’s not just about sending a cascade of information out on people and making stress worse. It’s actually kind of a partnership of recognition that this is the operating environment we’re in.
Sam Corcos (23:42):
I would actually add to that that one the reframes that’s always helpful in this context is oftentimes when people are making policies, people think, what is the worst thing that could happen in this situation? Assuming that the person we’re talking to acts in the most inappropriate way possible, how do we create a policy that protects against that? We often forget to reframe like, well, what if we can make the assumption that everyone here is an adult, what would actually be the best thing that could happen? And that often ends up being completely different.
(24:16):
The concept of treating people like adults, some of these are not new ideas. Treat others how you would want to be treated is something that most of us learn in first grade. And when you think, how would an adult want this information? Just knowing that we have people on this team who have opted into that and being comfortable talking about that. That’s a big part of it, is always thinking about, in the most optimistic lens, knowing that we work with a team of adults, how can we have those conversations effectively?
Josh Clemente (24:52):
Awesome. Thanks, Sam. No change to the main thin, Levels shows you how food affects your health. And we now have a quick beta numbers update, also, Sam.
Sam Corcos (25:06):
We hit some of our numbers, which is good, but you’ll notice that a lot of these have asterisks next to them, which we will be going into shortly. But we hit our 1,000 goal for beta conversions. We’re still waiting on retention numbers for Beta 1. I think we get that today. In fact, we may already have that ready by the time this is updated, and engagement so far is looking good for the next cohort. We slightly missed the engagement number for Beta 1, but that’s not super surprising for a really early beta. I want to quickly go over what these numbers mean and why the asterisks. We can go to the next slide. There are two ways that the beta could have gone. One is we knocked it out of the park and we see the light at the end of the tunnel and everything is wonderful and we know that we’ve hit product-market fit. The other is the numbers go really poorly and we know that we’re not going to get there in time and we have to reduce the size of the team. Next slide.
(26:19):
Where we are now is we’re not out of the woods yet, is, I think, the summary. The numbers for the 1,000 people that we got, a lot of those are existing members. We had to make a number of compromises around the quality of the data, where we thought about what it would take to get really good data with only new members, but making that compromise would have really put the support team in a bad situation, and it would have made it way harder for the growth team to figure out how to solve this because we’d have to exclude all these people. We decided to make some compromises on data quality to be able to get the number to understand more about product.
(27:03):
Similarly, with our retention numbers, we ran into some technical issues, where a lot of the people who wanted to sign up and continue using the product weren’t able to because of some technical issues that we ran into where they weren’t able to do that. This was our first iteration, where mistakes are expected and some of the data quality is not as high as we had hoped. We hope to have much better data during the May cohort, and certainly the June cohort, but just wanted to give the, in the spirit of treat people like adults, we are very much still in the uncomfortable middle period, where we don’t know what the right answer is yet. That’s, I think. The moral of the story for where those numbers are, and hopefully, over the course of the next month or two, we’re going to start getting much higher quality data as we do the rolling signups and growth is able to get people to sign up through our more normal growth channel, as opposed to these prefilled cohorts.
Josh Clemente (28:09):
Awesome. Thanks, Sam. All right, David.
David Finner (28:16):
Hey. Can people hear me? My internet just started cutting out. We’re good? Great. Coming off that, I just wanted to share the retention numbers from Beta 1 into Beta 2. Retention came in at 20.6%, lower than our goal of 30%. We spent the last week investigating the numbers, and we think we have a good understanding of several reasons why retention was lower than our goal. As Sam alluded to, perhaps the highest theme that we saw was a high-friction renewal process. That actually led to 101 tickets to the support team, some of the gray hairs that Taylor is mentioning in the chat thread here. I wanted to report these numbers on Wednesday, but we spent the last five days really digging in and resolving those tickets and trying to figure out what the true intent here was.
(29:06):
The big subthemes within that high-friction renewal, just to call out, are, it’s really a manual high-friction signup. Everyone in Beta 1, they went through a Typeform signup, and then we had to get them converted to the Levels system, reentering their address and their credit card. That itself was a high-friction path. Our users are not super tech-savvy. The Marine persona that we’re going after, compared to some of the original biohackers, what might be simple to us is actually pretty high-friction path for them.
(29:36):
The other ones are juggling two Levels accounts. We had a lot of people, I think there were 56 people out of the 267 total in our cohort, that we had to create a second Levels account for to put them in the Beta 1, but then we also had to do a forced migration off that account to support them in Beta 2. They also lost all of the logging investment that they had in Beta 1 and had to start fresh on Beta 2, on top of the confusion with signing in with a different account. And then fourth on this one is that, even when people were trying to sign up, no one knew what their password was because 100% of Beta 1 users had a random system-generated password, and so no one knew what that password was, they received it in an email the first time they signed up, and so we encountered friction around that, as well.
(30:18):
Beyond that, another theme was there was a little bit of a lack of clarity on what they were actually getting with the program. We intentionally kept it high level, which was great because that gave us flexibility to iterate, but people didn’t know exactly what to expect and they didn’t know how to get a path to a CGM. There was a significant number of people who wanted a CGM for Beta 2, and that was not officially built out as an option for Beta 1 people like it was for the Beta 2 new members. And then I think, as someone mentioned earlier, is that we were designing this program for Marines, and about 40% of our participants were not Marines. More analysis to come, but I wanted to give you a quick understanding of what we did over the last week diving into where this was and how we’re going to address this for Beta 2.
(31:01):
What to expect with the current cohort is that we’re moving to a frictionless monthly recurrence, and also a proper signup flow. No longer are we going to be signing people up via Google Forms. They are going to be using an official Levels web signup flow, and that will automatically remember their credit card, their address so they don’t have to enter it again the next month. And then we’ve been working on an official, actually, this is already live for our Beta 2 people, but they were able to either buy software only or software with the CGM and that worked out really well, so going forward, that’s already solved.
(31:35):
And then we’re working with Karin and the growth team to improve our signup positioning through those smoke tests and things like that, so hopefully it’ll get clearer and clearer, on the upfront side, what’s the messaging that’s going to land. And then on the software side, we’re really unifying our program and weaving that throughout the experience, so you know, from the guidance videos to the checklist and the logging, everything kind of works together as a system. That’s where we’re at, and stay tuned for another email from me in general on the deep dive into this.
Josh Clemente (32:05):
Awesome recap. Thank you for doing that, David. Over to Alan.
Alan McLean (32:11):
Hey. I’m going to give you a bit of an overview of some of the work coming up in Beta 2. This is very brief. There’s a lot of material to go through, and I think the whole team has actually put together, there’s analysis, there’s breakdowns, there’s metrics you can jump into, so I encourage you all to look on Notion to look at this stuff, because I think it’s actually pretty compelling. I do want to say right up front that we’ve gotten some really amazing feedback from people.
(32:37):
Overall, the sentiment that we got from our Beta 1 users was incredibly positive. Personally, when I see these things, sometimes I try to avoid looking at quotes like this, because it’s going to make you feel like you’re such an amazing person. I know I have work to do, but when you start reading this stuff, you realize these guides are really creating a connection with people. They’re really motivating them. They’re talking about the product in a way that you don’t typically see when you get feedback like this.
(33:06):
When we look at the actual data behind is it actually working, is this program actually helping people, we had a couple hypotheses that we wanted to evaluate. There’s a whole recap on this in Notion, but some of the most important ones were the structure of the program, things like we have this time-of-day format where we slowly transition you through the fundamentals, the morning routine, the afternoon, the evening, and we wondered, did that work? Is that effective? We think for the most part, that’s true, that yes, it seems to be helpful. We’re going to continue to evolve that, but it seems like the morning routine specifically was a really great place to start, and it’s possible that it becomes less helpful as you know more during the course of the journey, although I think the drop-off we’re seeing there is not too substantial, so I still feel pretty good about that.
(33:52):
We had some requests for different kinds of content for people. We saw, obviously, recipes, tips for families, stress, workouts, these all stick out. I think hormones and aging, it’s representative of this population, and we need to do more there, so that’s a good point for Azure and Sonja, Sissy, and so on, to start thinking about we have some content and it works for that. What were the most popular habits? It looks like breakfast protein. The nice thing about those kinds of content is we can really dig in and see how much are people engaging with this, and then do they go do it? It looks like protein for breakfast, post-lunch walks, these were really effective.
(34:31):
Were it people actually changing their behavior? I think the answer to this would be yes, it looks like… Oh, that shouldn’t be, I’m hiding the good stuff here. Oh well. But yes, it does look like we’re having an effect on people’s choices, which is what we love and what we want to see. Ultimately, if people are doing these things, I think they’ll be more retentive. All those sort of signup issues notwithstanding, we want to get to a place where we can show change in people’s metrics, in their lives, in their health, and that will hopefully keep them sticking around with us for as long as possible.
(35:02):
I highlighted this one down here because we think that accountability is something we’re going to need to spend a lot more time on. It looks like actually, just with the guides themselves, it’s pretty good, but when we start layering in things like labs and CGMs, I think we will get even higher on that. I should note, these are all out of 5, they’re not out of 10, so don’t worry. Beta 2, what’s happening in Beta 2? In Beta 2, we are essentially crossing the streams. We’ve got all these existing users from Beta 1, we’ve got these new Beta 2 users. Wait, I hope everyone understands what crossing the streams means. Is it only for old nerds like me?
(35:38):
Anyway, it’s all coming together. You’re going to have users coming in that will be hybridized, so they’ll have a CGM, and some won’t. There will be old users, there will be new users. This is actually a kind of considerable tech challenge, in that way we first started out the feed was pretty rigid, and now we’re going to start to have sort of conditional content. The Beta 2 program, it’s going to provide more structured guidance and it’s going to start bringing our guides together to help our users, so we can now do this thing, which will be helpful for support, I imagine, continuous onboarding.
(36:12):
This would become the beginning of a more dynamic content feed for users. Of course, in the past, it was very rigid, you’ve got to start on this day, and if you don’t stay on the top of the content, you’re going to sort of fall behind. It’s going to be sort of go at your own pace now, which would be really helpful, because some users were telling us they were overwhelmed. We’re also going to start providing targeted content for users. This is, just in the very beginning, is for CGM users. It will get much more sophisticated, I imagine, in the future, but just having a guide present information around how to get the most value out of your CGM is going to be really powerful. In the past, we’ve kind of just assumed anyone starting the app has a CGM, but of course, that’s not the truth anymore. We’ve got people that will come in and maybe they will have it sitting in the corner for a while, and we need to know when to surface that content to them.
(37:03):
Lynette has actually done an incredible job, and posted like 20 videos in one week, so let’s all give Lynette a big round of applause. She really was working hard, and we know that this content’s going to be super important for us moving forward. The feed will be a little bit less sequential now. You might just consider that in the past, the content feed was almost like a book, and you had to read a page about specific content as you were going. But now, we’re going to be able to have these things like what we were calling core looping, where we will be able to sort of insert something like a prompt to exercise, regardless of how long you’ve been in the experience.
(37:37):
One of the reasons we would want to do that is in case, if you are on this path where you’re going at your own pace, but you’re going incredibly slow, you’re never going to see some of this other content, so we need to find a way to be able to organically integrate it, and we think this is going to be really helpful. I hope you guys are enjoying Mike’s video content, too. I personally am. He’s always so positive, I feel like he’d be an amazing spin instructor. As I mentioned before, cohorts are blending. One of the requests we had in Beta 1 was having more advanced content, but some users actually felt like it was too complex. I think what that suggests is there’s a broad variety of different kinds of users, so we’re going to have a ton of content that probably falls more into the 201 category, and we’re going to start leaning more into an action layer with CGMs, where we’re asking you to do very specific things and then observe your response to that food.
(38:36):
Happiest explainers content. Oh yes, in terms of the guides. As I said before at the beginning, we’re going to start bringing the guides together here. Sonja will become our lead guide. She will be developing sort of the deepest connection with users, represented by this giant meditation picture I have on the right here. Stacie is going to be transitioning her users to Sonja, she’s going to sort of wean them off her content, with more of a focus on regular recipe updates. Mike D is our activity and exercise guide, and as I mentioned before around Lynette, she’s going to be helping users with getting the most out of the experience, like CGMs and so on, just figuring out how to use this thing. I think that’s it. I’m looking forward to seeing what happens in Beta 2.
Karin Nielsen (39:19):
Hey, everyone. Tough act to follow, as always, Alan. Next slide, please. Today, I’m just going to give you guys a quick update on what growth product and engineering will be focusing on for the next month or so up, until that 5th of June soft general availability date. We have two high-level priorities, the first is smoke testing, which is what I’m going to tell you guys about today, and the P2 is to prepare our signup flow for the monthly subscription and CGM optional experience. One of the benefits of running these smoke tests is that those two priorities actually kind of converge. As we are starting to experiment with our smoke testing flow, we’re actually also building out those components that we can then reuse in the GA signup experience, so we’re kind of getting two-for-one on that. Next slide, please.
(40:18):
In terms of the objectives for smoke testing, so just a quick reminder for those of you who might not be familiar with the term, smoke testing just means that we are putting A/B test experiments out there, but they don’t result in an actual conversion event, at least not a conversion event in the way that we’d normally consider it, which is for somebody to actually make a purchase. What we’re trying to do is to mimic a full-on Levels positioning and sign-up experience to the best extent we can, without actually taking the card details, to really gauge intent and get some solid data points around expected conversion rates and things like that going forwards.
(41:01):
The four priorities for us right now are that we want to test our positioning assumptions ahead of general availability. Obviously, we’ve changed our product experience slightly, so we need to make sure that we figure out how to actually describe that to our customers. The second is that we want to understand the relationship between different price points and how that then impacts our signup conversion rates. That is, actually, I guess a twofold problems that we’re trying to solve. The first is understanding how to position pricing, and then the second is the actual price point that we want to test. We also want to learn how different audiences from different channels respond to our new value proposition. This is something I’ll give some more context on in a second, but we have to kind of appreciate that we spent the last few years building a very substantial and specific audience, which is effectively culminating in 250,000 monthly unique visitors to our website.
(42:06):
That audience is very much primed for the product that we’ve been selling up until now. One of the goals of smoke testing is that we actually understand how different audiences respond to our new value proposition, as well, and particularly those who are not as aware about CGMs and the experience that Levels offers, or has offered until now. And then finally, we also want to surface customer insights for the product team. I’ll show you some really interesting results in a moment, but we can use these smoke tests to kind of get intelligence that can help the product team to shape the product experience in real time before we actually go live with new features, so that’s also an objective from running these tests. Next slide, please.
(42:58):
The very first smoke task we ran, we shipped last Friday and ended it on Monday, and this was really all about positioning. Before we can really get into the detail of testing price points, we need to make sure that people actually understand what we’re asking them to pay for. The four hypotheses we were testing here were that a digital-first product for metabolic health resonates with our current audience, because that still remains our primary audience, that Maureens are the primary audience for this product, that users are willing to pay around $15 a month for a digital-first product, and that users are also willing to pay for a CGM-enabled experience, but are a substantially lower, almost half of the price point, that they would have been able to sign up to currently. Next slide, please.
(43:50):
I just want to explain how we designed these experiments. Smoke tests can be designed in any way, depending on what the objective is, but specifically for the ones that we’re currently running ahead of general availability, there are really three components to these. The first is deciding which traffic we want to point at the experiment. There are three main traffic sources that we’re interested in right now. The first one is homepage traffic, and the main reason for that is because that is our current baseline that we’re measuring our Levels classic signup conversion rate off, so that gives us a really accurate indication of what our existing Levels audience that comes from all our referral sources, like partners and search and various others, looks like.
(44:41):
We can also point blog traffic at these experiments. The benefit of doing that is that we can get really specific. Let’s say we wanted to test a weight loss-specific positioning setup, then we’d be able to add links from our weight loss-related blogs that point specifically to that experiment, and that means we’re effectively tapping into an audience that’s already interested in primed for that experience. And then, finally, we can also look at paid. The benefit of pointing paid traffic at this kind of experiment is that we can kind of assume that those people are the coldest traffic sources. These are people who aren’t in our orbit already, and we can also get quite specific at the Facebook-level with selecting what audience we want to see what experiment.
(45:32):
For the purpose of this first smoke test, we primarily are focused on home page traffic, and that was also, number one, because we wanted a good baseline to compare against Levels classic, but also because it’s just a lot of work to actually set up these different traffic sources, and we wanted to give ourselves the best chance of shipping this first experiment quickly. The second component is the landing page, and specifically how we position different elements of our value prop. You can see here that we had a slightly different H1 to what we’ve used previously, and instead of using kind of fluffy marketing language here, we’re trying to be very specific about what we actually offer. We are leading with the fact that we’re now offering a plan, and it helps you get healthier based on your body, because we had a hypothesis that people who are in our current orbit are still very much here for the data and for the biofeedback.
(46:32):
You can see that in the H2, we then talk a little bit more about how we actually help users to achieve that, and we talk about things like coach-led plans and real-time biofeedback. And then finally, we also have a signup funnel, but this is not the same as our current signup funnel. For one, we can’t push people through a kind of IRB flow, that doesn’t really exist, and secondly, we don’t want to take card details and have all of those things on record if the user doesn’t actually convert. What we ended up doing was just capture their email if they convert at the point where they submitted their address, which is the highest intent point in time that we can use as a conversion event in the absence of taking payment. And we then also invited those people to take an additional survey with a bunch questions to help us understand who they are and to validate some of those audience-based assumptions, with the goal being that we can then tighten up our positioning the next time round. Next slide, please.
(47:44):
I’m going to share with you a little bit about what we learned about this traffic. When we ask people which primary health goal brought them to Levels, most of this probably won’t surprise people, but we only allowed them to pick one option here. People are here, from our current audience, to avoid disease and to lose weight. I think this very much lines up with our existing assumptions and is actually the sort of messaging we already use in our H1, as well. This is great, and this is very much what the product team is currently building towards. And we can then see that there’s some other objectives, as well. But you can really see that very strongly, 65% of people are here for the thing that we’re building towards today. Next slide, please.
(48:35):
The second question we asked them was, what is your biological sex? This one is really interesting, because we had considered when we designed this experiment whether we should lead with some very Maureen-centric imagery and positioning, but we decided to go with something neutral, because we actually wanted to understand first what the skew from our current audience actually is. I think it was David in his earlier presentation who mentioned that something like 40% of the beta cohort were men, and you can see that that’s actually very representative of what we’re seeing at the top of the funnel, as well. The takeaway here is that for the next experiment, we also want to make sure that we can experiment with the sort of hero imaging on how we tweak the look and feel of things, but we definitely want to make sure that we’re not leaning too heavily into one demographic over another at this early stage. Next slide, please.
(49:35):
This one, we did offer people multiple choice, because rarely are people going to a product for a single feature, and we wanted to understand which feature excites them the most, primarily so that we could figure out the information hierarchy on the landing page so we can prioritize the things that people are the most interested in up top. What we can clearly see here, which isn’t surprising given the nature of our traffic sources, is that the vast majority of people are still coming to Levels for the CGM experience.
(50:09):
Obviously, that might change once we start looking at other colder traffic sources, but I’m actually more interested in the top three answers here, because when you combine the first response with metabolic panel testing and all my health data in one place, I think it strongly implies that people are coming to Levels for our secret sauce, which is the fact that we offer you data to disambiguate this process towards achieving your goals. At the moment, we’re still very much leaning into that messaging with the first iteration of the experiment, as well, and we’ll probably continue to do that going forwards.
(50:50):
Next slide, please. This one really surprised me, actually. I had expected that more people, by this time, would be familiar with CGMs and would have tried one before, but 87% of people haven’t used one at all. Thankfully, nobody said that they’re not interested in using one either, so that’s good news. What this tells us is that we definitely have to do a lot more education on the website and at the top of the funnel to help people understand what they’re buying before they make that purchase. That’s very much reflected in the feedback we’ve had through our surveys and various other user research interviews we’ve been conducting, that people still don’t feel that they have enough information in many cases to make a informed buying decision.
(51:40):
I think when you look at all of this data combined, I’m starting to get from the view that we have this spectrum of people in our audience who, on the very far end of the spectrum, historically have been biohackers, so those were our early adopters, and we then have the CGM-aware people in the middle, who have been in our orbit for quite a long time, maybe they’ve discovered that they have a problem or a family history problem that might impact them later, and then they actively go out and search for answers and solutions and invariably end up on our orbit from listening to podcasts and reading our blog and all of those things, because obviously we set out to educate those sorts of people about CGMs. And then at the other end of the spectrum, we have the CGM-naive people. Those are the folks who hit our website and didn’t even know that that’s what we offer.
(52:36):
I think the majority of the people who have replied to the survey that we put out seem to be in the middle. They seem to be CGM-aware, but not at either of those extreme ends of the spectrum. Next, slide please. What are we going to test next? We have enough time between now and soft general vulnerability on the 4th of June to run one more smoke test. We are going to take all of the learnings from this first experiment to tighten up our positioning, and we really want to lead with the monthly value prop up top above the fold to make it super clear how cost-effective we’re now making it to get started with Levels.
(53:21):
We are also going to simplify the buying options. We’ve set up a Help Scout widget in the signup flow to get some feedback on what people were confused about, and it turns out that our sort of starter and pro positioning just confused the hell out of people, because they still thought that the $200 was a monthly cost. We are going to simplify the signup flow to essentially have them on a subscription by default and then just have them add a CGM to cart if they want it, and we think we’ll get a much stronger signal of intent about the propensity to buy the subscription-only versus the propensity to buy the subscription with the CGM on day one.
(54:04):
And then finally, we are also going to test $15 versus $30 price points just for the monthly app subscription. What we’re really trying to get here is some signal on what the optimum price point is and how that impacts our modeling as we go into general availability. That is it for today. We’ll be sharing more updates on how these experiments are running in the next few days.
Josh Clemente (54:30):
Really awesome update. Thank you, Karin. Lots to be excited about and for right there. Quick hiring update. We have added a new backend software engineer position, the first one we’ve opened in a while. We’ll be looking for someone to support on all the backend work that we have ahead of us, so check out that role. We also have our standard information submission at Levels.link/careers, so if you know someone in your network, Nicole just shared a link in the chat, and if you’re watching this, head to Levels.link/careers if you’re interested in that role, or forward it to someone else.
(55:08):
Individual contributions, we’ve got three minutes here. Go ahead and raise your hand if you want to share something using the reactions at the bottom. I’ll go ahead and kick this off. Let’s see. Professionally, I am really excited about the transition into Beta 2, hitting that on-time. I think I said this last week, but continuing to be very excited by that, and eager to learn. These learnings that Karin just shared are, as Sam said, they can be interpreted in a number of ways, and we have to continue to get more information to really understand exactly where we’re heading and how to tweak the trajectory. I’m glad we are doing this incremental and very fast cycle time, that’s just so important. It’s so important, and we’re doing it right. Let’s see. Personal side, I’m not excited about 90 degrees and 90% humidity, but I am hell-bent on getting back into my exercise routine, so hold me accountable. Anyone else? Chris.
Chris Jones (56:21):
On the product side, so one, just want to thank Josh and David for the calls to support. As I kind of had in my async, the last month’s been pretty rough, but I really appreciate the partnership, working with product, working with Inge, to identify issues, come up with solutions, and try to keep our customers on a happy path. It’s part of us moving so fast, so I just want to thank the team for everything that they do every day, and they should be thanking me and Miz that we’re not in the queue bringing the happiness scores down.
(57:03):
Taylor and I had a call with Truepill this morning, and I was kind of asking them generally, because they support several CGM manufacturers in the space, and I was kind of asking them their thoughts around the market, the players, are they seeing the same issues we are? And at the high level, they definitely confirm price is a sticking point across all the players in the space, and they also said Truepill views Levels as the most innovative company in the space. The things we’re doing around the IRB, things like with other manufacturers’ software-only, like their point of view is it feels like everyone else is almost more like copycats. It just reminding me around the quality of the team, us taking shots at goal, and continually to try to innovate on multiple fronts to correct it, so it’s pretty fun.
(57:55):
And then personally, on Sunday, I’ve got my first running race in about three years, so doing a Spartan Trail half marathon, basically climbing up the side of the mountain and back down, so my claws are going to be absolutely shot at the end of it, but it should be fun to get out there and get dirty.
Josh Clemente (58:12):
Awesome. Spartans are a blast. I think great reminder on the competitive front, just like I love how we stay kind of inwardly focused, we’re obviously doing market research and trying to understand what’s out there and how to position and price and all that stuff’s important based on precedent, but I really appreciate that we do the Levels thing. We don’t try and copy what’s out there. All right, everybody, awesome week. Lots more to do and learn, and appreciate everyone, and have a great weekend. We’ll see you next week.