April 7, 2020
Ep. 48 – with Michael Lawson & Woody Austin

Ep. 48 – with Michael Lawson & Woody Austin


All of our customers really should own their
data, own their relationship with their customers. Welcome to Honest eCommerce, where we’re dedicated
to cutting through the BS and finding actionable advice for online store owners. I’m your host Chase Clymer, and I believe
running an online business does not have to be complicated or a guessing game. If you’re struggling with scaling your sales,
Electric Eye is here to help. To apply to work with us visit electriceye.io/connect
to learn more. Now let’s get on with the show. Alright, everybody, welcome back to an episode
that I’ve been looking forward to for a very long time because I keep getting dodged by
my partners over there. But finally they came through I had to track
them down at KIaviyo Boston the other day, so I finally got some commitments. Welcome to the show, Woody and welcome to
the show, Michael from the data science team at Klaviyo, my favorite email marketing platform
What’s up, guys? Hey! How are you doing? Awesome. Cool. Excited to be here. Oh, yeah. And I’m excited to have some actual
scientists that I’m talking to on the show. That’s pretty cool. Yeah. Awesome. Alright, so what are you guys doing
before you ended up at Klaviyo? Yeah. I’m Woody. Before Klaviyo, I was in
grad school for the last six years at UT Austin for computer science and I was specifically
working on high performance computing and machine learning algorithm development in
recommendation systems. Alex Ikonn
Yeah, this is Michael. I was also in grad school. Also for six years. I was at the University
of North Carolina, doing an algorithm development and experimental design mostly in medical
research. That’s so cool. So what made you guys get
into the eCommerce space? What appealed about Klaviyo? Well, I was kind of looking for a classic
computer science job and I thought that eCommerce was kind of a nice place to go because you
get to work with so many customers who really see the product of your work. And after being in that ivory tower for so
long, I was really wanting to be able to interact with real people and develop something real.
I found Klaviyo because I moved up here to Boston be with my girlfriend who’s going to
school here. And I was kind of looking around for jobs. And when I stumbled upon Klaviyo I just had
fun interviewing and the culture was really great. We got to mess around with the product
a little bit. And I was surprised by just how nice it was to use. It’s somewhat similar for me. I was looking
for a little bit of a change after being in grad school. And one of the things I liked
the most about Klaviyo was the pace. Things happen at a very quick pace. Which can…
That was a nice change after. There will be times that I’d have to wait
to start on a project for three months because I couldn’t get access to a data set in grad
school because health data is just –for a very good reason– protected so strongly. Absolutely. So before you guys… As you guys
were joining the team, what were some of the features of Klaviyo that really stuck out
to you and be like, “Wow! This product is way smarter than people think.” Hmm, there’s a lot. It’s tricky. I think the
thing that really stuck out the most to me was being able to automate in an intelligent
way. So to set up a flow and say, “Only people who have bought a certain type of product
should receive this type of experience.” That’s exactly the sort of personalization
was at the core of what I studied in grad school and medical research. And it was interesting
to see a lot of the same ideas getting applied to guiding customers through their experience
with your brand. Yeah, I was really impressed with the segmentation
engine at Klaviyo. It’s dynamic and you can get really specific and it’s fast. I was also
really surprised –coming from the machine learning world– just how much powerful personalization
you can get out of Klaviyo without even diving into the data science features to begin with.
So I thought the tool is just laid out really nicely. And yeah. Oh, yeah. It is amazing. We used a few different
platforms for clients previously and then once we found Klaviyo, we’re like, “Oh, man,
this thing has it and it’s easy to use.” I think that’s the best part. With some of the
more legacy automation engines out there –Infusionsoft is one that comes to mind– it’s kind of scary
to get in there and understand how it works. With Klaviyo, it’s very user-friendly. Yeah, we try to make it that way. (laughs) (laughs) Mm-hmm. One of our goals. So what are you guys responsible for at Klaviyo?
What’s your day to day? So I guess my official title is machine learning
engineer. On the data science team, we are involved in the entire development of the
product. So we actually sit down and act as our own product managers. We get to decide
what the direction of each of our groups does. I’m working specifically on the product recommendations.
The recommendation systems for populating the product feeds and everybody’s emails and
trying to make them be as good as possible for the users that get them eventually. Yeah, I’m working on improving the experiment
experience in Klaviyo. So if you want to run an AB test in any area of our software, we
wanted to make it easy to use and as powerful as possible. We’re working on improving that. That’s so cool. So there’s some few features
that you guys shared with me in these notes. Let’s talk about the first one here. I’m sure
that some of the listeners that have Klaviyo aren’t even using this. Smart Send Time. What
is it? How does it work? And why should I be using it? So Smart Send Time, I guess, as the name indicates…
The first thing I’d say about it is that it’s smart. (laughs) It is a way to intelligently
target the time that you send your emails to when your customers will actually want
to receive and open them. The way that we do it is a little different from how kind
of the industry standard approach has been. The industry standard approach has been pretty
much (like), “Look at historic data, see when your customers have opened their emails and
go with that time.” There are some inherent issues with that.
It brings in some survivor biases. And because of that, Christina, on our team, –who developed
this algorithm– came up with an approach where, “Well, this is an eCommerce platform,
let’s actually test what works best. Let’s do science to figure out the right time to
send your emails.” And what we do is we test all the 24 hours
of the day against each other, so that you can actually see if 7:00 PM is the best. It’ll
outperform everything. It won’t just outperform 6:00 PM. It won’t just outperform 3:00 PM.
It will outperform all. A couple of other things that I think are
really cool about this algorithm is, Christina actually started out by trying to do what
the rest of the industry did. And she got the pretty graphs that everybody has showing
what your open… What your send time should be, what the open rate should be. And then
she just found out that in practice, we didn’t get any lift at all from those. On open rates
or click through rates or anything like that. So she went back to the drawing board and
came up with something completely different that actually does provide lift to our customers. In terms of “Why should we use it?” I guess
the proof is pretty much in what’s happened after it’s been out there in the world. Smart
Send Time, was released… A little over a month ago, I think? …a little over a month ago, I believe, in
web release. And since then, over 1000 customers have used it and they’ve seen a median 8%
lift in open rates. Yeah, and the median there is really important
because it means that we’re not letting huge outliers drag us one way or another. So when
people report the average lip, that isn’t quite as true or as telling what the median
does. That’s awesome. Support for today’s podcast comes from our
friends at Simplr: a new way to staff 24/7 sales and customer service on your eCommerce
store. It works with your existing email and chat
platforms, so setup is quick and easy. Simplr’s network of on-demand, US-based, Simplr
specialists are standing by to answer your customers’ most common questions. Set it up for free today and then turn it
on or off depending on your customer volume. You only pay $2.25 for every resolution. There
are no hidden fees, contracts or minimums. Close more sales with Simplr by staffing your
email and live chat around the clock with Simplr specialists. Start your free seven-day
trial at simplr.ai/honest. So when I’m going to send a new email blast,
first of all, I’m going to be split testing my subject line for sure. Woody and Michael
Mm-hmm. But is there any extra work that I need to
do to use this Smart Send Time as well? So yeah, it’s pretty easy. Any other users
could have used Klaviyo before, all you do is you go in and you select your scheduled
send time, but now there’s an option that I… I forgot what it’s actually called in
the UI? It’s like Spread Send Times… Uhm, Exploratory Send. When you’re discovering
your specific Smart Send Time, you choose Exploratory Send. That’s when they do the
scattershot across all 24 hours. And then once you’ve found your best send time, you
can just choose Focused Send at that time. And then it also splits a few emails into
a few hours before and a few hours after so that if characteristics of your audience change,
–Maybe your optimal send time was 7pm at a certain point, but now it’s 5pm because
your customer base has shifted their preferences a little bit– the algorithm could still pick
up on that. Yeah, and that’s really cool. At Klaviyo,
we’re trying to make things really easy to use. That means, we’re still exploring for
you so if there’s something does change, you don’t have to worry about it. You can kind
of set it and forget it. That’s super cool. Another really cool feature
that has just come out recently is “predicted gender.” How does that work? That’s just so
cool to me. Yeah, so it’s actually pretty straightforward.
We just look at the first name, and then correlate that with the way that names are distributed
across the entire US. So actually, it’s kind of funny because on the data science team,
we have a wall where every single member of our team, we printed out our Wolfram Alpha
entries for our names, and you can see the distribution for our names. But we use that
same kind of thing. We just look at the distribution across the
US and we say, “Are you likely male? Are you likely female? Or are we not sure?” And so
that allows you to segment your audience into those three segments. And you can target either
male and unknown, or female and unknown, or just male, just female. And it really helps
you target your message towards your audience and make them feel like you’re really talking
directly to them. And one thing that’s nice about this feature
being in Klaviyo is, you get everything that’s already there. So if you are, at some point
in your process, asking people for their gender, you can also have that override our predictions. Because if someone says they’re male, you
know. So you can just build that. You can set that up in your segmentation logic as
well which is a nice drag-along feature that you just get for free. Yeah. And you can add other segments that
you’re already using. So if you want to only send to your VIP customers who are likely
male or who have identified as male, then you also can do that, In terms of how to use it, there are definitely
some best practices. I think there are some very obvious “Don’ts” which… It’s almost
not worth even spelling enough. Obviously, don’t use any data stereotypes that’s going
to turn off your audience. You don’t want to do that. Don’t assume too hard either. I mean, I personally…
My name is Michael. It’s a very common male name. It’s one of the most common. But I’ve
met female Michaels. If you just assume because the name is likely male and you call them…
If you say sir, in your salutation in the email, that might be going too far. What we’ve seen some customers do is just
try to personalize some of the content and make it more relatable. So certain colors
that are chosen or the choice of, “Do you use a male or female model in a picture to
try to get them see themselves in the picture?” Things like that. Oh yeah, that’s that’s some great advice.
So while it is predictive, it’s still… A computer is just giving its best guess. (laughs)
So I wouldn’t… I hadn’t thought about it that hard. So, what are what are some of the
other data science features that you think people are underusing on the platform right
now? I’m a little bit biased since I work directly
on the recommendation systems but when we look at the analytics for all of the emails
that are sent across Klaviyo, very few people are actually using Personalized Product Recommendations
in their emails, or at least through Klaviyo anyway. So, I would say that one is really
underutilized. I know a lot of people, whenever they’re sending
campaigns, are going to want to tell a story so it makes sense that they wouldn’t necessarily
use personalized recommendations there. But within flows, if it’s like an abandoned cart
flow, then we can send them products that would be personalized for them. Yeah, that’s definitely a good one. Another
one that I think flies under the radar a little bit is Expected Date of Next Purchase. And
I think a really good illustration of that is, people tend to buy in their own case for
certain types of products. So you might have a customer that, very regularly
buy, something from you every three months. And if you set up your Customer Winback Flow
to email them after two months, you’re pressuring them and they won’t like that. If instead,
you know that they’re predicted date of next order is three months from now, then you can
wait until that point, give them a little extra time, make sure that they’re actually…
Maybe they were on vacation. Being able to actually use something predictive
rather than just a rule that you came up with –maybe in a few seconds while you were setting
up the flow– I think that’s also a very powerful one. And also super related to that is the Churn
Risk as well. So you can also predict how likely your customers are to churn. And you
don’t really need to badger somebody with the Customer Winback Flow if they’re highly
unlikely to churn but maybe whenever they’re getting more into that orange or red category. And now within my customer settings in Klaviyo,
am I setting up automations that trigger when people get into certain segments or is it
more of a, “You got to think a little bit more outside the box of how you’re going to
target these people.” How does that work? Over expected date of next order, for instance,
that’s actually a setting in your flow. So that’s a trigger that can trigger a flow.
It’s when someone reaches their expected date of next order, send them through this automated
sequence. I think for things like Customer Lifetime
Value and Churn Risk and that kind of thing, you can segment on some of those properties
within Klaviyo. And then, I guess, for product recommendations, that one should be pretty
straightforward. Whenever you’re creating your template and you go to the product block,
you can click “Use Personalized Recommendations.” That’s awesome. Let’s be honest today. All of your customers
are going to have questions. What are you doing to manage all those questions?
Do you have a helpdesk for your business? One of our sponsors of today’s episode is
Gorgias. Gorgias is the number onerated helpdesk for eCommerce. It integrates seamlessly with
Shopify. We have installed it on a bunch of stores. It’s also used by brands like MVMT
and Rothy’s. What it does is it takes all of your customer
insights and information, brings it into one amazing dashboard so you can solve their problem
as quickly as possible. If you want to give Gorgias a try, visit gorgias.link/honest to
get your second month free. So I know that you guys are big fans of the
product. Does Klaviyo have any particular philosophy towards data science that you guys
are happy about? Yes and yes, I guess, is the short answer.
So one of the things that… I guess one of the big takeaways from Klaviyo Data Science
is we actually are doing science here. As we talked about Smart Send Time, we didn’t
just settle for something that would work okay or something that is done by many people. We found the approach that actually had demonstrated
value and actually helped people. And that’s a big part. The rigor that goes into making
sure that our methods work. Yeah. I’m kind of piggybacking on that a little
bit. We really hire good people. Like, Michael has a PhD in statistics. Christina got very
far into grad school and her degree in statistics and then I studied machine learning for a
really long time. But you know, even though we aren’t the bosses, our bosses really listened
to us and they want to get the math right as well. So, they aren’t just going to put out some
subpar feature just because the industry wants it, we’re going to make sure whatever we put
out actually works. Another part of our philosophy that I really like is that, we try to focus
on automating the parts of marketing that humans aren’t good at.
Humans are really good at coming up with a narrative. Humans are really good at talking
to each other and communicating with each other. So we want to try to make it easier
for our customers to reach their audience. Reach their audience in a way that makes them
feel like they’re being talked directly to, not just being sprayed with like a mass email.
And so I really like working on this part of the platform as well. Absolutely. Now, machine learning is like
a really big buzzword as of late, along with AI or… There’s a bunch of buzzwords in that
same vein. Where do you think Klaviyo lands as far as the technology itself? Is it actually
artificial intelligence or what is it? I’m curious about your guys’ take would be on
that, coming from a scientific perspective. Yeah. We approach it in an ad hoc manner,
depending on what project we’re working on. So we want to have really rigorous statistics
on a lot of our projects and so those tend to be more on the pure statistics side of
things. Or I guess you could call it data science. For things like machine learning
or… Sorry. For recommendation systems, those typically
fall more in the “machine learning side” of things. And specifically for a recommendation
system that’s really hard to perform statistics on because you’re showing somebody a product
and you aren’t sure whether they’re going to like it or not. And so your historical data, just like with
Smart Send time, really doesn’t matter at all. So you have to really throw the kitchen
sink on it, but do it in a principled way. And yeah, I also had one more comment about
the previous question on our philosophy I kind of forgot to throw in there as well.
But now I’m blanking on that too. Sorry. (laughs) I’ll come back to that. I’m sorry. (laughs) I guess following up on that thread, I think
one of the things about, I guess, calling the platform AI… First of all, I personally
think that the term AI is overused in many places. And it’s a very buzzword thing right
now. And I think many things are called AI that maybe shouldn’t be. But in terms of Klaviyo as AI, I would say,
in some ways maybe (and) in some ways, no. Again, the platform is about making human
connections. So the AI parts –and this is going to sound very similar to what what he
said– are about doing the things that are hard for a human to know, like, “When should
I actually send my email? When should I know to stop my AB test?” Because I’ve reached
a point where I actually know what the result is. And it’s really enabling humans to use
their intelligence more effectively. Absolutely. And I just want to follow up on
that. Signing up for Klaviyo or signing up for whatever software of the month is buzzing
isn’t going to make or break your business. You know what I mean? If you have a good email strategy, and you’re
coming over from a less feature-rich platform, (then) yeah. I think Klaviyo is going to do
it for you and do it well, and probably make you more money. But you still got to do the
work. You still got to write these flows. You still got to write these campaigns. The
machines aren’t doing all the work. They’re just doing the really, really technical stuff. Yeah, I think one of the best examples that
I have (is) a machine is not going to be writing your emails better than your marketing team
or you yourself can write your emails but… Or at least if it is, that’s in the… That’s
a little science fiction, and it’ll be a while, right? (laughs) At least if you want to do it well. Even if
you… Even if the machine could generate it for you, –because there are algorithms
that can do that– you’re gonna have to go in and you’re gonna have to edit it and it’s
probably not going to be the right tone for you. But something that I’m sure everybody would
love to be able to do is to know every single one of their friends customers and really
like target the information to them. And so we can find the groups of customers that are
really similar, that have kind of obvious characteristic where they might like a particular
product from you. So that’s the kind of stuff that we’re trying to start with automation. That’s awesome. And now, is there anything
else that you guys see it’s coming out on the horizon here shortly that is going to
get you guys excited or features that are going to come into Klaviyo? I know SMS dropped
at your event in Boston the other day. Was there anything else that you guys can share? Yeah. Actually, both of the big announcements
at Klaviyo Boston were… I’m super excited to work on them from the data science side.
So first of all, there’s SMS. I think that opens up a bunch of really interesting questions. I think AB testing there is really important
because the space constraints are so strong in SMS and really finding ways to squeeze
something out of every single character is going to be crucial. And also it opens up
the question of, “Should I be sending an SMS or should I be sending an email and what’s
the best way to even know that.” So there’s a lot of just channel testing, like, “What
channels should I be using for this message?” That is going to be really cool to work on,
I think, and then customer analytics. Customer analytics are going to be… I think
that’s going to be huge because it just gives… It’s a feature that allows humans to learn
something that it is hard for them to know. For instance, “How are people who came to
me from this particular channel doing?” Like, “What are they buying? How often are they
buying?” It makes it possible to just understand things like that. And I think when you’re able to answer questions
like that, you can much, much more effectively use the data science features that we already
have and think of. We’ve already got some that were thinking of that are going to piggyback
off of that as well. Yeah. And it’s nice as data scientists, too
because we also have access to this new report builder and analytics tool so we’re going
to be able to leverage the data from our customers and our customers’ customers even more. Yeah. That’s fantastic. I can’t thank you guys enough
for taking the time to hang out with me today and get into the more technical side of –which
I’m not joking when I say this– one of my favorite apps to use in the ecosystem right
now. So is there anything that you want to say before we get off here? I did remember my other point that I was going
to make earlier (laughs)… …about the philosophy that we have. It kind
of goes along with the people who went to Klaviyo Boston. We introduced this idea of
Owned Marketing and Owned Growth and how all of our customers really should own their data,
own their relationship with their customers. And that really extends into the data science
side of things as well. So we aren’t sharing data between our customers. We’re trying to build the best models that
we possibly can. But those models kind of belong to our customers, our customers data
belongs to only that customer. And so we try to keep the idea of Owned Marketing throughout
the entire company and this really shows the culture out here. Nice. I guess I’d like to say, keep your eyes peeled.
I think there’s some really cool stuff coming from data science in the near horizon. Awesome. Thank you guys so much for the time. Yeah, thank you. Thank you. I cannot thank our guests enough for coming
on the show and sharing their journey and knowledge with us today. We’ve got a lot to
think about and potentially add to our businesses. Links and more information will be available
in the show notes as well. If anything in this podcast resonated with
you and your business, feel free to reach out and learn more at electriceye.io/connect.
Also, make sure you subscribe and leave an amazing review. Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *