0
SPEAKER_02
Welcome to the official Sastr podcast, where you can hear some of the best Sastr speakers.
0
SPEAKER_02
This is where the cloud meets.
0
SPEAKER_02
Hey, everybody.
0
SPEAKER_02
We re so close now to Saster Europa 2024.
0
SPEAKER_02
Me and the entire Saster sales team and over 3,000 SaaS and cloud enthusiasts will be together June 4th, 2nd, and 5th in London, England.
0
SPEAKER_02
Everyone from around the world will take over London.
0
SPEAKER_02
We re down to the last chance tickets.
0
SPEAKER_02
So use my code Jason20 to save 20% off the last tickets now.
0
SPEAKER_02
That s Jason20.
0
SPEAKER_02
Why pay more?
0
SPEAKER_02
Feel stuck in your talent search?
0
SPEAKER_02
Remote Talent is a leading job board for remote-first companies and startups.
0
SPEAKER_02
Visit remote.com slash jobs today to start building your dream team anywhere in the world.
0
SPEAKER_02
That s remote.com slash jobs.
0
SPEAKER_02
Get more when Northwest Registered Agent starts your business.
0
SPEAKER_02
They ll form your company fast and stand up your entire business identity in minutes.
0
SPEAKER_02
That means business-free domain, business email, website, hosting, address, mail scanning, business phone app, all within minutes.
0
SPEAKER_02
Visit northwestregisteredagent.com slash saster to start your business today.
0
SPEAKER_02
This episode is sponsored by our good friends at Pendo, the all-in-one product experience platform.
0
SPEAKER_02
Pendo brings together product analytics, guides, discovery, and replay, so you can improve your product experiences for every customer and employee.
0
SPEAKER_02
Try Pendo for free at pendo.io slash sastr.
0
SPEAKER_02
That s pendo.io slash sastr to try Pendo for free.
0
SPEAKER_02
Up today is a great session from Shopify s head of AI, Mike Tamer, who joined us at our first AI day.
0
SPEAKER_02
We re going to do a huge all-day AI summit at Sastry Annual this year in September.
0
SPEAKER_02
You should be there.
0
SPEAKER_02
And this was a gift.
0
SPEAKER_02
Together with Glasswing Ventures, they went all across how Shopify actually uses AI across their sales and products.
0
SPEAKER_02
So this is a great one for folks wondering how AI actually works in SaaS and B2B.
0
SPEAKER_00
Hello, everyone.
0
SPEAKER_00
What Mike and I thought about doing today is how does one think about AI adoption, both at an individual level in a business context, as well as the critical work that Mike is doing at Shopify and has previously done at Uber, Susquehanna, amongst others, as well as the cutting-edge work he does teaching at UC Berkeley.
0
SPEAKER_00
So by way of background, I m a managing partner at Glasswing Ventures.
0
SPEAKER_00
We are an AI originalist firm.
0
SPEAKER_00
We in fact founded Glasswing back in 2016 with a goal of investing in AI-native companies that solve real problems for the enterprise and cybersecurity markets.
0
SPEAKER_00
Mike, with that, would you like to do a little bit more on your background?
0
SPEAKER_00
I think I would be doing it a disservice.
0
SPEAKER_01
Well, I think you did more than enough, but just to recap, I m Ed Byrne, one of the distinguished ML scientists and running ML and AI functions here.
0
SPEAKER_00
Lovely.
0
SPEAKER_00
So let s delve in.
0
SPEAKER_00
The first part of this conversation, I will do a bit of level setting around what s going on with AI from an enterprise adoption and where we are in that cycle.
0
SPEAKER_00
An important level setting point is the fact that
0
SPEAKER_00
Today, there are many initiatives across large and mid-sized enterprises around AI.
0
SPEAKER_00
In fact, we tell our AI-native companies not to lead with AI when they re speaking to prospective customers because it s such a top-of-mind topic that it s a superficial indication of interest.
0
SPEAKER_00
But most of the engagement with few notable exceptions is still at that superficial level with early sort of measurements, if any, on the impact on the enterprise on an adoption basis.
0
SPEAKER_00
Now, we have seen a few green shoots, if you will, in this space, particularly, for example, what we saw with Klaviyo and the impact that they have had by bringing AI from a project siloed or customer interface type level usage to their core business.
0
SPEAKER_00
Where the automated 700 humans and over two thirds of customer success is now AI supported and generated and contributed 40 million to their bottom line.
0
SPEAKER_00
And it is because they brought it to their core product, to their core business.
0
SPEAKER_00
And as we expect in the mid and long term, the impact is sizable.
0
SPEAKER_00
But they are not worthy because they are one of the few companies and again michael talk to what shopify is doing is another leader in that category that have brought a i to the core business so.
0
SPEAKER_00
What is the mindset?
0
SPEAKER_00
What is the actual work that needs to be done around building the infrastructure to leverage AI for the core business and the highest impact?
0
SPEAKER_00
I would articulate it along three dimensions.
0
SPEAKER_00
One is software and the technology infrastructure and how, whether it s the debt, whether it s the tech decisions, whether it s the data availability, et cetera, but getting the enterprise or the facets of the business to be AI ready.
0
SPEAKER_00
The other piece is really around leaning to the data.
0
SPEAKER_00
So one is the algorithmic side of technology side.
0
SPEAKER_00
One is the data that you re going to feed.
0
SPEAKER_00
And oftentimes there is a almost naive assumption that the data will be easy to leverage.
0
SPEAKER_00
Oh, it s an asset and we ll use it and leverage it.
0
SPEAKER_00
And it turns out that oftentimes is the biggest problem.
0
SPEAKER_00
And thirdly, culture, culture of adopting AI.
0
SPEAKER_00
And again, we ll delve more into this as we proceed in the discussion.
0
SPEAKER_00
Where are we heading therefore as we try to adopt AI in the enterprise?
0
SPEAKER_00
I have a term that I use called ambient AI, which is basically that we, let s think of ourselves as professionals and productivity sort of driven professionals, we will be surrounded in leveraging AI in this co-pilot dynamic.
0
SPEAKER_00
or want to wrap one s mind around, what does it mean for me to be leveraging ambient AI?
0
SPEAKER_00
If the input is you, the business, individual leader, contributor, your activities, and your environment, and you think of that as the input into ambient, into your existing AI tools that you ve surrounded yourself over a co-piloting dynamic, the outputs are insights that you wouldn t have been able to derive on your own.
0
SPEAKER_00
And secondly the automation of the mundane the low productivity highly repeatable to free one up to be more valuable.
0
SPEAKER_00
Says we think about our roles as professionals and what will make us successful going forward.
0
SPEAKER_00
What s considering that in addition to the well-tried high impact productivity that people and the professionals will have one leg up.
0
SPEAKER_00
If in fact they are leveraging the tools if in fact they are intelligent business workers not simply in the intellectual sense but ever leveraging the intelligent tools.
0
SPEAKER_00
And as we shift to the enterprise, it s really these topics of data, infrastructure, and culture.
0
SPEAKER_00
And I will use this layout and framework to turn it over to Mike to talk about the impact starting with the data.
0
SPEAKER_00
My first question stated here, in fact, how do you know if you have the right data?
0
SPEAKER_00
And is data really, the claim that I made is oftentimes the biggest bottleneck and the biggest problem.
0
SPEAKER_00
And for this business audience, not necessarily a technical audience, how do we think about that change first?
0
SPEAKER_01
Absolutely.
0
SPEAKER_01
So you couldn t be more correct that data is often the messiest and the time that you think you re, while we have the data, why don t we just get to the building the models?
0
SPEAKER_01
When in reality, it s one of these iceberg cliches where most of the time you spend is on
0
SPEAKER_01
gathering the data and cleaning the data and making sure that it s ready, that it s of the right quality.
0
SPEAKER_01
And even once you begin the modeling, you re not done.
0
SPEAKER_01
Modeling is really an iterative process where you train your models.
0
SPEAKER_01
You see what signal you re getting from it.
0
SPEAKER_01
You see how good your performance is.
0
SPEAKER_01
And this speaks to a measurement and making sure that you have the right measures.
0
SPEAKER_01
And of course, every different ML technique is going to be slightly different, but very generic and generally.
0
SPEAKER_01
Um, and certainly when you hear about LLMs and a lot of the modeling techniques are invoked.
0
SPEAKER_01
Um, it s trial and error, right?
0
SPEAKER_01
A model will see an example of this is what information we have, and this is what we re trying to predict.
0
SPEAKER_01
It ll take the information that you give it that of what we have ahead of time and make its own prediction.
0
SPEAKER_01
And then look at how far it missed, right?
0
SPEAKER_01
How far is what it tried to predict from the truth of what it should have predicted.
0
SPEAKER_01
And then it ll take that error, right?
0
SPEAKER_01
We hear about failure is the key to learning.
0
SPEAKER_01
This couldn t be more true than in machine learning.
1
SPEAKER_01
learning, machine learning literally takes that error and with a bit of math tries to adjust all the little parameters, all the little calculations that it makes along the way from input to output in order to make that error just a little bit smaller the next time.
1
SPEAKER_01
And this leads into a lot of the things that you have to do with your data as best practices.
1
SPEAKER_01
So if I take all of my training data and I teach my model with my training data to close that gap between its predictions and what it should have done, then I might be overshitting.
1
SPEAKER_01
So overfitting is if your, if your model has taken that time to close the gap between its predictions and what it should have done, maybe what it s done is just learn for those specific examples that you gave it to learn from.
1
SPEAKER_01
Right.
1
SPEAKER_01
And so it has overfit to those specific examples.
1
SPEAKER_01
But we don t want to build models that can just take the inputs you give it and give the outputs that you told it to learn.
1
SPEAKER_01
We already have that data.
1
SPEAKER_01
What we want it to do is to learn the pattern.
1
SPEAKER_01
And to learn the pattern means we have to do all of these different things in order to constrain the model in various ways so that it predicts not just on the data that you re giving it, but it also predicts effectively on the data that it s never seen before.
1
SPEAKER_01
And we usually call that
1
SPEAKER_01
validation data and ultimately test data in different stages for figuring out how is this model going to predict accurately out of the training sample.
1
SPEAKER_00
And so if I am a P&L owner in XYZ business and I ve had this data for 20 years and Mike s team is helping me sort this out, why is it taking them multiple weeks?
1
SPEAKER_00
What s so hard about it?
1
SPEAKER_00
What s so hard about it?
1
SPEAKER_01
And there s two parts of this, which also speaks to, I think, the culture, right?
1
SPEAKER_01
Especially in a world where you can talk to an LLM and it seems like it s giving you right answers.
1
SPEAKER_01
And you want to say, well, you only live once.
1
SPEAKER_01
Let s just use whatever the answers are.
1
SPEAKER_01
And I m not going to name any deployments that have happened in the last several months, but we all know news headlines where maybe that didn t work out so well.
1
SPEAKER_01
So having the culture of checking your results and making sure that you have the right metrics to evaluate, whether it s an LLM or a more traditional model that s not generating text or generating images, but just estimating numbers or estimating classes of whatever the task is.
1
SPEAKER_01
Having that culture where you are focusing on what are your metrics and you re evaluating those results is really important.
1
SPEAKER_01
On the why is it taking so long side of things, when you take the data, and we re gonna have a very nice concrete example of this towards the end of this talk, it can be very, the goal, the business goal that you have might be something like, I want to make more sales, or I want to delight my customers and see that in feedback.
1
SPEAKER_01
Whatever that metric is, we have to translate that, speaking to the culture of measuring our results, we have to translate that to a concrete metric.
1
SPEAKER_01
And so this is going to be some sort of mathematical function of saying, how bad did my prediction miss from what I said the right answer was?
1
SPEAKER_01
And hopefully it s very close and you get some, some fireworks.
1
SPEAKER_01
Now, figuring out that answer.
1
SPEAKER_01
is actually very complicated.
1
SPEAKER_01
It s complicated in finance.
1
SPEAKER_01
It s complicated in producing recommendations to or returning a search for a query.
1
SPEAKER_01
There, that gap between what I want to mathematically measure, which is how the machine is going to learn based on that error and what my actual product result is really matters.
1
SPEAKER_01
Another area where you really have to think about what matters is how the data is actually structured.
1
SPEAKER_01
If I give a bunch of examples where this is the kind of input and this is output I would expect, or this is the kind of output I would not expect, the model
1
SPEAKER_01
is going to learn that specific pattern.
1
SPEAKER_01
So if you give it bad examples or you give it dirty examples, either examples with the wrong answers or examples where a negative case isn t as informative as an almost positive case, and we re going to get again to a concrete example of that soon, having that clean data is really going to be the lifeblood of how the model learns and ultimately how the product is successful.
1
SPEAKER_00
Or it s continuing our discussion on the infrastructure side.
1
SPEAKER_00
If you are a productivity business user, you can only affect the decisions that get made at the interest and the infrastructure level so much.
1
SPEAKER_00
What should one know?
1
SPEAKER_00
What constitutes a modern AI stack that enterprises can scale on the basis of?
1
SPEAKER_01
Ultimately, that s going to be a little bit dependent on the use case and the context, right?
1
SPEAKER_01
Every company is going to have slightly different needs and going to have slightly different applications.
1
SPEAKER_01
When it comes to doing the more advanced training and deep learning models that I ve been describing, you re going to need
1
SPEAKER_01
GPUs for one thing which means you may need a single GPU and you re a very small startup and you re trying to just work on a budget.
1
SPEAKER_01
You may need a cluster of GPUs in which case you need infrastructure management.
1
SPEAKER_01
In the last several years there s been obviously a very big shortage of GPUs and so
1
SPEAKER_01
It has historically been the existence of cloud platforms, AWS, GCP, Azure, has been very helpful in sporadic needs.
1
SPEAKER_01
Oh, I need to rent a GPU for a few hours to do my training and then I don t need it anymore versus
1
SPEAKER_01
Purchasing or long-term rental, right?
1
SPEAKER_01
Where you reserve an instance for days or for months, and that s going to end up costing thousands of dollars in the off time.
1
SPEAKER_01
So being able to size up what kind of training you need and then picking what level of, are you going to need it at all?
1
SPEAKER_01
Are you going to need it as a resurgence?
1
SPEAKER_01
Maybe you buy your own infrastructure, which is also a cheaper answer if you already have an infrastructure, an on-prem infrastructure system.
1
SPEAKER_01
That s really going to depend on what your AI training expectations are anticipated.
1
SPEAKER_00
So I m going to throw a curveball.
1
SPEAKER_00
If we think about the migration that enterprises made from on-premise to the cloud, the underlying value prop was, look, the variability is fairly cheap.
1
SPEAKER_00
You can ramp up fairly fast.
1
SPEAKER_00
In fact, I have this term that I use that it s a variable fixed cost on your P&L, meaning that you can scale it up and down.
1
SPEAKER_00
The line will always be there for you as a business, and that s the cloud cost.
1
SPEAKER_00
As we move to the world of AI nativeness and leveraging it, is your belief that the existing incumbents will actually, whatever you re seeing with the large foundation models providers, will become the Swiss army knife?
1
SPEAKER_00
So I m speaking, for example, to the AWS and Amazon approach, where they re saying, listen, you want to use open AI, you want to use anthropic
1
SPEAKER_00
We have our offering.
1
SPEAKER_00
We have bedrock.
1
SPEAKER_00
You don t have to switch.
1
SPEAKER_00
It s going to be one more feeder.
1
SPEAKER_00
So does this line remain or do we have a new fixed variable line?
1
SPEAKER_01
It s, it s a very good question.
1
SPEAKER_01
And certainly when you talk to AWS, you talk to GCP, they all have partnerships.
1
SPEAKER_01
And obviously Azure very much or so with, with open AI, they have these partnerships, but.
1
SPEAKER_01
And it s worth being aware of this using a first party solution, let s say open AI or with Anthropic is going to have different performance, different latency than if you do it via an intermediary, like a cloud.
1
SPEAKER_01
So you maybe get some simplicity by not having to onboard different vendors and different, different support, but.
1
SPEAKER_01
there is going to be an impact in terms of performance.
1
SPEAKER_01
And that s something that might be a temporary problem.
1
SPEAKER_01
It s certainly a very real consideration.
1
SPEAKER_00
Is it a high pain point in this moment in time, the delta in performance, or do we believe it persists?
1
SPEAKER_00
And your view is?
1
SPEAKER_01
Right now, it s worth not ignoring for the use cases that I ve had.
1
SPEAKER_01
And something else maybe hopefully transient for 24 is the option for ad hoc GPUs.
1
SPEAKER_01
When you go higher than like an L4, if you need an A100 or an H100, something that can host some of these larger, open source, large language models.
1
SPEAKER_01
We talked about commodity models.
1
SPEAKER_01
What if you want to do something like a 70 billion LAM or minstrel or mixed or any of these.
1
SPEAKER_01
Those are going to be very hard to get on an ad hoc basis, and you re going to end up having to pay the price for a resurgence if you want to be able to serve something like that.
1
SPEAKER_00
And those are, you ve now touched on the trade-offs between using open source platforms versus existing walled garden incumbents.
1
SPEAKER_00
Culture, how do you effectively implement AI from a cultural process people point of view?
1
SPEAKER_01
One of the both beautiful but also double-edged things that is happening with
1
SPEAKER_01
the ease of libraries.
1
SPEAKER_01
In the olden days, 10 years ago, if you wanted to put together a model, a neural net model, you would probably have to layer it, put it up layer by layer.
1
SPEAKER_01
You d have to design it yourself.
1
SPEAKER_01
For the most part, now you get a pre-trained, that is to say a model that somebody else has trained, and then you re just going to tweak its decisions a little bit to adapt it to your particular data.
1
SPEAKER_01
Or the days of coding up your own specific classification model, or there s a whole host of supervised and unsupervised, et cetera, models that you might want to do.
1
SPEAKER_01
The days of having to have ML experts do that by hand, that usually doesn t happen at
2
SPEAKER_00
you re doing tweaks, you re doing minor corrections.
2
SPEAKER_00
And what that means also is that you can have non-experts who are not used to a lot of the things that I ve already touched on, like, how do you make sure that your model s not overshitting?
2
SPEAKER_00
How do you make sure that your data is clean?
2
SPEAKER_00
How do you trust and have the right metrics so that you don t have an embarrassing situation where your LLM offered
2
SPEAKER_00
free acts of service.
2
SPEAKER_00
And now you re on the hook for actually backing up whatever your LLM did, even though it wasn t trained, or aligned, it s called for that particular mistake.
2
SPEAKER_00
So enabling your engineers in order to leverage all of these great use cases, including the models that seem to
2
SPEAKER_00
take care of themselves sometimes deceptively and making sure that you still have very rigorous metric supervision and evaluation of how your models are doing and when the worst cases are going to happen.
2
SPEAKER_00
making sure that you stratify another technical term.
2
SPEAKER_00
That means you consider all the different situations carved up into little boxes so that you make sure you re not just performing well on one, you re performing across the board of all of the different use cases.
2
SPEAKER_00
Having that strong culture to back up the performance of your model before you go into release is probably one of the biggest issues in terms of having a strong ML culture when you re, as it gets easier and easier for non-experts to start leveraging these tools.
2
SPEAKER_01
And if in a five minute conversation with a business owner, you have the opportunity to completely overwhelm them by throwing all sorts of technical, ML terms.
2
SPEAKER_01
If that is not what they have done day in and day out, but they re very good at running the P&L and all of a sudden we are in a new paradigm, how can they stress test what you are doing?
2
SPEAKER_01
They don t speak your language.
2
SPEAKER_01
You throw the three letter acronyms, like where do they find themselves?
2
SPEAKER_00
That s a wonderful question.
2
SPEAKER_00
I love it.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
So it s very easy to throw technical acronyms and to maybe go deeper than is really necessary when you re coming from a technical perspective.
2
SPEAKER_00
Understanding exactly what a transformer model does or what a multi-head attention network does and why you need rotary versus non-rotary positional embeddings or different softmax loss functions and temperature.
2
SPEAKER_00
That honestly doesn t make a big difference in terms of making the business decision.
2
SPEAKER_00
Oh, is this model going to perform the way I want?
2
SPEAKER_00
Is it going to perform out of sample?
2
SPEAKER_00
That is say when we re live, are we cleaning our data?
2
SPEAKER_00
Are we using, or do we have metrics that match what the model was trained to do?
2
SPEAKER_00
our loss function, our leading metric, how did my prediction match the truth versus what the product is supposed to do.
2
SPEAKER_00
And asking questions about why those match is going to be the most valuable thing for you to dig in on, because that s what s ultimately going to determine the performance of how your product that s AI driven is going to succeed.
2
SPEAKER_01
I couldn t agree more.
2
SPEAKER_01
So let me turn it over to you to delve into some of these examples.
2
SPEAKER_00
Yeah, Shopify is one of the largest e-commerce platforms in North America.
2
SPEAKER_00
In particular, that means making sure that we can answer a fundamental question of if a buyer has a query and is searching for a query, which products would match that query, right?
2
SPEAKER_00
Which ones are relevant matches and which ones are irrelevant matches?
2
SPEAKER_00
Now, that becomes relevant when there s a particular merchant, there s a single shop who is using Shopify and wants to, you want to, and if somebody goes to that shop and wants to search for particular products within that shop.
2
SPEAKER_00
We also have a shop app where you re searching much other e-commerce experience across multiple merchants and multiple products that are hosted across several different providers.
2
SPEAKER_00
And so search relevance in those search contexts, those kinds of search contexts that as an end user, you re very familiar with are first and foremost, very important.
2
SPEAKER_00
Now, there s also just making sure that you have high quality, right?
2
SPEAKER_00
If I search for boots and I end up with boot socks and boots and ski boots and a lot of things that might not be relevant, I need to be able to evaluate that.
2
SPEAKER_00
And I need to be able to evaluate that, say, on the first and different results to make sure that when we do these key searches, we re giving the end user buyers the right kinds of results.
2
SPEAKER_00
And to do that, we need to have an evaluation, right?
2
SPEAKER_00
How does this product match with this query?
2
SPEAKER_00
And again, this goes back to what I ve been banging the drum on all along is you need to care about your metrics.
2
SPEAKER_00
So this is as much a metric as a product itself.
2
SPEAKER_01
Mike, if I may interject because I think we can t stress the metric point enough.
2
SPEAKER_01
It s also not all metrics are created equal across different businesses.
2
SPEAKER_01
So what s a good enough metric?
2
SPEAKER_01
I m going to make up a basic example.
2
SPEAKER_01
If I search for an item on Shopify, the boot example, and I get the right item 50% of the time, that might be great or not.
2
SPEAKER_01
You will tell us what s relevant in one context.
2
SPEAKER_01
but completely insufficient in another context.
2
SPEAKER_01
So tell us a little bit about not all results are created equal back to this metrics point.
2
SPEAKER_01
Context matters.
2
SPEAKER_00
Sure.
2
SPEAKER_00
Yeah.
2
SPEAKER_00
And maybe to your point, in product search, there s two dimensions.
2
SPEAKER_00
There are two principal directions that you might want to have relevance in, right?
2
SPEAKER_00
One might be I m looking for a product.
2
SPEAKER_00
And so I want to see that product.
2
SPEAKER_00
I want, I m looking for boots.
2
SPEAKER_00
I want boots.
2
SPEAKER_00
Another might be.
2
SPEAKER_00
I m shopping for boots.
2
SPEAKER_00
And so I m going to buy boots and I m going to buy socks that match the boots.
2
SPEAKER_00
And I m going to buy maybe some cleaner that, that make sure that my boots stay in a nice condition and the weather in the winter time.
2
SPEAKER_00
So cross-selling and relevance, right?
2
SPEAKER_00
And these can actually be two different metrics.
2
SPEAKER_00
And maybe what you would want to do is have one metric that shows for direct relevance and another that shows complementary relevance.
2
SPEAKER_00
Right.
2
SPEAKER_00
And then it is somewhat a product decision of how much you want to balance these, but it s also something that you can balance with GM with your, your bottom line.
2
SPEAKER_00
Right.
2
SPEAKER_00
with how much new, you know, if we balance for say, 50% are going to be direct on the nose relevant.
2
SPEAKER_00
50% are going to be more complimentary.
2
SPEAKER_00
Maybe if we tweak that to 60, 40 or 70, 30, you might actually see a change in your bottom line.
2
SPEAKER_00
And that s the sort of, that s the way your product, your end goal is going to be determined by certain parameters that you choose in your actual model deployment.
2
SPEAKER_01
And so back to this interdependency between your teams and your organization and the P&L owner or the merchandiser, whoever is responsible for the business unit and that bottom line, how does that interaction back and forth work?
2
SPEAKER_01
Whatever tweaking you are doing on for your 50% or 2% improvement has ramifications in terms of their bottom line, but also the resources that you have leveraged.
2
SPEAKER_01
So if
2
SPEAKER_01
We wanted to express the metric in terms of results for the end customer in this example.
2
SPEAKER_01
What about from an ROI perspective?
2
SPEAKER_01
How much do you have to put in to get what out?
2
SPEAKER_00
Yeah.
2
SPEAKER_00
So there s a couple aspects to that.
2
SPEAKER_00
One is as a data person, obviously I love seeing the numbers and letting the numbers tell me.
2
SPEAKER_00
And so having experimentation, being able to experiment with, to take the simple example of how do you balance these two kinds of experiences is one answer.
2
SPEAKER_00
And then you need to invest in.
2
SPEAKER_00
experimentation frameworks, right?
2
SPEAKER_00
And hopefully, you know, again, these things get easier and easier as there s existing open source frameworks and tools and vendors that you can leverage for that.
2
SPEAKER_00
That s one answer.
2
SPEAKER_00
The other answer is really keeping close communication with your stakeholders.
2
SPEAKER_00
And this really speaks to the culture aspect of it, right?
2
SPEAKER_00
Just having ML engineers and ML scientists working in a vacuum and coming up and saying, this is your model.
2
SPEAKER_00
I m going to go work on the next fun technical thing is a mistake.
2
SPEAKER_00
And it s been a mistake.
2
SPEAKER_00
And we ve seen different versions of this back when my job title was called data scientist.
2
SPEAKER_00
And that was all that was the only kind of title there was.
2
SPEAKER_00
What you need is to have a pretty mix.
2
SPEAKER_00
You need to have engineering for serving.
2
SPEAKER_00
You need to have your ML scientists and engineers, and you need to have product.
2
SPEAKER_00
You need to have product in the trenches with your ML builders.
2
SPEAKER_00
So that way they can understand what are the pros and cons of not just making a tweak in a parameter and what a metric is, but also that key value that I keep driving home, which is does your end goal product metric.
2
SPEAKER_00
and what you want to get there match your metrics that your model is trying to learn from.
2
SPEAKER_00
And that s going to make a huge difference in terms of what you actually see when you get your bottom line results.
2
SPEAKER_01
Got it.
2
SPEAKER_01
Vectorization and multimodal LLMs.
2
SPEAKER_01
LLMs most people have heard in one form or another.
2
SPEAKER_01
Multimodal is initially we started with text-to-text, now it s text-to-image, text-to-video, 2D to 3D, etc.
2
SPEAKER_01
Vectorization in plain English and then how
3
SPEAKER_00
that come, there s relevance in the context of the exchange we ve just had?
3
SPEAKER_01
Yeah, we had in the good old days, it was keyword search, right?
3
SPEAKER_01
And people have learned, oh, I just put in one in the good old days, maybe 15 years ago.
3
SPEAKER_01
When you re searching, you search for one word.
3
SPEAKER_01
You don t really phrase it in a sentence.
3
SPEAKER_01
You don t talk to Google that way.
3
SPEAKER_01
And we used not to do that because it s just looking for the main words.
3
SPEAKER_01
It s ignoring that we call the unimportant words and trying to match those in the products or whatever it is that you want to do.
3
SPEAKER_01
You want to search for it, the documents, if you re doing, if you re doing a search on a web pages.
3
SPEAKER_01
For is it s a somewhat complex topic, topic, which really can be boiled down to, to a very simple image, which is if
3
SPEAKER_01
in a multi-dimensional space, and you can imagine three, every word in its context and every image in its context can be represented as a point in that space.
3
SPEAKER_01
And if you re doing your vectorization correctly, then similar words from similar contexts or similar images are going to end up located in that
3
SPEAKER_01
similar part of the space, right?
3
SPEAKER_01
So all your cat pictures are going to end up here.
3
SPEAKER_01
All your dog pictures are going to end up close to the cat pictures, but not the same place.
3
SPEAKER_01
All of your fireworks predictions, you ve seen a lot of those are going to end up way far away from those.
3
SPEAKER_01
And the words corresponding to those are going to be in civil.
3
SPEAKER_01
If you do it right, your image vectorization, where your pictures end up as a point in that space are going to end up in the same part of that space as your words.
3
SPEAKER_01
So if I describe
3
SPEAKER_01
cat, it s going to end up where the cat pictures are.
3
SPEAKER_01
If I describe a dog, it s going to end up where the dog pictures are.
3
SPEAKER_01
And when I say in context, what if I say bank?
3
SPEAKER_01
Once I say bank, I could say the robbers were leaving the bank and they were going at high speed and crashed into the river bank.
3
SPEAKER_01
And I ve said bank twice here, and those are very different words, right?
3
SPEAKER_01
One is going to be in the second one in that context is a place by a river.
3
SPEAKER_01
The first one is a place that has money.
3
SPEAKER_01
And so modern techniques for vectorization have done a very good job of looking at the rest of the sentence, looking at that context and figuring out that the first instance of the word goes where with pictures of industry, industrial places that manage money.
3
SPEAKER_01
And the other one has pictures of rivers and that sort of thing in it.
3
SPEAKER_00
What Mike just described is about 80% of the backbone of Gen AI and why we have the ability to interact with some of the applications in plain English and get human-like interactions, because leveraging these vector technologies and delivering what s called embeddings, which is really leveraging the vectorization through deep learning, just so that folks actually internalize what this all means.
3
SPEAKER_01
Yeah.
3
SPEAKER_01
And when you hear embeddings, it s, it s pretty much a synonym to how I m using the word vectorization.
3
SPEAKER_01
It s embedding the word or the image into a vector space.
3
SPEAKER_00
So Mike, you re using this on how you re constructing the, whether it s the merchandising, the images, the text, the descriptors.
3
SPEAKER_00
Pricing, are you leveraging AI for dynamic pricing of any kind?
3
SPEAKER_01
So dynamic pricing is a complicated area.
3
SPEAKER_01
And very generally, I know from past lives that you never want to do dynamic pricing.
3
SPEAKER_01
You want to do dynamic discounts.
3
SPEAKER_01
As a product framing bit of advice.
3
SPEAKER_01
Now, certainly having tools, helping merchants to understand things like demand and supply curve and how that those trade-offs are.
3
SPEAKER_01
That s certainly something that Shopify is interested in providing for merchants as the first party customers of Shopify.
3
SPEAKER_01
We want to give them the tools so they can be effective entrepreneurs.
3
SPEAKER_01
Real quick, just to frame the evolution of what s going on here, right?
3
SPEAKER_01
We started with keyword matching.
3
SPEAKER_01
Then we might compare vectors, right?
3
SPEAKER_01
And typically I had on the previous slide, cross encoding and two towers.
3
SPEAKER_01
These are just very technical neural network architectures where we actually look at the vectors at the same time versus looking at the vectors.
3
SPEAKER_01
separately.
3
SPEAKER_01
And if you want, if I do a query, I don t want to have to compare that query to every single product across all of Shopify, let alone even maybe in a single large merchant.
3
SPEAKER_01
And because it s not scalable.
3
SPEAKER_01
If you have hundreds of millions to billions of products to search for, you can t do that.
3
SPEAKER_00
Now it s scalable.
3
SPEAKER_00
We mean it s very expensive to from a compute perspective.
3
SPEAKER_00
Yeah.
3
SPEAKER_01
Expensive.
3
SPEAKER_01
Yeah.
3
SPEAKER_01
Expensive.
3
SPEAKER_01
And, and obviously prohibitively, it would take too much time in order to return it.
3
SPEAKER_01
So that s one side of, and that is in terms of serving, in terms of giving results, when somebody produces a query, you re going to want to sever those.
3
SPEAKER_01
Now, if you want to just look at quality and you can do just smaller quality checks, then you can look at them simultaneously.
3
SPEAKER_01
You can get a little bit more information.
3
SPEAKER_01
The same story is now true with LLMs.
3
SPEAKER_01
LLMs, if you domain adapt them right, that is to say you train them on your data and have them learn your specific behaviors that you want to see correctly, can be very good at also taking you in a slower context.
3
SPEAKER_01
I look through this query, I have this product, are they relevant together?
3
SPEAKER_01
Or I have this product, what are the relevant features?
3
SPEAKER_01
What are the attributes?
3
SPEAKER_01
What are the sorts of structured features?
3
SPEAKER_01
If you have a t-shirt, it might have a v-neck, it might have a turtleneck, it might have different colors.
3
SPEAKER_01
What sorts of things can I extract about this image and this product description that I can use in search queries down the line?
3
SPEAKER_01
And so LLMs are remarkably, almost too good to be true, effective at this sort of work.
3
SPEAKER_01
Good.
3
SPEAKER_01
What s the cons?
3
SPEAKER_01
They re very slow and they re very expensive.
3
SPEAKER_01
And if you pay a commodity LLM, which right now does usually beat out open source LLMs, you re going to end up paying per token and you can t scalably, if you have the right size of a business, do this for every query or every kind of use case.
3
SPEAKER_01
So what do you do?
3
SPEAKER_01
You might want to take an open source model and do what we call distillation.
3
SPEAKER_01
Distillation is a technical term.
3
SPEAKER_01
It means I have a small model.
3
SPEAKER_01
I have the answers that I ve gotten my big model.
3
SPEAKER_01
I know what goes into the big model and what comes out of the big model.
3
SPEAKER_01
And so I trained my little model to learn to mimic the big model answers.
3
SPEAKER_00
And how is that different than fine tuning?
3
SPEAKER_01
So fine-tuning is, you might have any size model, the parameters are already trained on a lot of data by someone else for their data.
3
SPEAKER_01
And that data might be close to yours, but it s not exactly the same data as yours.
3
SPEAKER_01
So if I m going to fine-tune my model, I m going to take my specific
3
SPEAKER_01
data, and I m going to adapt to my specific domain on my data, the base model.
3
SPEAKER_01
So for an LLM context, LLMs are really remarkably great at grammar and understanding the subtle, in these days, understanding the subtleties of language and often doing multilingual things.
3
SPEAKER_01
And there s a lot of baseline things that I don t want to put in the work on that.
3
SPEAKER_01
Right.
3
SPEAKER_01
That work has been done and it s been done very effectively.
3
SPEAKER_01
And there s no reason to have no reason to have everyone be another open AI or another anthropic when it comes to baseline.
3
SPEAKER_01
But when it comes to my specific data, whatever it is, I m going to want it to be extra, extra good at those nuances.
3
SPEAKER_01
And there are all sorts of ways of fine-tuning.
3
SPEAKER_01
There s all sorts of ways of, there s a whole lot of technical literature on different ways that you can adapt your model to your specific data.
3
SPEAKER_01
That s fine-tuning is the main one.
3
SPEAKER_00
And so back to your example, you were going to do the LLM, the large language model, and then almost like a small language model that s separate from fine-tuning.
3
SPEAKER_00
You re fine-tuning on top of the LLM.
3
SPEAKER_00
This is above and beyond sits on top of it.
3
SPEAKER_00
I just want to make sure people get the image of it.
3
SPEAKER_00
Okay.
3
SPEAKER_01
Yes.
3
SPEAKER_01
You fine tune your smaller open source.
3
SPEAKER_01
And when I say smaller, it might be seven to 70 billion, not so small, just small compared to the bigger one, right?
3
SPEAKER_01
You might fine tune it in order to mimic the performance of the large commodity model.
3
SPEAKER_00
Got it.
3
SPEAKER_00
I know we have some more context, but what do you wish I had asked you about your work that we didn t get to?
3
SPEAKER_01
Maybe it d be worthwhile just spending 60 seconds to talk about the difference about the data issues.
3
SPEAKER_01
Cause we, we, we Easter egg that.
3
SPEAKER_01
So in particular, let s think about search results.