“If we have data, let’s look at data. If all we have are opinions, let’s use mine,” former Netscape CEO Jim Barksdale reputedly once said. At social networking service LinkedIn, it is the job of senior engineering director Ya Xu to ensure that major decisions around the site’s design or services are not left solely to the gut instinct of one person, however well-intentioned.
For the past six years, Xu has led the company’s data science team, which works on how to use the data to improve products, optimize the sales channel, identify the right targeting for the company’s marketing efforts, and so much more. A big part of this process is relentless experimentation, which has become a core part of how LinkedIn utilizes data to make decisions.
We spoke with Xu to learn more about how testing is built into LinkedIn’s core infrastructure, how analysis is incorporated into each of the company’s business units, and what lessons she brought from her graduate word in statistics and from her previous, similar role, at Microsoft Bing.
So your team is part of the business unit. You have a set of data that is both internal and external and you try to turn it into actionable information?
Yes. Let me give you a concrete example. Let’s say, if you use LinkedIn, we give you a feed of data from various individuals in your network. Our team uses data and experimentation to think about not only providing the value to whoever is reading the feed updates, but also the ones who are creating it. For example, we think about what the value is of you liking your friend’s post.
And your team, are they data scientists? Do you have programmers? How would you describe the makeup?
It’s a mixture. My team is a combination of data-savvy individuals who are analysts, scientists, and engineers. We tend to be people who are good with data coding, math, and stats and also good at understanding the business or the domain areas such as our engineering infrastructure. For example, my team works on problems like how we should optimize the utilization of machines or CPUs.
And your deliverables get handed to the developers on LinkedIn or do you work embedded in different groups? How does that information get transferred over to be realized?
It depends. I have folks who are embedded into different domains, and they work in a cross-functional setting, together with our engineers, product managers, marketing teams and sales team, etc. And we have teams who are more in a horizontal role. For instance, in the Applied Research team, their job is to come up with a wide methodology and then they would build those methodologies into the tools, platforms, and infrastructures that we have here at LinkedIn so that everybody is able to then utilize them. So it’s a mixture of both.
For example, we have a team that is very focused on the platform — the tooling, the scalability, the ease of use and the democratization from the platform standpoint. But also at the same time, I have data scientists who are embedded in each of those domains. They are the experts at using experiments. On the one hand, you have the platform that is able to help scale the company and helps scale the rest of the data scientists who are in each of the domains. But also at the same time, the domain-embedded data scientist can also really push the boundary of experimentation and then bring that back to our platform.
Do they understand that particular part of the platform and so they know what question to ask?
Yes. And they are also indispensable when it comes to pushing a culture of experimentation, too, because they are in those domains, they work very well with all the cross-functional individuals in the team and know how to use the platform. Therefore, they can also be the advocate, the champion for running more experiments and evolving the culture, in addition to evolving the platform itself.
So, “democratize decision making.” What does that mean?
I don’t know whether you have heard of this quote from, I think Jim Barksdale, the former CEO of Netscape, “If we have data, let’s look at data. If all we have are opinions, let’s use mine.” Data is key to push the decision power to more people. LinkedIn is a company where we really care about data and we have a lot of it, so there is a very strong culture around the use of data to help inform decision making.
When I started at LinkedIn six years ago, we didn’t have an easy way of running experiments and looking at the results of experiments. But today, pretty much every single change that we’re making, from changing the color of a button to making a very sizable shift, is experimented.
For example, when we revamped linkedin.com entirely about two years ago, every change went through our experimentation process. By doing that, we were able to actually understand not only whether intuition was right but also whether this is actually a good thing for our users. It allows us to make the decision a lot easier and a lot more objective as well.
It is also important to note that the decision power doesn’t just happen at the hands of our senior leadership. Because of experimentation, it is democratized to our engineers and PMs [project managers] and everybody in the company. Let’s say for example someone says, “Hey, I have an idea” or, “Oh, you know, we should do this.” There’s always constructive criticism around, “Is that really a good thing to do?” And a lot of times these ideas would get stale sitting around because the team cannot come to an agreement. Because of our experimentation and data-driven, decision-making process, everybody in the company is empowered to try new things and then we can test it.
And if the results are not good, then we should not launch it to our users. This goes beyond the decisions of senior leadership; it’s about what the results are telling us through the use of our data. So I think because of our culture of experimentation and because we have the right platform to support it, the decision-making process has become a lot easier for anybody in the company to participate in. Not only does this democratize decision-making, but also helps us to be bolder as well — because we can always rely on experiments to learn which ideas will take off.
What’s your workflow from moving an idea into a testing environment?
We have a platform where anybody in the company can actually set up an experiment themselves. We have a UI that helps any employee set up and manage their experiment.
After the experiment is finished running, we also give you a report with 4,000 metrics regarding how your experiment is impacting the whole ecosystem. And again, not just in terms of “Hey, what is your changing of this button color from yellow to green is impacting the click-through rate?” But we actually have a holistic way of looking at what is impacting our top line, company-wide metrics.
When you arrived at LinkedIn, did you find that the system was to provide tools to help you do this testing? Or is it something that you had to work to implement?
We had to do a lot of work to make it happen. When I first joined LinkedIn, we had a way to do feature flagging that was primarily used for deployment purposes. However, there wasn’t enough of an understanding of the impact looking at the results and analysis. So one of the reasons I was actually hired was to build out our experimentation platform.
Today, we launch over a hundred experiments a day. It’s a pretty massive scale. LinkedIn is well recognized when it comes to experimentation in terms of scale, in terms of sophistication.
So, you were at Bing before LinkedIn. Were there any lessons that you brought from Bing over to LinkedIn?
Definitely. My work at Bing was focused on experimentation, so I learned two big lessons. One is how important a strong platform of experimentation can help evolve the culture. When I was at Bing, there was already a platform that was running, and when I was there, I saw that the platform evolved to be more trustworthy, to be able to solve more cases. I also saw how the culture changed, and how it was able to transform an organization. So that was a big lesson for me.
And the second for me actually relates to my background in statistics. To make experimentation work, it takes more than just statistics. Statistics is a very core component in controlled testing. But to transform a culture it takes more — it takes statistics, it takes engineering, it takes thinking about the user experience. If a platform is only able to be used by data scientists, you can never reach the democratization of decision making.
The platform needs to be able to be used by engineers who are not necessarily savvy in statistics. It needs to be used by the product managers and even by the executives who do not have too much time to go through the details. So understanding that a full package was needed to make it work is another lesson that really helped me when I moved to LinkedIn. These were all things I took into account to help create the culture, the team, and to transform how LinkedIn does experimentation.
