GTAC 2015: Mock the Internet

>>Yvette Nameth: Next up, we have Yabin, from
LinkedIn, to talk about mocking the Internet. [ Applause ]
>>Yabin Kang: Hello, everyone. [ Applause ]
I know, it’s about time for lunch. But let me use the ten minutes to give you something
more interesting and may help in your day-to-day work.
So a little bit about myself. My name is Yabin Kang. I’m from LinkedIn New York team. I have
been with LinkedIn for three years. So today I’m going to talk about the topic
Mock the Internet, which I will first introduce the new mocking system that we are currently
working on, which is called Flashback, and also give you the overview of the mocking
strategy that we have in LinkedIn. So Flashback is a network-layer mocking system
that is taking care of all of the outbound traffic that goes on the Internet, where you
really don’t want to see, like, the third-party downtime network issue or results constraints
that break your test. So that’s why we created Flashback.
Before we jump into the details, let’s talk a little bit about why we like mocking, not
only for unit tests, but also for service-level integration test.
So why mocking? Imagine we have the LinkedIn profile page. Hopefully, everybody in this
room has a LinkedIn profile. Otherwise, you miss a chance to connect to other 400 million
professionals. [ Laughter ]
This (indiscernible). No. The Marketing team asked me to do.
Okay. Back to the profile page. So it requires more than 100 downstream dependency
servers to render this page. If you test — and most likely your test, it just tests one tiny
specification on this page — you don’t really want to rely on all of these dependencies.
Right? And things are getting worse if one of the
dependencies is actually an external service that — on the Internet, which you don’t have
control. So the best way to solve this problem is mock
them, mock all the dependencies. Then you have a good test.
So take a look at this high-level picture of LinkedIn ingestion service. (indiscernible)
ingestion service is built by our New York team, and it is a service to grab content
from the Internet to enrich your profile to be more attractive.
So besides the — besides the typical three-tiered architecture on the left, you will see there
is a gateway service in the middle. It is — it is working as a proxy. And here is a
good opportunity for us to mock in this layer in order to test our ingestion service results
rely on the external providers. And this is where the Flashback fits in.
So Flashback still acts as a proxy, handles the HTTP, HTTP has traffic in terms of request
and response and store them. It is controlled by test framework, so you can start it, like
record or play. You can also change the mock data on the fly from your test case since
we have the API. Or, as you wish, you can manipulate the response in a way to test the
different variation of (indiscernible) case for your service. So, basically, you have
the full control of what’s going to be mocked on the network.
So let me show you some code. This is the implementation about a record
and replay inside Flashback. Let me — Recording, we actually take the response returned from
the external provider and are pairing it with the request that came from the internal service
and store them locally. When we’re playing, we load the local stored
response according to the request that came from the internal service and just return
them back. So there’s — there’s not going to be anything — any traffic that goes on
the Internet. Also, I want to show you again some of the
mock data, the actual mock data. You can see we have all the information about the request
and response. And like I mentioned, you can actually manipulate the response by modifying
this file. Like, we can change the standard code to something else, from 200 to 404, to
indicate a not found response. Or you can modify the content of the AGT body to something
else, anything you want. In this way, you can test your service in a lot more scenarios.
Then you can see how strong, how reliable your service is.
So, okay, Flashback is a good solution for us to test our ingestion service without relying
on the external provider. But when we step back and take a look at the whole picture
of how Flashback works, it’s actually — it seems a common solution, just doing the mocking,
the network, the traffic. Right? If we can make the Flashback more generic to be essentially
at the service or the system, it can help in a lot of other place in the same manner,
like internal service. So LinkedIn internal service are all RAS files, which means you
have a common library being used by the service and all the clients. So to — adding to Flashback
is just like a config change to a Flashback-enabled HTTP client. Then you can use that. It’s quite
easy, and nothing needs to be changed inside your service.
It’s also a good solution for data service. As long as your service is going through HTTP
protocol, then you can enable Flashback, and it will take care of the mocking part, mocking
the data from the data service. Also, it’s more interesting to talk about
the mobile client testing. So when we do mobile client testing, we actually need to launch
the emulator, launch the app, and deploy your service, and having the real traffic between
the client and your service. But instead of making all this live traffic,
if we can set up the proxy inside the emulator and point to the Flashback, and then you can
take advantage of over the Flashback and the proximity of network mocking.
So it’s quite easy to set up the proxy inside emulator. And it also works for real device.
And, again, there is nothing that needs to be changed inside of the app. You can just
use Flashback. Okay. I hope you guys have the idea of what
we are trying to build, the Flashback. It helps you in the area of doing the service-level
integration test that requires a network layer mocking. And we are trying to build it in
a way that could be open sourced. And it will be a library for anyone who wants to use or
anybody who even wants to make contribution back. Or you can, like, come and join us in
LinkedIn — [ Laughter ]
— and let’s work together and mock ‘n roll. Thank you.
[ Applause ]>>Yvette Nameth: Thank you.
What’s the difference between using dedicated Flashback service versus using mocking frameworks
at the class layer like Mockito, EasyMock, et cetera?
>>Yabin Kang: So the — Flashback is still, like, network-layer mocking. So for those
unit test mocking is kind of mocked at the class layer. So when we mock the network layer,
so you actually have the real traffic goes out to — goes out from your service, so that
that’s verified, like, a lot more than, like, just inside the class.
>>Yvette Nameth: How does Flashback compare to WireMock or other mock proxy tools?
>>Yabin Kang: That’s a good question. I knew someone would ask this question.
So WireMock is — Sorry. So Flashback was — the idea for Flashback was for our ingestion
service. We built that as a proxy. Then we realized we — actually, there is, like, WireMock
or Betamax. There is a lot of other mocking — network mocking stuff there. But some of
them is now doing, like, a proxy way. Some of them is all handled at the https traffic.
So we end up with we wanted to build something ourself.
>>Yvette Nameth: How do you deal with post requests? And how do you deal with URL GET
with variable parameters, like time stamp, random session ID, et cetera.
>>Yabin Kang: The post request is the same — is still just a request. It’s just different
method. So we have the post. Like in the mock data, we have the method tab for get or post.
To deal with the different parameter, like time stamp, random session. Okay.
So this problem is — we don’t have this problem for our ingestion service testing, because
we — most of our case is, like, static content — static URL and content. But we do — have
thought about this. We can make some parameter or variables inside your URL, or you help
to indicate, okay, this part of data is, like, dynamic. We don’t care. We just care something
like the — important is a get and what’s the URL, what’s the body, the content. Yeah.
>>Yvette Nameth: Is Flashback open source?>>Yabin Kang: Yes, it is going to be open
sourced.>>Yvette Nameth: That one’s a repeat.
How do you manage situations where third-party providers change their layout or schema?
>>Yabin Kang: Mm. Okay. That’s another good question.
So, basically — so for the case for our ingestion service, we — we don’t have the idea when
the third party will change their content; right? So we keep running our test for using
the mock data, which is making sure our service is still working for at least the old data.
Then we have some live traffic to making sure that’s going to be working for the — for
the new data. And also, since you have the mock data, you can theoretically check the
data versus the new data from the — the third party. Then you will know, like, is there
any change? Is there any schema change or some content change? If there’s a significant
change, you can rerecord the stuff and do the test again.
>>Yvette Nameth: How does it handle https requests?
>>Yabin Kang: Okay. So since you know we are using, like, the proxy way, (indiscernible)
require we have to have a man-in-the-middle proxy. So we kind of implement that. And that’s
the most challenge in Flashback, since most of our URLs is going through https traffic.
So we spend a lot of time on, like, handling the https and doing the man in the middle
thing. Yeah.>>Yvette Nameth: How do you keep recorded
data up-to-date?>>Yabin Kang: This is sort of like if the
content change around the third party. So like I said, for whatever the mock data
that you have, you should be able to test — to run your test. But if the data changed,
you need to have a mechanism to check if anything changed between your old data and the new
data from the third party. So — so, like I said, you need to periodically check to
see.>>Yvette Nameth: How do you ensure mocked
tests don’t get forgotten about when internal service APIs change?
>>Yabin Kang: Okay. That’s — that’s sort of similar to the last question. But I think
it’s more like asking about the internal service. Since we are testing, if we are running that
against internal service, so you have the idea of what version you’re using for your
downstream service. So, basically, you have a service — you have version for your own
service, and you have another dependency service. So when update your dependency service, you
know something might be changed; right? So in the Flashback, either you do rerecording
every time there is a change for the downstream version, or we want to have some, like, mechanism
to say my test knows my service, and we know we only care about some particular data, like
I only care about the first name, last name. I don’t care about, like, maybe someday there
is a new field that says title or something. So I can have a way to verify, like, if the
new service — if I construct the data schema for the new service, still have the last name,
first name, then it’s good, I can still use that data, I can still test my service. And
it should not break my service, should not break my test. And should — there is nothing
I should worry about for the downstream update.>>Yvette Nameth: And one last question. They
keep moving. Where’s the new one?
How is Flashback different than LinkedIn’s other product, Sephia?
>>Yabin Kang: So — so the Flashback started with, like I mentioned, for the ingestion
service, the proxy. All our other products is kind of doing either the mocking inside
the service or inside the class, which is — you need — you have to modify something
inside your service or inside your code, which this kind of give you the benefit of proxy,
as long as you can still have a proxy, then you’re good to go.
>>Yvette Nameth: Great. Thank you very much.>>Yabin Kang: Thank you, guys.
[ Applause ]

, , , , , , , , ,

Post navigation

Leave a Reply

Your email address will not be published. Required fields are marked *