Video Transcript

Good afternoon, and welcome to our session about “A Sneak Peek into How Realtime Marketing Works.” In this session, we are going to dive into the magical world of realtime marketing. My name is Tom, and I am a R&D team leader of tracking realtime data and interpretation.

Together with me today is Noy who’s also an R&D team leader of mobile tracking and execution. We’ll begin our session with understanding what is realtime marketing, and why should we, at least on some level, understand how it works under the hood. We’ll break down realtime marketing into three components: collection, interpretation, and execution.

We’ll end our session with some real-world examples and takeaways for everyone. So, here is Noy. You’ve met him already. Noy walks inside a mall and sees a poster. This poster is a pre-scheduled campaign. So, Noy enters the brand shop, the brand store and heading towards the jacket section.

We also see that Noy is wearing a high-fashion brand, right? So, is there a way to give Noy an incentive to purchase and to maximize his potential to make money? Well, the answer is yes.

How? A salesperson addressing Noy to the high-fashion brand and showing him the new collection. So, the poster was the pre-scheduled campaign, and the salesperson addressing Noy to the high-section brand and communicating with him is the realtime campaign.

Realtime campaign can give us the ability to market to customers. We would argue that the vast majority of the audience here don’t know how realtime marketing works. Our claim is that once you get the mechanics behind the scenes, you’ll be able to leverage this channel to its full extent.

In this session, we aim to answer questions such as: how data is tracked, how are customers identified, and how to trigger offers, and all of that in realtime. We’ll break down our session into three components: data collection, data interpretation, and campaign execution, campaign execution as I just I said.

We’ll begin with data collection. In this step, we’re going to capture all the actions made by the user. You might be wondering now what is the entity that we are collecting. So, the entity that we are collecting is an event. An event is an internal action on an app or a website. Reported events from an app are associated with a specific user.

An event can be a page visit, a button click, registration form submission, item added to cart, item removed from cart, deposit made, won a game, lose a game, or any other desired actions that you would like to report.

We would probably want to report some additional information regard at event, for example, if we’re talking about item added to cart event, we want to report its SKU, price, color, size, or any additional information that will help us understand or describe this product specifically. But when we are talking about data collection, we would probably want to support thousands of customers and visitors on the same time.

This is measured by traffic… Sorry. Measured by throughput. A throughput is the amount of events we are collecting on a specific time frame. It will change from lower peaks on the one hand, and high peaks that caused by maybe active campaigns on the other hand.

In order to support events that are made from thousands of customers and visitors on the same time, we must support unlimited throughput on a specific timeframe, on every timeframe, sorry. Well, it’s not completely true because unlimited is not really unlimited because unlimited is very resourceful.

Hence, we need to support an architecture that will both support average traffic rate, but also scales up when there are unexpected peaks of events. The average traffic requirement is met by maintaining a certain number of resources all the time.

Then when the system recognizes a peak, it will automatically increase its resources in order to support that peak. So, now that we have collected our data, let’s see how we interpret it. The difference between analysis and interpretation is that not all events were born equal.

Simply counting the number of events received on a specific time is not enough. Now, what will be the next thing I would like to understand about the data I collected? Have I seen this user before?

Do we know them? So, all events that are reported from our app are associated with a specific customer or any user. A user can be a first-time visitor, someone we don’t know or don’t have any information on.

It can be someone who visited, and now they are returning. Someone who logged in or even a paying customer. Once we identify the user, the next question that we would like to ask is, “What is the timeframe that we capture all the actions?” Here I’d like to introduce a new term, a session.

A session is the educated timeframe in which there is a connection between the actions that made by a specific user to the time they happened. A session is a collection of events from the first one to the last one that happened that have mutual purpose. A new session will start when a minimum time has passed from the last event from the previous session.

So, after we have collected the data and we interpret the session, what insight can we derive? A certain session can be for browsing products or another one for making a purchase. Looking on a single event doesn’t give us any context, but when we are looking on a sequence of events, we get the context.

We can learn behavioral patterns such as browsing frequency, product preference, activity length, engagement, usage trends such as platform device or operating system. So now that we interpret the data, can we learn more about the user from the data that we have in our databases?

Can we connect fast data with slow data, realtime with historical? The simple answer is yes. And let me explain. Data constantly flows into our system. We can use historical data such as the number of times a user visited our service and connected with actions, and then we detect actions made on realtime, we can leverage this action on the spot, and use it for further usage.

Here is Noy, and he will continue and will explain all about it. Thank you.

– Thanks, Tom. We have now collected the data, we interpret it, and now we need to decide whether or not to take an action.

This might not be as easy as one might think. The first question that we ask is, “Is the user eligible for a campaign?” Eligibility in realtime execution system depends on the completion of a sequence of events as the same events that arrived from our data collection system into our execution system arrive on a specific order.

Once a sequence is completed, such as visiting three pages of our website, the user is marked as eligible for a campaign. But let me ask you this. Can you imagine the size and complexity of calculating all these sequences in realtime? If we look at the average event size of half a kilobyte, and when multiplied by the average number of events per session, sessions per day, and daily active users, we get to approximately half a gigabyte of data to be processed each day.

This number might not sound so scary for data engineers, but for a realtime execution system, this could be a challenge. One way to efficiently handle this amount of data is states. What is a state? I’m glad you asked. State is like the summary of all the events that happened until a specific point in time.

If you look at this graph, user starts with the first started state. When the user admits event number one, she activates transition one and moves into state number one. She has a choice, and she sends event number two which activates transition number two and moves her into state two. If she have activated event number four, she would then move to state four. Yeah, okay.

I see that you’ve lost me there. Let’s use an example. Player one joins the game. She’s now logging in, and in the started state, and when she views the main screen of our game, she’s state number one. And she has a choice. She chooses to join the chat room thus moving her to state number two. When she leaves the chat room and goes back to the main screen, she moves back to state one and then starts playing a game and moves to state four.

As you can see, this is a memoryless system. We don’t care about the past events but only about the present. We are size efficient. State is efficient, state is small, and state is awesome. It’s not a coincidence that I keep saying, “Keep data small.” Small data is part of our key answer to the next question.

How can a campaign be executed in a realtime manner? Latency, latency, latency. Latency is the delay that it takes to perform a certain action. We reduce latency through our system by performing various actions. If I’d have to sum them up into three main points, they would be this.

One, minimize the size of the data that crosses your system. Two, provide sufficient computing power and keep monitors to make sure that your system doesn’t choke as more users are becoming more engaged with your app. And number three, use in-memory storage system. For the techies, the RAM is your friend.

For the non-techies, every computer has two memory chips, one for the short term, and one for the long. And much like us humans, it is easier for the computer to recall memories from the short term chip. I know what you might be thinking now, I’m going to put all of my effort into reducing latency through my system, and then I can get rid of all progress bars for good.

I’m going to stop you right there. Up to a certain point, reducing latency can be fun and efficient, but then it becomes an expensive, tail-chasing nightmare. How many supercomputers are you willing to maintain to present five dollar discount pop-ups? I didn’t think that much. If there is one takeaway I want you to have from this slide, it’s this.

There is no real, realtime. The question is, how near realtime we need to be? We know that the user is eligible for a campaign, and we know how to execute it in a realtime manner. Now, the question is, what exactly is the user eligible for? And that really depends on you guys, the realtime marketers.

What have you set up for your users? From our standpoint, we only need to know the desired state and the campaign to execute for that state. When you create a realtime campaign, we keep mapping in memory, of course, latency, latency, latency of all the desired states and the campaigns to execute for these states.

The data that this campaign information holds is something like, “What is the template to deliver to the user?What is the execution channels endpoint?Who do we talk to, to get this campaign delivered?And what is the data that the execution channel needs in order to deliver the campaign to the desired destination?” It’s really crucial that we read the small letters, and we make sure that the execution channel supports realtime deliverability because if it doesn’t, all of our efforts would be in vain.

So, we know that the user is eligible for a campaign. We know how to execute it in a realtime manner, and we know what the user is eligible for. It is now time for our last bonus question. Can a campaign be activated based on lack of activity? For us humans, it is easy to understand the concept of inactivity.

If we play an online, multiplayer game and we see one player just standing there doing nothing, we know that the fact that the user is logged in means nothing. The user is simply not there. For a computer, it’s nearly impossible to understand inactivity without some assistance. So, let’s help him. We define a threshold of one hour, and we say that if one hour has passed since the last event was emitted by the user, the system is going to send itself a user left game event, thus moving the user to a left game state, and triggering an email telling the user to come back immediately and help her teammate beat the zombie apocalypse.

We have reached the conclusion of our main segment. Now let’s recap everything that we’ve learned. Every realtime execution system starts with an event. An event describes the action that a user performs on our mobile app or website. Our data collection system needs to support high throughput, so that when more users become more engaged with our app we’ll be able to support them, collect these events, and still not harm their user experience.

Our data interpretation system assigns sessions to events. And with the sessions, we can have realtime data interpretation, and later on, thoroughly investigate users’ behavior. Finally, user is eligible for a campaign when it reaches the final state. And thanks to low-latency system, the campaign will be delivered immediately.

Let’s connect theory with reality. We’re going to walk through four scenarios that will illustrate how this data flow is implemented in real life. And we’ll start with an easy one. You’re a fashion store, an online fashion store, and you want to offer first-time visitors the chance to register and win five dollar discount.

How are we going to do that? Well, we identified that the segment is first-time visitors, and our trigger, also the desired state, is landing on the landing page. Our offer is a five dollar coupon, and our channel is pop-up. Pop-up, because the users are right there, and we want to engage them, but also, these are first-time visitors, we have no prior information about who they are or how to contact them.

Okay. It seems like you got me. Let’s move on to the next example. Tom here, he saw the pop-up and was immediately impressed. He really wants to buy some clothes from our store. So, he logs in and then goes to the shoe section and lo and behold, Tom views another pop-up for a one plus one discount specifically on shoes.

How did we know that Tom really wanted to buy shoes now? We’ll start with a segment, first-time users, and our trigger is viewing a store section. Notice how I say a store section, a generic store. If you’ve been paying attention, you’ll remember that event can carry additional information.

This additional information also known as parameters can contain the name of the section, and then we use this parameter to offer Tom a customized pop-up specifically for him at that moment in time. This is a perfect example of leveraging fast data in order to offer an immersive pop-up.

In this example, we’re going to bring in slow data into the mix. Time has passed, and Tom has become a fully-blown VIP customer. We know when he likes to make his purchases, we know what he likes to buy, and we especially know that he loves spending money on our store. That’s why when he proceeds to the checkout screen, with less than four items in his cart, we ought to do something about it.

We’re going to present Tom with a pop-up, offering him a list of recommended items. So, our segment is VIP customers. Our trigger is reaching the checkout screen with less than four items in your cart. But this is not the clever part. The clever part is to offer recommended items. Here we use our prior information about Tom’s preferences and bringing slow data in order to provide something really unique, something that Tom could not refuse.

Also here the execution channel is a pop-up because Tom is right there. Unfortunately, Tom got a phone call, and he never completed the purchase and four hour later, we understand that he’s not coming back. Luckily for us, we have attended a realtime marketing workshop, and we know that we can tell our system that after four hours of inactivity, so I mean, four hours after sending the last item added to cart event, we can trigger ourself with the cart abandoned event, move Tom to the cart abandoned state, and then send him an email asking him to, “Please, Tom, come back.”

– Thank you, Noy. So to wrap it up, hopefully, now you get the basic understanding of what is realtime marketing, and how it works under the hood. Ideally, you would be able to create campaigns based on realtime actions. You’ll be able to catch users on the spot.

You’ll be able to take advantage actions and making the most out of it. We saw that we can actually target unidentified users whether they are first-time visitors or returning visitors, and we can leverage their behavior into actual conversion.

By tracking an enormous amount of data, of visitors and customers, we have insights. Those insights are improving our segmentation. We have scheduled campaigns of these common behaviors. So, we can take those common behaviors over time and include them in realtime action.

By including realtime actions with our historical data, we can make our campaign precise. Thank you very much for listening. Have a great day. And if you have some questions for us, we’ll be here to answer in the following four minutes. Thank you.