Because you’ll be networking and Black Friday will be crazy and all the servers will fail (not! let all the servers be up) I asked my colleague Irina who is the CTO (chief technology officer) at 2Performant to talk about how we prepare for Black Friday from a technical perspective and how the technical department works with with marketing and all the others to prevent any problems that have happened during Black Friday, when, because of very high traffic spikes, many stores were down. Irina, please Thank you, Dorin Are we on? Yes My name is Irina Dumitrașcu, I am very happy to talk to you today about what we do for Black Friday I have been honored to be in charge of the technical part of 2Performant for three years already, and I will talk about Black Friday mostly from programmers’ viewpoint. I don’t know if there is any programmer in this world who would say “Black Friday is coming! Whoo-hoo! So cool!” “Black Friday is coming! We need to keep the platform up, everything to run smoothly, and if anything goes wrong we’re screwed!” So the emotions surrounding Black Friday, for us, are… not so nice (uncertainty, fear, anxiety, concern, uneasiness) What is my purpose, as a technical team leader? What to do about these emotions? Well.. What I wish is for all of them to… RAT-A-TAT-A-TAT-A-TATAT for all of them to disappear! And, of course, their place must be taken by something (nothing disappears, everything transforms) What I wish is to have a big, big, fat feeling of… BOREDOM I want everything to go so well: Services: up. KPIs: reached, Statistics: up to date, Sales: registered so the main problem of the technical team, these days, to be “What will we eat this Black Friday?” Do we get Turkish food, like we did last year, or should we try something new? Will we have plain salted popcorn, or the buttery kind? This is the kind of things I want to have on our mind! And the amazing thing is that we are actually doing it Sounds like a fairytale, but yes, we had a gloriously boring time at the office in last years. I want to share with you our “magic formula” And it’s very simple: first of all, we start from planning and communication, to which we add an investment of time and money. I will share, for each of these directions: what do we do, and what I think it will be useful for you to do as well. We start from planning and communication. For us, the first thing is to assess, each year, “where do we stand?”, as we change the code a lot throughout the year. And for this we simulate Black Friday. What does this mean?
On the production servers, (that have been upgraded to the Black Friday configuration), while everything is running as usual (we register traffic and conversions,
the platform is used by our customers) We generate 3 times more than the expected Black Friday traffic. We do that long enough to be sure we get coherent results And this is the baseline we start from. The next step is to gather all departments: marketing,
product development, affiliate management,
user success, – finance
– business intelligence and ask all of them what are their objectives for this period What are the numbers they expect? We are interested in: expected traffic, conversions volume We want to know all the new initiatives they want to start And we ask them a question that might sound strange: What will be different from a regular day? Of course, Black Friday is a totally out of ordinary period, but framing it this way we will get all the information. They will tell us everything they want to do and we will create a big calendar with plans, ideas and needs. Of course (ups, sorry) We will do this 2 months ahead We will start only 2 months earlier because we have been through this process 4 or 5 times in the last years and now it’s quite predictable So we will start on September 17th If you haven’t been doing such planning before, I recommend that you starting tomorrow… or even today. Then we prioritize all the ideas and needs that we have, because we’ll never be able to implement all of them And for us the priorities go like this: Most important is registering traffic and conversions That has to run impeccably and that has to run fast. Here are our numbers: If on a regular day, the the maximum number of requests goes up to 150k on Black Friday it jumps to 450k and it stays up for a longer time. We went a bit over the board with testing: we went up to 1.5M requests per hour up to that we know the systems handle the load. Our target is for response time of the tracking system, is under 40 milliseconds Second in line by priority is the platform, especially the functions that matter most on Black Friday commission listing, messaging, some stats that are helpful on the short term, adding money to advertiser accounts. Here the numbers also go up during Black Friday: if on a regular day the maximum number of requests reaches 30K per hour during black Friday if goes up to 50K requests per hour And the platform is used more intensively: starting on Monday and throughout the whole week, we get a lot more traffic For the API, our target is to keep the response time under 150ms. Next is the administration system for the our colleagues,, that needs to work properly for them to be able to take decisions and to communicate with customers. The next set: we optimize what is used most during Black Friday We improve the order in some operations are performed For example, last year picture we implemented a new feature: “when a store sends a message to all their affiliates, the message is sent first to the affiliates that have results in that program, and only after that to the others.” We might add new functionality: “targeting score”, that Dorin showed earlier, was created after the Black Friday planning last year And there are some things that we set on priority “minus infinity”; meaning that under no circumstances will we make: architecture changes, major upgrades, launch new important features, one month before Black Friday. What do I recommend, regarding planning and communication? First of all, think of your objectives, express them as numbers, and go over them with the technical team. It is very important that these people know what to expect, what are the parameters under which the system must work. Together as well, make a plan. Prioritize. Decide what is vital, what is important, and what doesn’t matter for the sales to be performed successfully. And do this, I’m going to say it again: (initially I had this line 15 times throughout the presentation) do this a lot of time in advance. For us, 2 months is enough; start as early as possible. First of all, make sure systems are stable; Black Friday will not be successful if sites are down. Prioritize the flows that people will use during Black Friday: for stores, it’s obvious that adding products to the cart must work that all the offered payment methods must work, that users must be able to authenticate easily. Test all the tools that you are creating. Affiliates: test (and test again) that the links are correct to the right programs, that they include your id correctly that they lead to the right product pages. Check that you respect the terms and conditions both of the platform and of the affiliate programs you work with. Also, we recommend that you prepare and communicate both the resources that you created, as well as the requirements that you have both from your team and from external collaborators, as much time ahead as you can. The next component of our magic equation is “time and money”. In this period We invest time and money to make sure we get good results. First of all, we pay to host high traffic resources with external providers that scale very well In example, we use Amazon S3 and Cloudinary to host images, we use 3 performance and uptime monitoring services, that alert us if something doesn’t work correctly. And because we have our servers in EC2, the scaling part is a lot easier than in traditional hosting. This is our production server fleet, that we use to navigate in a regular day, it’s 17 servers; For Black Friday, sometime ahead we add new servers, that we “fatten”, as I like to say: we give them 4 times more memory and 3 times more CPU In addition, we have a special server fleet, all installed, prepared, in stand-by, ready to go into production really fast if needed. From performance testing we have numbers the traffic and load that require adding more servers So far, we haven’t used the special fleet, but every year we have it ready, in case we need it. We also have an investment from the team from Thursday to Sunday the technical team is available: we split in two, the day team and the night team. The day team works the regular hours and, at least in theory, sleeps at night. (they don’t buy games, phones and other stuff). The night team comes to the office and… gets bored. We hope this year the boredom-meter will be at least as high as last year. What I recommend regarding this direction: First of all: allocate time and money for performance monitoring As a technical person I recommend doing this all the time, not only doing Black Friday To scale highly used resources, move them to external providers. Invest in performance optimization. Last year we started testing “nightmare scenarios”. We take all the things that went wrong, from a technical point of view, throughout the years we add our imagination – what we fear the most, what might go wrong to make it really bad, and… we make it happen, on the testing environment. We stop services, destroy connections, we pretend external partners are down And this yields a lot of benefits: first of all, we can see first hand what’s going on, we can improve both the code and the way we respond to errors. We take time to decide who does what in such circumstances, both technically and regarding communication with our colleagues and with customers. We hope that none of this scenarios will happen, as we improved a lot of things this way. I also recommend to allocate time to communicate with your partners: the technical team, the external partners, 2Performant, with affiliates, or the stores you work with. And this is a great time to create new relationships, both before Black Friday, as well as while it’s going on. I hope I have inspired to you with this advice, I think that if you implement at least some of it, you will be in the same situation that we find ourselves in: we hope that our main problem will be, again… “What are we going to eat this Black Friday?”