Last Friday – I was a hunter – I knew something should go wrong.
But, let’s get back to square one.
Tomek Dubikowski inspired me with his talk “Proper gun makes testing fun” to think more about performance testing, as lack of those would concern any of us in the most surprising situations. Following slide from his presentations would give you very simple guide of what every tester should check, to make sure, that his website or any other service will survive unexpected circumstances.
This post won’t be a guideline about how to set up your environment in order to test the performance of your service – but rather a tour through bad examples and good practices in performance testing.
I am not an expert – I am just gathering knowledge!
Black Friday is quite a new term in Poland. It started to gain on popularity a year or two ago, maybe, but it became quite on spot last week. All regular shops in malls and most of the online shops with clothing, cosmetics or electronics started 20% discount weekend.
I am a mean tester, so I decided to start my hunt in the evening when traffic in online shops usually goes high. What did I hunt for? Dramatic errors, of course.
As you can guess, such discounts encouraged more customers to visit online shops and do the shopping. For those, that has not been prepared, it did not end well. Most websites that I’ve visited were significantly slower than usual, they had some visible issues or problems with loading pictures. Unfortunately, some of them did not pass the load tests at all.
I’ll provide you with three examples of what might happen, if you won’t test the performance of your websites, I hope you’ll prepare better for Christmas Sales 😀 (or whatever you wish).
All of them are taken from online stores – so the ‘only’ effects might be the shortage in the store’s income and customers’ trust for the brand.
When such bloomer happens to some notional institution – it may end up in the news and in multiple memes. In 2014 we had this situation with counting votes in regional elections. Our government decided to introduce online vote transfer from local committees which ended up with tremendous fail.
As the system was not supporting high loads of transfers (you had this one job), we’ve been waiting for election’s results for a week.
One of my favorite memes from that annoying time – “It works on my machine”:
Let’s get back to Black Friday issue, shall we?
What can I do
I would divide those performance problems into two groups (both customer – facing):
- It doesn’t work and you have a real problem here
- Tell it to your customer in some human-readable way
Don’t get me wrong – if that happens to the service of yours – you’re f*** anyway 😀 It depends on you, though, how do you communicate it to your customer and if he would eventually come back to you or not.
If your service doesn’t have performance checked – too many visitors result with too many transactions for its capacity, which finally terminates the service with a timeout. Reserved.com ‘died ‘without a warning and prompted us with extremely professional ‘504 Gateway Time-out‘.
Douglas.pl died as well, but somebody had at least done his job on UI beforehand and displayed a user-friendly information ‘We are sorry’. Better – but still…
Most of their sites displayed this kind of messages anyway. At least, they tried.
One of our Polish Facebook Group about testing – “Testowanie Oprogramowania” member – Artek Zwoliński – spotted fantastic fail as well at X-KOM – online store. X-kom informed us even what kind of database do they use as well 🙂
We may make fun of it, but those examples show the real problem with lack of performance testing and lack of effective solutions for peaks and unusual loads on websites. On the one hand, brand owners may claim that those are unique situations within the year and performance testing is too expensive to do it, on the other hand, it is just risky for the brand and customer’s trust.
Performance testing becomes quite a trendy thing now, so you can easily use online courses or watch some tutorials about how to get along with JMeter or Gatling. However, these are just the tools. It is you – who should come up with ideas and foresee the need for testing performance. Do the risk analysis with your business and try to convince them about the need and benefits.
In addition, do your homework and don’t be sloppy in preparations. Equip yourself with knowledge and prepare (buy) an isolated environment, that will be 100% accurate to your production. If not – performance testing does not have sense, sorry. No matter how much will you invest in learning JMeter’s functions (it is already old-fashioned tool BTW).
The isolated environment is the key to successful performance testing, otherwise, happy with meaningless results, you’ll be testing your service live with end users, as many online shops did on Black Friday. 🙂
We, as testers, must be aware of such risks and know what might happen if we skip performance testing part, especially, when we develop a service, which has even a slight chance to be on top for some reason (sale, promotion, unique product etc).
Last, but not least, preserve any results of performance testing, as it may be useful in the future. The best idea is to store all the results directly in the database (for example in MongoDB) in order to have it accessible and ready for analysis.
A cheat sheet of performance testing
- Know your PURPOSE
- Isolate the environment
- An environment must be 100% same with production
- Optimize the configuration
- Store data
- Repeat the test at least once
- Get professionals to analyze your results
Yeah, I should get some shiny table for that 🙂
I hope it was useful and will guide you a little more into performance testing.
In case of any comments – let me know down below or stalk me on Twitter of Facebook.