Following up from our round-up of day one of the eMetrics Summit in London we’re back with more talks from the second day. Today’s topics include A/B testing, measuring visitor intent, data confessions and more.
We will be updating this blog post through the day with each of the talks.
A/B Testing – the proven approach from Wooga
By Alexandra Paun (@ToastingIdeas)
- Pre Test Significances: A/A Testing. Pre test significances (testing your control group against your test group prior to implementing the test). This can identify biases in your subjects.
- Don’t be blinded by increased revenue – consider what impact this is having on other aspects of your business. E.g a game from Wooga gave the option to buy a pass to the next level or invite 3 friends. More people were buying their way through which created increased revenue, but this meant they were losing out on increased player numbers.
- The opportunity cost test: could this be making more money? Dropping your price on one product can lead to the canabilisation of another. In a game, cheap plasma bubbles meant less fire bubbles were being brought. This meant users were getting bored quicker.
- How long you run your test for depends on your KPIs and what it is you are looking to achieve from the test. It can be beneficial to run the test longer than necessary, just keeping a very small control group. This will allow you to gain extra insight on the decision you would have made. Even if you get the data you think you need, always ask more questions. Ask more questions about your tests than you need to.
- AB Testing is beneficial even when other tests have been carried out. For example, user feedback may suggest that a feature should be implemented. But once the feedback is implemented users start to drop. But without the AB testing you can’t determine if it was the new satire which caused the drop or something else independent. Consider the impact your changes may have on the wider environment which can affect your statistics. E.g. An update to your game may affect where it is shown on Facebook. This will then effect your rate of new users. So the change of Facebook has caused the drop, but the change of Facebook was caused by the change to your game.
Maximum Marketing Impact with £5K?
By Barbara Pezzi (@barbarapezzotta)
- Buzzwords do not translate to business outcomes, directors want to know what your work is providing the business.
- Choose your goals, (ROI, Exposure, Retention), decide what metrics are key to measuring the progress of those goals, decide what delivery mechanism is best to achieve this goal, ensure all the necessary tracking is in place to measure those metrics.
- Identify any obstacles before any campaigns go live (landing pages a bad etc). This allows you to determine realistic expectations.
- More traffic is not always better. E.g. Sending traffic to a hotel where the rooms are all sold out. This is wasting your budget. Identify issues where receiving the traffic is not necessarily beneficial.
- Look at what regions within your target area are converting and target those regions only – this means you will not waste your budgets on clicks which won’t convert.
- Tips: 1. Make the most of scripts – stop ads when your traffic is less likely to convert. E.g. Stop bidding on keyword “holidays in Mexico” if there is a hurricane in Mexico. 2. Use consumerbarometer.com to see how users search in countries you don’t know about. 3. Make the most of multi-funnel attributions to determine where more budget should be allocated (particularly good at getting budget for social).
Visitor Intent: Smart clues for understanding customer journeys
By Carmen Mardiros (@carmenmardiros)
- Visitor intent is your customer’s agenda. This will change throughout their lifetime as your customer. Your website experience must match the visitor intent. Different jobs for different customer intents. This allows you to satisfy their agenda.
- Intent matters most when uncertainty about customer journey is high.
- The best insights are often on the visits which don’t convert, therefore understanding intent allows us to set goals to view these by setting up micro goals.
- Train your eye to look out for the basic being behind an interaction and what it means about the intent.
- Segment by first and early actions. The actions taken by users early on in their visit can be an indicator of their intent.
- Consider what is the best indicator of success within each segment of intent. It may not always be a purchase – what will satisfy that users visit.
Three Months to Success
By Xavier Colomes (@xavier_colomes)
- In the first 3 months we are seen by the different areas of the company as leaders of change and strategic business partners. So it is important to focus and do less to achieve more. 80% of success will come from only 20% of changes – this must happen due to a lack of budget and resource. Why 3 months? Because you have access to the right people, your projects are prioritised faster and you have great impact. Therefore we must make the most of this period. It is important to work efficiently in this time before we are seen as just one of the employees.
- Define (on boarding, the plan, tools, validations), Automate (extraction, dashboards, automation, education), Deliver (the deep dive, segments, VOC, testing plan). Defining and automating means we can get rid of the tasks in the future that do not add value and then we can focus on where the most value is created – delivering.
- If you have to fight one battle when implementing web Analytics – Fight for a tag management solution.
- Software such as Google Analytics and SiteCatalyst are web Analytics data gathering tools and should not be used for analysis. Extract this data into Excel and combine it with other valuable data sources such as CRM etc. And perform your analysis on this aggregated data.
- When creating dashboards only include data which should be acted on; alerts and critical few. This stops wasted time. Also, do not report data to people that they do not require, even if they have asked for it, The do not understand it, they do not know how to interpret it. This will just lead to wasted time and poorly supported decision making. If necessary, teach the office to view the web Analytics data themselves.
Confessions of a Data Geek
By Craig Sullivan (@OptimiseOrDie)
- Have a can do attitude – use the data to pull teams together and break down the silos.
- The first John Lewis website wasted almost one million pounds.
- Keep users at the heart of everything that you do.
- The no assholes rule: get rid of the assholes who bring the office aura down.
- Develop your skills in analogies, similes and metaphors – make your data simple before presenting it.