Asking the Right Answers [#44]

I have been taking part in Google Rewards for over a year now. For the most part, I complete the various surveys to feed an ongoing habit without feeling like I’m being too indulgent or wasting money. It’s a fast and easy way to make a bit of completely disposable income and, honestly, the service works well.

Broadly, the surveys I get fall into three categories: store feedback, google reviews and marketing surveys. Store feedback is usually a case of confirming that I visited a given location and then rating them out of five. It’s quick, interesting enough to see which businesses feel the service is worthwhile and lets me provide some limited feedback. I don’t really imagine that the data is all that worthwhile, but enough stores do it, some of which having done so for an entire year at this point, that they must get something from the results.

Google reviews are a little more tedious but also have a higher reward, so I quite enjoy receiving them. I’m one of those people that routinely reviews online purchases, fills out in-store questionnaires and generally says “yes” when asked if I have a minute. I totally understand why most people ignore these types of things, but I try to do them whenever I have spare time for two main reasons. The first is that I’ve worked retail, I’ve been the person with the clipboard and I am fully aware how much that role sucks. I literally spent two months, for 4-5 hours a day, wandering around Durham trying to get people interested in taking a flyer for a store I worked for, and that was difficult enough. Getting people to actually engage with you for longer than ten seconds… that sounds like hell on Earth. The second reason is that I like having a record of my opinions, which should be fairly obvious from this website (and elsewhere), and that extends out to the services I’ve used and the items I’ve purchased.

So, the first two groups are easy for me to understand and pretty common. But once every month or so I’ll get a survey from group three: marketing research. Not market research, but questioning me on the adverts that I remember having seen or my awareness of brands. I imagine most of these are Google trying to gauge how well its own advertising algorithms are, something which is totally apparent when I get a survey like the one I received this morning.

That survey was incredibly quick and began by showing me a thumbnail of a Youtube video by Philip DeFranco. The video was several years old (I could see the uploaded date on the image) and the survey wanted to know if I had watched it. Now, I’ve been subscribed to Phil since I first created a Youtube account back in 2009 and had already been watching him for over a year before that. I quite literally created my account just to be able to track which of his back catalogue of videos I had watched. As a result, I could say with pretty high certainty that I had watched the video they were showing me. I also assume, considering that Youtube is tied to my Google account, that they already knew that I had watched the video. The first question on these surveys tend to request confirmation of known information, so that made sense.

But then they did something which I don’t understand, at all. I think what they were trying to do was refine their suggested videos algorithm but the way they went about it was just weird. There were two more questions to the survey and both showed another thumbnail of one of Phil’s videos from over a year ago. Both asked me to rate, out of five, how useful these would be as suggested videos on Youtube. Now, I don’t propose to understand the exact results or answers Google are looking for here, but I can imagine that they’re hoping to confirm that, yes, someone who wants to watch a video on current affairs would like to watch more videos on current affairs. The problem, though, is that their survey is completely ignoring my own video watching history. I am subscribed to Phil’s channel; I have watched every video he’s uploaded in the past decade. I don’t need to have his old videos suggested to me because I’ve already seen them. However, none of that information has been requested by the survey, so from the perspective of the questions I’ve been asked then, yes, based on the fact I enjoyed watching the first video I would want the other two videos to be suggested.

Yesterday I was reading an A List Apart article on why asking the right questions in user testing is key to not screwing up. Perhaps because that was on my mind, this survey through me round a loop. On a personal level, completely honestly, those videos are useless suggestions to me and I would have liked to rate them 0 out of 5 (which is, irritatingly, never an option). However, I’m a huge fan of Phil and want his channel to keep growing. Saying “Yes, I watched that one video of his and never want to watch another” seems wrong. I don’t want Google to take that message away from this survey. On the other hand, I hate how my current suggested videos feed is full of videos I’ve already seen and content from channels I’m already subscribed to. It’s a personal pet peeve of the current Youtube setup because it makes that page incredibly pointless, so I really don’t want to reinforce that behaviour and say that these are good suggestions.

At this point, I’m definitely over analysing what’s going on, but you would hope a company the size of Google would understand that the way they present a survey will have differing impacts. The questions are needlessly broad and non-specific, leaving the interpretation open to the user, but the subject matter leaves me stuck trying to guess what data Google actually want from me. Do they want me to know if I like those types of videos or do they want me to ‘confirm’ that suggesting other videos from channels I’ve watched before is a good thing? Unfortunately, I don’t know which it is, which means I don’t really know what the question is, and if I don’t know that, how can I answer it?

In the end, I just stuck them both at 4/5 stars. Typing this up now I feel that was probably the wrong thing to do, but oh well. At the end of the day, Google asked what seems like a fairly innocuous question, but one which has two wildly different answers. I doubt I’m the only person getting that question but I’ll probably be an outlier in my response. Still, it’s a prime example of where the phrasing, setting and simplicity of a question can leave it horribly ambiguous. The result will likely go on to inform some form of policy at Youtube, which is a shame, because no matter what question they thought they were asking I doubt it’s the one they’re actually having answered.

Design Tool Survey 2016

For the second year running, Khoi Vinh of subtraction.com is asking for designers/developers to fill out a survey detailing what tools they use in their day-to-day workflow. I’m probably not the true, intended participant, as it would be hard to argue that I am actively developing anything at the moment (dabbling here aside), but I’ve followed the process and outcome of the 2015 survey with interest and wanted to get a more hands-on feel for it this time around.

If you are at all interested, I’d definitely recommend it and you can do so here. Personally, I was a little disappointed by the lack of scope. The results of last year’s input were both insightful and very well presented, but now I’ve actually seen how those results were garnered I feel they’re certainly a little biased.

My main issue was the lack of personal detail requested. To be clear, the survey absolutely does not require email addresses, personal names, locations (outside of country) etc. and, correctly, refrains from these clear breaches of privacy. However, I would have though that determining the OS or main software environment people use would be fairly crucial. Similarly, asking what software you use right now is great, but I’d personally love to see what people want to use as well.

I’m a prime example of the warped outcome you can get without these details (if you ignore the fact that I’m barely an example, that is). My answers will likely group me in the box of “outdated dinosaur”; someone who is using the same tools now that they were a decade ago. Though this is largely the case (iPad aside), the reality, however, is that this isn’t my choice. I choose to use Windows as my core OS; I would do even if I was still actively freelancing as my main income source. But that means I can’t choose to move away from Adobe – at all. Would I preferentially wireframe in Sketch? Absolutely! But that isn’t an option because Sketch, like the vast majority of tools being surveyed, isn’t available to the vast majority of the world.

The gap between innovation and accessibility in the design world is becoming truly enormous, especially now even the iPad is undergoing large scale price hikes. Unfortunately, I don’t feel it will be possible for Subtraction’s survey to adequately reflect that issue, which I feel is a missed opportunity. Personally, I’d love to see how much of an impact this gulf is having in the West – it scares me to think how negatively impactful it may be elsewhere in the world!