I am delighted to announce the launch of our automated theme discovery tool, themeit.
themeit harnesses
the latest generative AI technology (think Chat GPT) to supercharge your
verbatim coding process.
How does it work?
Once you’ve loaded some data into codeit, you can access the new themeit tool by clicking “analyze themes”.
themeit will then auto analyze your data, extract a set of suggested themes and code as many of your verbatims as possible. Once this is complete, you can view the results in a simple and easy to use visualisation. You can view the results by each theme, and the verbatims that fit into each theme. It will also highlight the relevant section of text within the verbatim relating to the theme.
themeit screenshot
So, if you’re starting coding on a brand new project, themeit can save you hours of time by
auto-generating your codeframe and coding a large chunk of the verbatims for
you.
Extract | Refine | Apply
themeit is tightly integrated into all
the existing elements of codeit,
and works in a 3-step process: Extract – Refine – Apply:
Extract: AI-powered theme and sentiment
extractor autocodes your data and presents it in a visually engaging display.
Refine: Easily refine your results –
allowing for a better understanding of those nuances in your data that require
human judgement calls.
Apply:codeit’s unique machine-learning system continually improves its AI
model to incorporate your refinements. Saving you time, money, and boosting
quality.
How much can themeit actually code?
Every coding project is different, so it’s hard to
give a precise answer. However, typically, themeit
is capable of autocoding up to about 40% of your verbatims, at an accuracy
level comparable to human coders. Accuracy is an important consideration here
because as researchers, it’s important that we get accurate and reliable
numbers from our surveys. Similarly, as coders we don’t want to spend our time
correcting mistakes made by the AI. So, there’s a balance to be struck here. We
want the AI to do as much work as possible for us, but we need to restrain and
control it so it doesn’t run amok and create bad data. So, in designing themeit we’ve been very careful to get
this balance right.
Speed
up the coding process
themeit is designed to
speed up the coding process, but the process still remains human-led. You stay
in full control, so if you want to refine themes or alter the codes applied,
you can click the “Edit” mode tab, to switch into the low-level detailed data
view, and make any changes to the data you want. Any changes you make are
then immediately reflected in the themeit
view when you switch back.
themeit provides a huge productivity boost and allows us to deliver even more automation than ever before, without losing the power of human oversight and the value that people can bring to the process.
Future
Developments
As always, we’re interested in
feedback. Much like the main codeit
tool that has evolved in partnership with user feedback and involvement, we
would like users to help us evolve themeit
too with feedback, comments and suggestions.
themeit is available now and added to codeit explorer at no extra cost. If you have questions about the benefits of codeit and our new automated discovery tool, please get in touch with your Askia Key Account Manager.
As researchers, we
understand the value of asking open-ended
questions in our surveys. They
help us get to the “why” that lies behind the closed questions you ask. It’s no
use finding out that your customers are dissatisfied if you don’t also ask them
why.
The challenge with collecting this kind of free-text data is that it usually needs to be measured. For example, what are the top five most frequent reasons your customers are dissatisfied? How does that trend over time? And so on . . .
Turning qualitative data into quantitative measures usually involves coding the data, which is generally considered a time-consuming and painful process.
For several years, codeit has been helping Askia users overcome this challenge. Its in-built machine learning works alongside human coders, learning from them and automatically coding as much as it safely can, to considerably speed up the coding process.
This year, however, there’s a new kid on the block.
As you’ll no doubt be aware, “generative AI” (most notably, ChatGPT) has broken through into the mainstream and promises to disrupt many areas of the research world. Given the hype around this, surely we don’t need coders anymore? Now we can just pass our verbatim data through ChatGPT and it will automatically sift through it and spit out a perfect analysis in seconds, right?
So, can generative AI automate verbatim data analysis?
The short answer is, no – it can’t do a complete and accurate analysis on its own.
The longer answer is, if all you need is a high-level read of the data then generative AI is a reasonable tool to use. You can pass in your verbatims, ask it to summarise the main themes and you’re done.
But the problem with this approach is it’s still more qualitative than quantitative. If you need reliable quantitative measures from your data then we have to tread more carefully.
In our experiments, we have found that ChatGPT can generate a really good draft codeframe and autocode between 10% and 40% of your verbatims at a “human standard” level of accuracy. Whether you’re on the lower or higher end of this range depends on a number of factors. For example, the number of verbatims you have, the complexity of the text within them and whether you’re using GPT3 or GPT4 will all make a difference to the performance.
What about codeit’s machine learning?
If you’ve used codeit before, you will know that behind the scenes it contains a powerful machine learning capability using the latest deep learning techniques. As you code data codeit learns by example, and builds a model that is tuned to the nuances of your specific project, codeframe and coding examples. Once trained, codeit can use this model to autocode further verbatim data.
In our testing, codeit’s machine learning significantly outperforms ChatGPT with autocoding rates typically around 60%. The reason for this comes down to specialisation. ChatGPT is a generalist (“Wide AI” in the jargon) – a Jack of all trades, if you will. That means it can do lots of things quite well, but it isn’t exceptionally good at any one thing (it’s not going to beat you at Go anytime soon, for example). The machine learning built into codeit is a specialist (“Narrow AI” in the jargon) – it has been trained on your specific requirements and is optimised for the task of autocoding.
But here’s the rub – in order to do this, you need to feed it some example coding to learn from. This is great for large projects (1000+ verbatims), or on-going tracking studies, but for smaller, ad hoc projects it’s not as useful.
Clearly then, generative AI can help plug this gap. If you’re beginning a project from a standing start, with no historical data to teach the system, then generative AI can get us a long way down the line really quickly. Most likely you will want to review the output, make refinements and sense-check the results, but this is still a big leap forward and a huge help.
OK, so now what?
Our sense is that the world is beginning to get a handle on generative AI. Some of the hysteria of early 2023 is dying down, and people are gaining a better perspective on its strengths and weaknesses and exactly how it can be useful.
At codeit we feel we have also learned a lot this year and we’ve been putting that learning into practice within our software. Next month we will be unveiling some exciting new features that will bring all of this work to fruition. Watch this space, as they say, or as ChatGPT would say: “I’m sorry, but I do not have access to current news updates as my knowledge only goes up until September 2021.”
If you’re interested in seeing an early preview of our new features, please contact your Askia Key Account Manager.
Survey fraud is a serious problem that can have a significant impact on the quality of market research data – the very life blood of the insights industry. And it is a growing problem. Some industry commentators now believe that up to a third of the data being collected is being discarding because of quality concerns and panel fraud. This is a big deal.
There are several different ways to commit survey fraud,
including:
Impersonating respondents: This is when someone pretends to be someone else in order to take a survey. This can be done by using a fake name, address, or phone number and using a VPN that suggests they are coming from a location different to where they really are.
Responding multiple times: This is when someone takes a survey more than once. This can be done by using different email addresses and/or phone numbers and/or manipulating the survey link.
Giving false answers: This is when someone intentionally gives false answers to survey questions, often when they want to be able to qualify for a survey and reward. For example, when they declare that they have just bought a new car, when they don’t even drive.
‘Bots’ filling out online surveys.
This kind of survey fraud can be difficult to detect, but there are ways to reduce the risk of it happening to you. Here are some practical methods to consider:
Five steps during survey set-up . . .
Always try to keep surveys short, engaging and to the point. Even genuine respondents will get bored if the survey is too long or too dull and may start to give less reliable answers as they just try to finish the survey. This may be unintentional deception, but bad data is bad data – and this will affect project quality just the same.
If you are buying in sample, use reputable panel companies that require their members to validate their email addresses and use two-factor authentication. Ask your panel provider about how their members are verified.
Use Google’s CAPTCHA (Completely Automated Public Turing Test to Tell Computers and Humans Apart). This is a system intended to distinguish humans from bot activity. It works by presenting tasks (such as recognising and inputting displayed characters) that are straightforward for humans but challenging for bots, enhancing the reliability of the responses. See our KB article on adding this at the start of your online surveys.
Limit survey access. One way to ensure your respondents are genuine is to restrict access to surveys using unique URLs that contain access codes and/or participant IDs. This makes it much harder for bots or duplicate respondents to gain unauthorized access. Always encrypt your links, to prevent respondents from editing the links and using them multiple times. See this KB article for more details of prettifying and encrypting survey links.
Use smart questionnaire design. Strategic questions, including “red herring” questions, can catch out bots.
For instance, it could be a simple instruction like, “Please select the third response for this question”. A human that is paying attention will understand this and obey it, but bots (and humans) that are simply pattern-matching or picking answers at random would mostly fail, thereby revealing their non-human nature.
Including two similar questions throughout your survey can help reveal any inconsistencies in answers. For example, adding questions such as “in what year were you born?” and “please tell me your age?” at the beginning and end of the survey can help you distinguish authentic and fraudulent responses.
Five steps once the survey is in field . . .
Conduct both automated and manual data checks. It’s crucial to examine data for unusual response patterns, duplicate entries, or inconsistent data that may suggest fraud. Straight-lining is an obvious example of this.
Open questions are often a good place to start when searching for fraudulent activity. Check for repeated answers or general nonsensical answers or odd formatting including the use of bullet points. A neat trick is to check the number of keystrokes on the page and the number of characters in the open question box. If the number of characters in the open question box is higher than the number of keystrokes, then you know someone, or something, has done a cut and paste with the response rather than typed something out. We posted a recent blog article on this topic and provided the code in this KB article.
Use time-based response limits: automated processes and cheats will fill surveys faster than genuine respondents. Placing pragmatic time limits on survey completion can expose bots and cheats, who can be rejected, leaving only the respondents whose time to respond fits what we’d expect. In Askia you can put time checks anywhere in the survey, so you can test timings on a sectional basis or even question by question. You can also prevent page navigation for a specified duration, to try to prevent people speeding through each question. The JavaScript to perform this trick is described in this KB article.
Check IP Addresses: All devices connected to the internet use an IP address. Through tracking and monitoring this data, you can look for multiple responses from the same IP address, which could flag potential fraud. You can also check that the geolocation of the respondent matches where the respondent claims they are living. Many fraudsters will be using VPNs that disguise their IP Address, but not all of them.
Check Device Fingerprints: This process refers to identifying a respondent’s device based on its unique settings, configurations or activity, helping to track and flag suspicious or recurring device usage in your surveys. Askia gathers this metadata from a respondent, including the device type, screen size, browser and OS version. There are also some very good third-party solutions out there that provide additional digital fingerprinting safeguards and stop a suspected fraudulent respondent from even starting the survey.
In conclusion, although no single method can eliminate the
risk of fraud, employing a combination of these techniques can significantly
mitigate the risks and maintain the quality of your survey data. As with any
anti-fraud measures, the techniques will need to keep evolving to stay one step
ahead of the fraudsters, and Askia remains committed to doing all we can as a survey
software provider.
One of the most enjoyable aspects of my role is engaging with clients in person and gaining insight into how they utilise Askia. I am constantly amazed by the ingenuity of our users and the resourceful ways they optimise their survey businesses with our platform.
A recent visit to Warwick to meet with Consumer Insight provided a shining example of this. We were discussing the thorny issue of survey fraud and data quality with Craig Meikle and Jack Wood. I was relaying some of the excellent information that has been shared on this topic at the recent ASC Conference at the Oval last month. If you missed the conference, I highly recommend taking the time to watch the informative sessions, as all of them are available for free on YouTube: ASC Conference – Do Not Pass Go! Survey Fraud, Data Quality & Best Practice.
Craig and Jack are both passionate about data quality and highlighted how responses to open questions can be a good area to focus on when identifying poor or fraudulent respondents. In addition to examining open question responses for nonsensical comments, Jack mentioned a simple yet highly effective trick to combat fraudulent respondents and bots: he applies some relatively straightforward JavaScript to his open questions, comparing the number of keystrokes on the page with the number of characters entered into the open question box. This allows for easy detection of respondents or bots who have pasted a response instead of typing it out. The JavaScript is clever enough to recognise on-screen keystrokes via an on-screen keyboard.
Jack has generously allowed us to share this technique with other Askia users, as he believes it is in our collective interest to address survey cheating and stay ahead of those who seek to defraud us.
You can see an example of the code in action with this demo survey link.
It’s up to you how you choose to handle a respondent who clearly hasn’t typed in their response. You could remove them from the survey immediately or perform the same check post-completion, perhaps also reviewing a few other questions/responses to confirm the suspicions. One note of caution regarding this technique is that you should ensure it doesn’t interfere with respondents who require accessibility tools, such as speech-to-text solutions. The same goes for using our partner solution Phebi, which does automatic transcription on audio and video open questions. It wouldn’t make sense to use this technique on a Phebi open question.
For all the details of how to add this check to your open questions, then Jordan Grindle has written up a helpful KB article on this topic that includes an example QEX file that can be copied.
Once again, I’d like to express my gratitude to Jack and Craig at Consumer Insight for allowing us to share this valuable tip for survey quality.
Testing a survey thoroughly before it goes into field is an absolutely critical part of running a research project. Askia is therefore delighted to introduce a really useful option when testing a survey, whereby vital “back-end” information is displayed for the benefit of the survey testing team.
With this new Survey Testing Information option switched on, the name of the question shortcut is shown, as are the response entry codes. Perhaps most valuable of all, any response that has survey routing behind it, has the ability to show the routing logic that has been set by the scriptwriter.
This new mode should be really useful to all testing stakeholders, wanting to check that the survey matches and follows the questionnaire that has been provided by the research team on behalf of the client.
Our ADX Studio product owner, Jatin Bhatt, has recorded a short, three-minute demo showing how this mode can be set in AskiaDesign and demonstrates what appears on the screen when running through a survey with this mode switched on.
I define the vision of where the Askia software range is going. I also fix bugs in the code I wrote 20 years ago.
My proudest work achievement . . .
Founding Askia with Patrick and making it a home for so long for so many of my colleagues. It does sound paternalistic, but I can live with that. My even more paternalistic achievement is to be the father of four boys – two of them I am not related to biologically – and I tell you that’s work!
If I wasn’t working for Askia, I would probably be . . .
I’d like to think I’d be a game programmer, but more likely I’d be writing Cobol for a bank.
The best advice I ever received was . . .
Measure twice and ask a professional to cut it.
If I could time travel back 10 years into the past I would . . .
Still bitch about the weird aches in my joints.
My most interesting fact/stat . . .
At
the age of 30, on Christmas day, I found out I was Jewish through my maternal grandmother
who decided to do her coming out possibly to piss off my Catholic Polish father.
The soundtrack to my life is . . .
The Cure – A strange Day from the 1982 album Pornography
‘Filtered Weighting’ is an exciting new feature recently added to AskiaAnalyse. It removes the need for manual intervention when implementing certain weighting requirements.
Probably the best way to explain the feature is to show you a working example, so I’ve put together a five-minute video demonstration of the new feature which can be viewed below.
In the example, the requirement is to equally weight all but one of the responses of a question. The remaining response is to be left unweighted i.e. its observed percentage does not change before and after applying the weighting.
In the video I firstly show how we would have achieved this prior to the new feature using a method requiring a manual update when more data is received. I then show how we can achieve this now in a more automated fashion using the new filtered weighting feature.
For this particular weighting scenario, we think this feature will save a great deal of time and also reduce the chances of error, always inherent in a manual process.
I’m
one of the Support Engineers at Askia. I assist our clients with a variety of
technical topics, including (but not limited to) survey script writing, survey
design, fielding setup, software installations, data exports, data analysis,
etc. I also assist Askia and our clients with technical training, database
management, and server maintenance. I wear many hats.
My proudest work achievement . . .
Due
to a combination of luck, brute force, and experience, I’ve solved several
“needle in a haystack” installation issues that I’m quite proud of.
If I wasn’t working for Askia, I would be . . .
I’d
probably go back to school, or maybe work on further improving my existing Cloud
computing knowledge. I always thought it would be neat to run my own I.T.
consulting company too.
The best advice I ever received was . . .
Don’t be afraid to ask for help. If it’s important, keep three backups of it, in three separate locations.
If I could time travel 10 years in the past or future, I would tell/ask myself . . .
In the past: buy more Bitcoin! Invest in Artificial Intelligence! Purchase the cabin from your parents! Buy a house! Keep exercising!
In the future: does Metro ever finish building the train to LAX? What about the bullet train from Los Angeles to Vegas? What about a bullet train from Los Angeles to San Francisco? Choo choo …
My most interesting fact/stat . . .
I used to be a (pretty good) baseball pitcher (Little League and Junior League). I also used to be a first chair Clarinet and Saxophone player.
The soundtrack to my life is . . .
Too many to list! I’ll give you three of my favorites: 1. “Tura Lu” by Bollox.
2. “Data & Picard” by Pogo. 3. “Jack Sparrow (feat. Michael Bolton)” by The Lonely Island.
I
will often be listening to some “Classical Piano” or “LoFi” station while I’m
working.
The three things I would take to a desert island are . . .
Survival Guide for Dummies.
A book on “What to do with all this sand?!”
A “Stillsuit”, like those worn on the desert planet of Arrakis from the book “Dune” by Frank Herbert.
My favorite book/movie . . .
Book: “Ready Player One” by Ernest Cline. Movie: “Big Trouble in Little China (1986)”.
“Have you paid your dues, Jack? Yes sir, the check is in the mail.”
Those clever people at MyForce have launched a new version of their BISON Recording Management Tool. We think that many Askia call centre managers and supervisors will be thrilled to find out that managing quality control can be made so much easier. A great deal of effort has been made to develop a time-saving tool that is both user-friendly and powerful.
CATI
Supervisor Software
Call Centre managers already using the CTArchitect dialler alongside the Askia CATI solution will know about the increases in their call centre’s efficiency when using the two systems together. For example, the extensive monitoring options in the system help improve conversation quality. But supervisors will also likely confirm that making random quality checks can be a somewhat convoluted and time-consuming task.
BISON Recording
Management Tool (RMT)
So MyForce thought this should be made easier. With a user-friendly and intuitive interface, BISON RMT simplifies the whole process. A connection with the Askia environment allows a supervisor to immediately see the projects that have been assigned to them.
At a glance, the interface shows all the information that is typically needed for quality control. The interview data, the interaction details, the recording, etc. So there is no more clicking through interviews, noting down callIDs and downloading & managing large recording files. Moreover, every change that the supervisor makes, automatically flows to the Askia dataset.
The Bison Recording Management Tool neatly brings everything together, making quality control easier, more efficient, and less prone to errors. And if you have an efficient quality control then you will also have:
A nicer user experience for the supervisor.
An increase in the number of quality checks you
can achieve.
More time for coaching the agents.
And as a cherry on the cake, speech analytics can
also be added seamlessly, to further enhance the quality control process.
If you would like a demo or more details on how Bison can help reduce your costs and improve your quality control, then please contact your Askia key account manager.
Most of my career I have done HR and employee relations, but these past few years my focus has been more on key account management and office administration. This has not been a career change, but I must admit that I am enjoying it very much!
My
proudest work achievement . . .
To continue to have an amazing working relationship with my team and our US-based clients – it’s been almost eight years now.
If I
wasn’t working for Askia, I would be . . .
Volunteering in
Africa or the Caribbean.
The best
advice I ever received was . . .
Treat
others the way you would like to be treated.
If I
could time travel 10 years in the past or future, I would tell myself . . .
If I could travel back 10 years then I would probably try to learn French, as I think it is such a beautiful language.
As for 10 years in the future, I would tell myself to keep enjoying life to the fullest and don’t mind the little things.
My most
interesting fact/stat
My childhood dream was to become famous. I studied modelling as a teenager and my father wanted me to become a news anchor. But I have come to realise that I don’t actually like too much attention and direct focus on me.
The
soundtrack to my life is . . .
Save
the last dance for me by The Drifters.
The
three things I would take to a desert island are . . .
My favourite book Mr. Stratton Water
My favourite
book/movie . . .
Book: The Bible Movie: The Princess Bride
My superpower would be . . .
To fly above the white, beautiful clouds.
My party
trick . . .
Dancing. The
sound of music is wasted if it is not danced to, so you should always
dance to the music, even if it’s only in your heart.
