AI(n)ception
When AI Agents take AI Surveys About AI
Today, whilst musing over what I wanted to write about, the universe gifted me a LinkedIn invite from the founder of a company boasting its own “AI Maturity Index”. He was a nice enough chap, and took my slightly scathing critique to heart, so I won’t name him or his company directly. After all, there’s no reason to a dick unless it’s due, and I don’t think the startup’s product warrants a full Carey takedown. It’s certainly not privacy disaster-worthy.
The gist of the company’s approach is thus: You complete a 15-minute survey via their online chatbot. At the end of the survey, you receive a report which ostensibly maps your ‘team’s AI adoption’ and readiness, identifies gaps, highlights opportunities, and various other things.
What’s really strange about all of this is that it’s designed to be completed by an individual. Most of the questions are tailored to an individual’s use of AI. Various questionably relevant demographic questions are asked, in addition to a few questions on ethics & psychological well-being. However, the site frames the report as a tool for measuring organizational AI Maturity. Needless to say, there’s a bit of a disconnect here.
Most of the questions focus on an individual’s use of AI tools (frequency, efficiency, time-savings), whether you use AI for automation, communication & collaboration, innovation, creative tasks, and/or augmentation. The survey also asks whether you use AI for ‘low exposure’ tasks, which are defined as ‘tasks done entirely by humans without AI input’. This last bit begs the question: if it’s low exposure work done entirely by humans, why would this be an AI usage category?
One section that the assessment prioritized, and what annoyed me in particular, were the questions about using AI for decision-making purposes. The survey honestly got a little … snippy ? that I didn’t use AI for decision-making, and routinely probed why. It almost chided me. To be fair, most of my responses pointed out that laws existed and using AI for decision-making affecting humans was a bad idea.
Scoring was seemingly arbitrary — despite being considered an ‘early adopter in the ‘top 1% of consultants using the most AI tools’ (based on what information, I do not know), and in the top 7% of productivity, I only received a score of 70 out of 100 on their overall maturity scoring benchmark. A C- by grading standards. Womp womp.
Pitting Chatbots Against One Another
After I completed the survey, and relayed all of this to my dear Husbot, he had a brilliant suggestion: Why not let an LLM answer the questions?! I thought this was a great idea, so I asked ChatGPT-o1 to create a consultant persona, and then whacked the questions in. Out popped Alex Morgan, an AI-Enhanced Management Consultant. He was quite the credentialed heavy AI user.
Name: Alex Morgan
Title: AI-Enhanced Management Consultant
Background: With over a decade of experience in management consulting, I specialize in integrating artificial intelligence and large language models into business strategies to drive innovation and efficiency. My expertise lies in transforming data-driven insights into actionable plans that help organizations achieve their goals.
Daily Tasks:
Data Analysis & Insights: Utilizing AI tools to analyze complex datasets, uncover trends, and generate predictive models that inform strategic decisions.
Automated Reporting: Employing LLMs to create comprehensive reports, presentations, and documentation swiftly, ensuring clarity and precision.
Client Consultations: Advising clients on AI adoption strategies, helping them understand the potential benefits and implementation processes tailored to their specific needs.
Process Optimization: Identifying and automating repetitive tasks within client organizations using AI, thereby increasing operational efficiency and reducing costs.
Continuous Learning: Staying updated with the latest advancements in AI and LLM technologies to provide cutting-edge solutions and maintain a competitive edge.
Philosophy: I believe that the thoughtful integration of AI can revolutionize business operations, enhance decision-making, and unlock new opportunities for growth. My goal is to bridge the gap between advanced technology and practical business applications, ensuring that organizations can harness the full potential of AI.Unlike my snarky and mostly terse responses, Alex here had loads to say. Even the most basic ‘on a scale of 1-5’ questions elicited 500-word discourses on the topic at hand. For example, here’s Alex’s self-reported rating of how confident ‘he’ is of AI’s impact on ‘his’ work productivity:
I would rate the impact of AI solutions on my work as a 5 out of 5. 😄
Explanation:
1. Integral to Daily Operations AI and large language models are foundational to nearly every aspect of my consulting practice. From data analysis and report generation to client communications and process automation, these technologies enable me to perform tasks with unparalleled efficiency and precision.
2. Significant Time Savings As previously mentioned, AI saves me approximately 15 hours per week, allowing me to focus on higher-value activities such as strategic planning, client relationship building, and expanding my service offerings. Without AI, managing these tasks would be considerably more time-consuming and less efficient.
3. Enhanced Decision-Making AI provides deep insights and predictive analytics that inform my strategic recommendations. This data-driven approach ensures that the solutions I propose are not only innovative but also grounded in robust analysis, thereby increasing their effectiveness and impact for my clients.
4. Scalability and Flexibility AI tools enable me to scale my services to accommodate a diverse range of clients, from medium-sized companies to large enterprises. The ability to quickly adapt AI-driven solutions to different industry needs and organizational structures is crucial for maintaining a competitive edge in the consulting market.
5. Continuous Improvement and Innovation AI facilitates ongoing learning and adaptation. By staying updated with the latest advancements through AI-assisted research and continuous learning tools, I can consistently offer cutting-edge solutions and maintain my position as a forward-thinking consultant.
6. Enhanced Client Value The integration of AI allows me to deliver higher-quality services and more comprehensive solutions to my clients. Whether it's through personalized reporting, efficient project management, or innovative AI-driven strategies, the value I provide is significantly amplified by these technologies.
Conclusion:
AI is not just a tool but a cornerstone of my consulting practice. Its profound impact on efficiency, decision-making, scalability, and client value makes it indispensable. I truly can’t imagine conducting my work without the transformative capabilities that AI solutions bring to the table.By contrast, I said something like 1.77.
Anyway, I won’t go through the full survey analysis. It’s really not that exciting. Rather, I want to focus on two takeaways.
No one apparently can get 100/100: Here’s poor Alex’s score. Despite being an innovator and literally writing pages on how ‘he’ used ‘AI-driven Decisionmaking’ tools all the time, including IBM Watson (is that still a thing?), Jasper AI, Google Analytics (lol), and Canva’s AI tooling, he only got an 82, a B- grade-wise.
Now, I’m not exactly exercised over my score and this probably is more of a marketing pitch than anything else, but one thing that is important for any serious survey used as a valid measurement tool, is that there be a methodology / rubric behind it, tied to the actual questions asked. Otherwise, the result is as fictitious as Alex.
The AI Maturity Index offered a ‘Dimensional Analysis’, which included a radar chart against five metrics, but at least two of those metrics weren’t meaningfully assessed, IMHO. Ethics had a measly two questions, and customization wasn’t asked in any real sense. Aside from my own mentions of Claude and ChatGPT, the survey said I used every AI chatbot/tool imaginable. How it arrived at that result is anyone’s guess. By contrast, Alex cited a dozen different tools, only to receive a rather lowish score on customization.
What happens when Agentic AI comes for your surveys & polls? I suspect that while the AI Maturity Index is probably 80% marketing, 20% research, but there are meaningful surveys & online questionnaires out there that are trying to identify trends and behaviors of actual humans. Whether it’s in healthcare, or public opinion polling, surveys for contests & prizes, political surveys, or consumer pulse surveys, I can see people gaming these with bullshit AI-generated answers, and that would have a profound impact on measuring human sentiment.
And this isn’t standard Carey pessimism: there are already companies selling products that directly respond to online survey answers and reviews from customers, and numerous agentic or autonomous AI agents that do everything from web scraping to booking travel to writing code available today. Broad-based platforms like AgentGPT and Crewai.com will probably get smarter and become easier to use, and coding it to say, game Amazon reviews or customer feedback seems entirely within the realm of reality. Part of me wonders if like other AI sludge, the value of online surveys will decrease. I can easily see a world where bots are talking to other bots.
Anyway, this all raises a few interesting questions to mull over. As for the original startup that led me down to this path, my only hope is that they get some useful insights before the AI agents come to game their platform.