Real Agent Cases

Complete a Qualtrics survey with OpenAI Operator

Posted
by @Ingar30
Last checked
Method:
How we verify

Evidence: Screenshots or logsThe source shows screenshots, a transcript or the agent's output.

Outcome

The poster, a survey researcher, asked Operator to complete a Qualtrics survey they had created; they call the result very promising for Operator but very concerning for survey researchers.

Agent

OpenAI Operator

Prompt Recreated prompt

The source didn't share its exact prompt, so we wrote this from what it describes.

Run this task now, exactly as written below. Everything you need is here, so start straight away and only stop if something required is missing.

Goal: Test my survey end to end like a respondent would.

Fill in before running: Which survey link, and what kind of respondent should I act as?

What I need from you:
- Survey link
- Respondent profile to follow
- Whether this is a test or live link

Rules:
- Answer every question consistently with the profile.
- Note any confusing question, broken logic or errors.
- Submit only if this is a test link.
- Time how long the survey takes.
- Flag any question where a bot could answer easily, as a data-quality risk.
- Don't submit more than one response.
- Answer with neutral test responses unless I give you specific answers.
- Note any question that is confusing, broken or forces an answer.

Output: Confirmation of completion and a list of problems spotted.

You'll know it worked when: the survey is completed and any issues are listed.

Source

https://x.com/Ingar30/status/1900096859697262976

Related cases

  • Test five web tasks with Project MarinerImproved promptEvidence: Shown on videoPartial

    In a test of 5 tasks, Project Mariner found YouTube video view counts, but failed to generate relevant video ideas, couldn't preview HTML properly, refused to log into email and couldn't send emails.

    Project Mariner · Research · Source date:

  • Report global NAND market share with Gemini Deep ResearchImproved promptEvidence: Screenshots or logsFailure

    Asked for a simple report on global NAND market share, Gemini Deep Research returned figures the poster called an incredible failure when compared with Sandisk's investor-day slide and Claude's answer to the same query.

    Gemini Deep Research · Research · Source date:

  • Write a one-page sales brief on a target company with ManusRecreated promptEvidence: Article or interview

    Asked for a B2B sales brief on Google for a tl;dv pitch, Manus produced a one-page brief with revenue, tech stack, key decision makers, recent announcements and a sourced section mapping Google's pain points to tl;dv features; the reviewer says it took two minutes.

    Poster says:About two minutes

    Manus · Research · Source date: