← All projects

HCI research · Human–AI interaction

Directive vs. Reflective AI

How does the way an AI gives help affect how people think?

We compared two AI programming assistants: one that gave direct solutions and one that encouraged reflection. Our six-person pilot explored how conversational style relates to cognitive engagement, task time, and perceived collaboration.

Role

HCI researcher and co-author; shared responsibility for recruitment, study sessions, surveys, analysis, and writing

Team

Cory Cowden, Elyssa Abbott, Saanika Medishetty

Context

CSC 486 research project, Cal Poly San Luis Obispo

Method

Randomized between-subjects pilot, six participants, three per condition

Outputs

Custom AI conditions, research paper, presentation

01

I contributed across the research process, from recruiting participants to interpreting and communicating results.

Our three-person team shared the research work evenly. I recruited participants, ran study sessions, contributed to surveys and analysis, and co-wrote the research paper. The custom AI setup was handled by a teammate.

02

We asked whether prompting reflection would lead to greater cognitive engagement.

An assistant can offer a complete solution or help someone work toward one. We wanted to examine whether that difference changed how actively programmers felt they were reasoning, developing ideas, and understanding their solutions.

We hypothesized that the reflective assistant would produce higher self-reported engagement and that the directive assistant would support faster task completion.

03

We changed the conversational instructions while keeping the underlying model the same.

Both assistants used the same base model with different custom system prompts. This let us focus the comparison on the intended interaction style rather than comparing different models.

Directive AIReflective AI
Gives concrete answers and complete solutions when askedAsks clarifying questions and encourages reasoning
Minimizes questioningOffers hints before complete solutions
Prioritizes efficient assistancePrompts users to explain their approach

These describe the designed behaviors, not verbatim excerpts from participant conversations.

04

Each participant used one assistant to solve the same programming problem.

We randomly assigned six fourth-year Computer Science or Software Engineering students to one of the two conditions. Participants were not told that two versions existed. Each completed the same medium-difficulty LeetCode problem, *Letter Combinations of a Phone Number*, using their assigned assistant.

Giving each person only one condition avoided having their experience with the first assistant influence how they approached the second.

Pre-task survey → Programming task with assigned AI → Post-task survey → Analysis

05

We measured perceived engagement separately from task time and collaboration.

The engagement survey asked whether participants felt they were actively thinking, mentally challenged, developing their own ideas, and understanding why the solution worked. Collaboration items asked whether the AI felt helpful, improved problem solving, and was a style they would use again.

We also collected self-reported task times and completion status. These measures describe participants’ experience and perceived task success; they do not establish learning gains or independently verified solution correctness.

06

The results did not support our prediction about reflective AI.

The directive group reported slightly higher engagement. The reflective group spent about four minutes longer on the task on average, and collaboration ratings were close across conditions. None of the comparisons reached statistical significance.

MeasureDirective averageReflective average
Self-reported task time18.00 minutes22.33 minutes
Cognitive engagement, 1–54.003.67
Collaboration quality, 1–52.893.00

With only three participants per condition, these descriptive differences are exploratory. They do not establish that either assistant style is better.

07

The unexpected direction raised a better question about what engagement means.

Our hypothesis emphasized reflection, but direct assistance might support engagement in other ways, such as helping someone check an approach or move past frustration. These are possible explanations to investigate, not mechanisms demonstrated by our data.

The study shifted the question toward which forms of engagement different AI behaviors support, and how we could observe those behaviors more directly.

08

The pilot showed where our measurement approach needed to become stronger.

Our small, relatively similar sample limited the conclusions we could draw. Self-reported times introduced measurement uncertainty, and completion status could not reliably substitute for independent correctness scoring. The study did not establish changes in code reliance or longer-term learning.

I would strengthen a follow-up with a larger, more varied sample, automatic timing and interaction logs, consistent solution scoring, and an independent follow-up task to assess understanding after AI assistance ends.

09

The design opportunity is to investigate when each style helps.

Rather than recommending one conversational style for everyone, this pilot motivates testing how assistance should respond to a programmer’s goals, experience, and stage of work. An adaptive assistant that shifts between hints and direct help is a future design direction, not a validated outcome of this study.

Next projectCáritas Valencia Mobile App ↗