You've probably noticed that ChatGPT sometimes gives you brilliant answers and sometimes produces nonsense. The difference often comes down to how you ask.

Researchers recently put this observation to the test in a systematic way, examining how university students interact with large language models like ChatGPT and whether the quality of their instructions—what AI experts call "prompts"—actually affects the quality of the responses they receive. The answer was striking: the way you craft your requests to an AI system matters enormously. But here's the catch: most people, even students, don't know how to do it well, and the reasons why reveal something unexpected about how humans naturally interact with machines.

The study involved 45 university students who were asked to complete two tasks using ChatGPT. One was to plan a four-day trip to Andorra. The other was to help plan a research project on automated essay scoring. Students were told to use the AI to do most of the work, copying their prompts and the AI's responses into a structured form for analysis.

What the researchers discovered goes beyond confirming that good prompts produce good results. It revealed that students unconsciously treat AI systems like human conversation partners, using polite phrases and questions rather than clear instructions. Many students asked the AI "What are the cheapest ways to get to Andorra?" instead of instructing it to "Create a detailed itinerary for a budget-friendly four-day trip to Andorra, including transportation, accommodations, and activities." These are fundamentally different approaches.

The Quality Gap That Nobody Talks About

When researchers analyzed the prompts students wrote, they found that prompt quality predicted output quality with remarkable consistency. For the travel planning task, students who wrote higher quality prompts received outputs that were substantially better. The effect was even stronger for the research project task.

But here's what makes this finding sobering: the average quality of student prompts was mediocre. On a scale of 0 to 6, where 6 meant a prompt included all the essential components of good instruction, students averaged around 3. Yet when asked afterward, most students said they didn't find it difficult to write prompts at all. They felt confident in their ability to interact with the system.

This gap between perceived confidence and actual competence is a red flag for education. If students are getting poor results from AI systems but don't realize their prompting is the problem, they may blame the technology, miss learning opportunities, or develop misconceptions about what these tools can do.

Why People Treat Robots Like Humans

The qualitative analysis of student prompts revealed something psychologically interesting. When students began working with ChatGPT, many of them incorporated polite social elements—"Hi," "Thank you," "Please"—into their requests. Some even asked the AI for its opinion, as though consulting a knowledgeable friend rather than requesting information from a computational system.

One student wrote: "I like research question 5 because it addresses the issue of the objectivity of machines. Can you give me two more similar questions for an undergraduate thesis?" This phrasing assumes the AI cares about the student's preferences and reasoning, almost as if the system has genuine interests and feelings.

This tendency is so consistent that researchers have a name for it: anthropomorphism. The conversational interface of ChatGPT, with its natural language responses and human-like writing style, naturally encourages people to project human qualities onto the system. But when you're trying to get precise, useful output from a machine, treating it like a person can work against you.

The reality is that AI systems don't have preferences or feelings. They don't understand politeness or interpret your inner motivations. They respond to explicit instructions about what you want. Knowing this difference—understanding that you're communicating with a tool, not a person—could dramatically improve your results.

The Role of AI Literacy

The researchers also tested whether something called "AI literacy" made a difference. This is essentially your understanding of how AI systems work, what they can and cannot do, and how to interact with them effectively.

The results were mixed and somewhat surprising. Having broad AI knowledge didn't consistently improve prompt writing or output quality across both tasks. However, one specific component of AI literacy did matter: understanding the different roles that AI can play in human-AI collaboration and knowing the distinction between AI technology and other kinds of technology.

Students with this particular knowledge tended to write better prompts for the travel planning task. But this advantage didn't carry over to the research project task, raising questions about whether the relationship between AI literacy and effective prompting is more complex than a simple cause-and-effect.

One possible explanation is that AI literacy might matter for different purposes in different contexts. Or it could be that prompt engineering is partly learnable through practice and experience, independent of how much you formally know about AI.

A Different Way to Ask

The researchers identified six components that should ideally appear in well-crafted prompts. These are: a clear verb indicating what action you want the AI to take, a focus describing what the action is about, context explaining the scope or parameters, a specific focus and condition narrowing what you want, alignment with your desired goal, and constraints or limitations the AI should follow.

When students were asked to plan a trip, a prompt with all six components might look like: "Create a realistic four-day itinerary for visiting Andorra in September that stays within a budget of €1,000, includes a mix of outdoor activities and cultural sites, and is suitable for solo travelers. Format as a day-by-day schedule. Do not include expensive luxury hotels or restaurants."

Compare that to what many students actually wrote: "What are the best places to visit in Andorra?" or "Can you help me plan a trip?" The difference is stark. The detailed prompt gives the AI clear direction. The simple question leaves interpretation up to the system.

Interestingly, when analyzing the structure of student prompts, researchers found that most prompts were phrased as questions, roughly 40 to 50 percent across both tasks. Questions can be useful, but they often don't provide the same level of explicit instruction as imperative statements or detailed descriptions.

Why This Matters for Education

The study raises an important question for educators: should colleges and universities teach prompt engineering as a skill?

The researchers argue yes. They contend that as generative AI becomes increasingly prevalent in both education and the workplace, knowing how to interact effectively with these systems is becoming essential. Students who graduate without this skill will be at a disadvantage. They may struggle to use AI tools effectively for research, writing, problem-solving, and other tasks central to knowledge work.

Moreover, the study found that students were remarkably interested in using AI, rated their overall experience positively, and said they would use these tools again. This suggests that generative AI is not a passing novelty in education. It's here to stay. The question is whether institutions will provide formal instruction on how to use it well.

The researchers also suggest that education in AI literacy—understanding how these systems work and their limitations—could improve prompt engineering. When students understand that AI systems have no consciousness or preferences, and that they respond to explicit instructions rather than implicit requests, they may naturally begin writing different kinds of prompts.

The Practical Path Forward

The findings suggest several practical takeaways. First, if you're using ChatGPT or similar systems, the time you invest in refining your prompts is time well spent. Clear, specific, detailed instructions produce better results. Generic questions produce generic or unhelpful responses.

Second, perceived ease of use is not the same as actual competence. Just because you can write something down and get a response back doesn't mean you're using the tool effectively. Testing your approach and iterating on your prompts—trying different wordings and comparing results—is a normal part of using these systems well.

Third, recognizing that you're communicating with a computational tool, not a person, can liberate you to write more effective prompts. You don't need to be polite to a machine. You need to be precise, clear, and explicit about what you want.

For educators, the research suggests that incorporating instruction in prompt engineering and AI literacy into existing curricula could give students valuable skills for the future. The fact that students naturally approach AI as though it were human, rather than understanding its computational nature, suggests this education is necessary, not optional.

The study examined only 45 students and focused on two relatively straightforward tasks, so it has limitations. Prompt engineering might work differently for more complex tasks or in different contexts. But the core insight stands: how you ask an AI system to work matters, and most people could be doing it better.

As artificial intelligence tools become woven into education, work, and daily life, the ability to communicate effectively with them will likely become as important as typing or searching the internet. The students in this study were learning in real time, without formal instruction, how to interact with a transformative technology. The question now is whether education systems will catch up and teach these skills deliberately, or whether the next generation will have to figure it out on their own.

Credit & Disclaimer: This article is a popular science summary written to make peer-reviewed research accessible to a broad audience. All scientific facts, findings, and conclusions presented here are drawn directly and accurately from the original research paper. Readers are strongly encouraged to consult the full research article for complete data, methodologies, and scientific detail. The article can be accessed through https://doi.org/10.1016/j.caeai.2024.100225