ALEXSOFT INSIGHTS
Why does the same AI produce such different results?
Even with the same AI, results depend on the knowledge and experience of the person explaining the work and checking the output. Reflections from software development on the difference instructions and feedback make.
Many people now use generative AI such as Claude, ChatGPT, and Gemini at work.
The change is particularly noticeable in software development.
Work that once meant writing code ourselves now often means explaining it to AI, then reviewing and revising the code it produces.
Recently, I have also been using AI in almost every area of my work, from development to planning and project management.
One question keeps coming back to me.
“Why do people produce such different levels of work with exactly the same AI?”
At first, I thought it was simply the difference between people who could write good prompts and those who could not.
(There is some truth to that, of course.)
But the longer I have used AI, the less adequate that explanation has seemed.
What I feel these days is that assigning work to AI is surprisingly similar to assigning work to a colleague.
How you assign the work changes the result.
Anyone who has worked in an organization will recognize this.
Simply telling someone, “Please make something like this,”
and explaining why it is needed, who will use it, how it connects to the work before and after it,
and which parts matter most lead to very different outcomes.
In fact, I think feedback makes an even bigger difference than the initial instructions.
Saying, “Something feels off. Please do it again,”
is very different from saying, “This structure may become a constraint as the business grows. We should separate this responsibility and change the data structure along with it.”
These are fundamentally different kinds of feedback.
The same applies to AI.
Give it a vague request, and it will produce something that looks quite plausible on the surface.
The key is being able to judge whether that output is actually good.
AI does not always make a wrong answer look strange.
The more difficult problem is that it can make a wrong answer look convincing.
You need to understand the work to direct it well.
The more I use AI, the more often I find myself thinking this:
To give AI good instructions, I need to understand the work myself.
And to give useful feedback on what it produces, I need even more knowledge and experience.
Software development is a good example.
Ask AI to design a system, and it will propose an architecture, design a database, and even write the code.
Again, it looks quite convincing at first.
But an experienced person notices other things.
It may work now but run into trouble as the data grows.
The real business process may not work that way.
The structure may cause problems when it connects to other systems.
Or important exceptions may be missing.
Ultimately, someone has to be able to recognize that there is a problem.
That judgment does not come simply from knowing a lot of facts.
It grows through building things,
experiencing failures,
seeing problems arise in production,
maintaining systems with poor designs,
and dealing with requirements that keep changing.
(After all, knowledge truly becomes our own when we experience it firsthand.)
Looking back, I often wonder why I made some of the decisions I did.
But those experiences helped me make different choices the next time.
They are also what I draw on when I give AI feedback.
AI can supply depth, but people must choose the direction.
Another thing I notice as I use AI more is that we have less and less need to keep every technical detail in our heads.
When we do not know something, we can ask AI.
We can research technical documentation, get an explanation of an unfamiliar field, or even try writing working code.
We can explore an area much faster than before, when learning it might have meant spending weeks with a thick textbook.
That does not mean human knowledge has become less important.
If anything, the opposite is true:
The knowledge and experience to judge where to look, what matters, and which direction to take are becoming more important.
AI can supply a considerable amount of depth when we need it.
But deciding what deserves that depth is still our responsibility.
Perhaps this change is also increasing the value of connecting multiple fields, alongside detailed knowledge of a single specialty.
I will write more about that separately.
What matters most is not AI itself.
No matter how capable AI becomes,
people still need to decide what to build,
why it should be built,
which direction it should take,
and whether the result is actually good.
That human role remains important.
In the past, the ability to make things ourselves carried more weight. That balance is now shifting.
Understanding the problem,
structuring it,
setting a direction,
giving AI precise instructions,
and validating the result:
these abilities are becoming increasingly important.
That is why I see AI as a very powerful tool rather than a replacement for people.
Just as with any tool, the result can be completely different depending on who uses it.
This is how we use AI at ALEXSOFT, too.
We draw on our experience in software development, business analysis, and business planning to use AI for faster, better results, rather than handing everything over to it.
On this blog, I will continue sharing what I have learned and felt through actual work: thoughts on AI and software development, ways to turn business processes into systems, and the differences between good systems and poor ones.