It may be worth abandoning hopes of a career in prompt engineering as new research from VMware labs shows that AI models are far better suited to generating their own prompts.
Based on tests which compared human-generated “positive thinking” prompts against “systematic prompt optimization,” or AI-generated prompts, researchers found that automated processes yielded better results.
Lead researchers Rick Battle and Teja Gollapudi found that using an “automated prompt optimizer” was the most effective method of “enhancing performance” regardless of model size.
The study also found that the AI optimized prompts which scored the best on their testing metrics also exhibited an unexpected “degree of peculiarity.”
“As anticipated, the prompts that underwent automatic optimization consistently equaled or surpassed the effectiveness of our manually generated “positive thinking” prompts in nearly all instances,” the research said.
Although the paper described the act of prompt engineering as an “enjoyable endeavor,” researchers also argued that the study clearly shows human prompt engineering to be “time-inefficient.”
Developers relying on GitHub Copilot could be creating dangerously flawed code
This was especially true, researchers said, when systematically assessing all the modifications from a scientific standpoint. Instances where human prompt engineering bested automated prompts were limited to a handful of tests on Mistral-7B and Llama2-70B.
The research indicated almost no clear trend when approaching prompt engineering from a human-generated approach - “the only real trend may be no trend,” it stated.
In light of the data, the two researchers put forth their rationale for prompt engineering as a whole.
“What’s best for any given model, dataset, and prompting strategy is likely to be specific to the particular combination at hand. Thus, we turned from hand-tuning the system message with optimistic “positive thinking” to automatic prompt optimization,” the research said.
Prompt engineering isn’t a safe bet on the job market
Prompt engineering is a hugely important process in AI development, as the fine tuning of AI inputs helps models to operate at their fullest extent, utilizing potentially untapped aspects of their data, according to Bartek Roszak, head of AI at STX Next.
“Prompt engineering is significant because it enables us to extract precisely what we need from LLMs,” he told ITPro.
As this new research suggests, though, this is a task that’s increasingly vulnerable to automation from the exact technology it seeks to utilize.
Expanding on his research, Battle concluded that humans should never have to manually optimize a prompt again. Battle further criticized the process of a human prompt engineer who wastes time having to “figure out” an effective combination of words.
Battle hopes this research will convince prospective prompt engineers to move beyond manual prompt engineering, advising AI users to instead “develop a scoring metric” for the model to be able to train itself.
Roszak also looked to a future of AI autonomy in the realm of prompt engineering, suggesting that this will eventually become an obsolete skill.
“Ideally, everyone could interact with LLMs, and if someone were to provide a vague prompt, the LLM would engage by asking for details and assist in refining the prompt itself,” Roszak said.
“Such advancements could lead to a future where in-depth prompt engineering is no longer a specialized skill” he added.
If AI’s can train themselves to create their own prompts, the role of the human prompt engineer could be dead in the water.
That's not to say, though, that there aren’t other areas rife for human involvement in the realm of AI optimization.
Solve application modernization challenges with generative AI
‘The evolution of career paths in AI will likely pivot towards oversight, ethical programming, and the innovative application of AI outputs,” Peter Wood, CTO at Spectrum Search, told ITPro.
A different type of prompt engineer might stick around as well, according to Peter Van Der Putten, lead scientist and director at Pegasystems. Speaking to ITPro, Van Der Putten said that long-term there will likely be a sharpened focus on more of an overarching, enterprise-level understanding of prompt engineering.
“Prompt engineers will focus a lot more on industrialisation, ranging from building dynamic prompt templates, connecting AI to knowledge bases and search engines, to building out advanced agents' infrastructures,” Van Der Putten said.
Though prompt engineering is still a necessity in the burgeoning field of AI training and usage, staff in the tech industry will need to make sure they stay ahead of the curve if they want their skills to be useful.
“At this moment, the work of a prompt engineer is key to the effective utilization of LLMs; however, research is actively pursuing ways to make this job obsolete,” Roszak said.
