![](https://bharatshakti.in/wp-content/uploads/2024/03/LT-BSI-Defence.gif")
Editor’s Note
As militaries increasingly incorporate Artificial Intelligence (AI) into combat operations, the primary challenge may not be the capabilities of machines but rather the trust of humans. Drawing on military history, battlefield experience, and the latest developments in AI, the author, who previously served as the director of Artificial Intelligence in the Indian Army, contends that autonomous systems should be evaluated based on their performance in real-world scenarios rather than an unrealistic standard of perfection. The article also emphasises that future military risks may arise not only from an over-reliance on AI but also from a failure to trust and utilise it, especially when adversaries are already doing so.
𝗧𝗵𝗲 𝗠𝗼𝘀𝘁 𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻 𝗶𝗻 𝗠𝗶𝗹𝗶𝘁𝗮𝗿𝘆 𝗔𝗜
Almost every discussion of Artificial Intelligence in warfare eventually comes to the same question: Can we trust the machine? Governments debate autonomous weapons. Military leaders discuss human control. Ethicists warn about delegating life-and-death decisions to algorithms. Technology companies publish responsible AI frameworks. Yet beneath all these discussions lies a deeper and far more consequential question – How is trust created in the first place?
Military organisations are built upon trust. Soldiers trust commanders. Pilots trust wingmen. Sailors trust captains. Commanders trust staff. Entire military campaigns depend upon human beings placing their lives in the hands of others. Yet we rarely stop to examine how that trust emerged. The future of autonomous warfare may therefore depend less upon advances in Artificial Intelligence and more upon our understanding of trust itself.
The decisive challenge may not be teaching machines how to fight. It may be teaching humans how to trust them.
𝗧𝗵𝗲 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 𝘄𝗶𝘁𝗵 𝗖𝗼𝗺𝗽𝗮𝗿𝗶𝗻𝗴 𝗠𝗮𝗰𝗵𝗶𝗻𝗲𝘀 𝘁𝗼 𝗣𝗲𝗿𝗳𝗲𝗰𝘁 𝗛𝘂𝗺𝗮𝗻𝘀
One of the most common arguments against autonomous systems is that machines make mistakes. This concern is entirely valid. Sensors fail. Algorithms misclassify targets. Data can be corrupted. Software can produce unexpected outcomes. Yet an uncomfortable question immediately follows – Compared to what? The benchmark against which AI is often judged is a mythical human decision-maker who is perfectly rational, fully informed, immune to fatigue and capable of making flawless decisions under pressure. Such a person does not exist. Real battlefield decisions are made by human beings operating under uncertainty, stress, fear, information overload and severe time constraints.
History provides numerous reminders of human fallibility. In September 1983, Soviet early-warning systems reported a nuclear missile attack from the United States. Lieutenant Colonel Stanislav Petrov chose to distrust the machine. His judgement proved correct and may have prevented a catastrophe. Yet history also provides examples pointing in the opposite direction.
In February 2019, an Indian Air Force Mi-17 V5 helicopter was shot down by friendly air defence systems near Budgam during a period of heightened tension following the Balakot airstrike. Seven lives were lost. Artificial Intelligence did not cause the incident. It was caused by human error operating within the fog and friction of combat.
Recent conflicts in the Middle East have produced similar cases of fratricide and misidentification amid immense complexity.
The benchmark for military AI is therefore not perfection. The benchmark is human performance under battlefield conditions.
𝗧𝗵𝗲 𝗦𝗼𝗹𝗱𝗶𝗲𝗿 𝗮𝗻𝗱 𝘁𝗵𝗲 𝗠𝗮𝗰𝗵𝗶𝗻𝗲
Imagine a future patrol moving through hostile territory. Beside the soldiers moves an autonomous combat vehicle carrying sensors, communications systems and a weapon station. It never becomes tired. It never loses concentration. It never experiences fear. It never panics. It can process information faster than any human operator and react faster than any soldier. Yet despite all these advantages, a fundamental question remains unanswered. Will the soldiers trust it?
The soldiers are unlikely to be asking whether the machine is intelligent. Their concerns will be far more immediate and personal. Will it recognise me as friendly? Will it understand my commander’s intent? Will it behave correctly when communications are disrupted? Will it stop when ordered? Will it know when it is uncertain?
Most importantly, can I trust it with my life? This question lies at the heart of autonomous warfare. The challenge is not merely technological. It is psychological. The future battlefield may not be defined by the intelligence of machines but by the willingness of humans to rely upon them when lives are at stake.
𝗧𝗿𝘂𝘀𝘁 𝗜𝘀 𝗡𝗼𝘁 𝗢𝗻𝗲 𝗧𝗵𝗶𝗻𝗴
A critical mistake in many discussions about military AI is the assumption that trust is binary. It is not. Trust exists on a spectrum and changes dramatically depending on the role being performed. A soldier may completely trust a robotic mule carrying ammunition while remaining uncomfortable with an autonomous weapon system. A commander may trust an AI system to process intelligence while refusing to allow it to authorise lethal engagements.
The reason is simple. Trust is determined less by technological sophistication and more by the consequences of failure. Autonomous logistics systems require relatively low levels of trust because failures are usually recoverable. ISR systems require greater trust because incorrect information may influence decisions. Decision-support systems require even greater trust because they shape courses of action. Autonomous weapons require very high levels of trust because the consequences of error can be immediate and irreversible. Strategic decision-support systems, particularly those associated with escalation and deterrence, demand the highest levels of trust of all.
It suggests that autonomy is unlikely to arrive as a single revolutionary event. Instead, it will probably emerge gradually through a trust ladder. First, machines will carry our loads. Then they will show us the battlefield. Then they will recommend actions. Then they will coordinate systems. Then they will fight alongside us. Only much later may they be trusted with decisions carrying major strategic consequences.
The future debate is therefore not whether soldiers will trust machines. Soldiers already trust machines every day. The real question is how much authority they are willing to delegate as the consequences of failure increase.
𝗧𝗿𝘂𝘀𝘁 𝗛𝗮𝘀 𝗔𝗹𝘄𝗮𝘆𝘀 𝗕𝗲𝗲𝗻 𝘁𝗵𝗲 𝗗𝗲𝗰𝗶𝘀𝗶𝘃𝗲 𝗧𝗲𝗰𝗵𝗻𝗼𝗹𝗼𝗴𝘆
To understand how trust develops, consider one of the most remarkable demonstrations of military trust in existence. During precision aerobatic displays, aircraft of the Blue Angels fly only a few feet apart while travelling at extraordinary speeds. In some manoeuvres, one aircraft may be flying inverted directly above another. A minor mistake by either pilot could prove fatal. Most observers ask how the aircraft flies upside down. The more important question is: How do the pilots trust each other enough to do it?
The pilot flying upside down cannot continuously verify every action of the pilot beneath him. At those speeds, there is neither time nor space. Trust becomes a necessity. That trust was not created by technology, nor by regulations, nor by courage. It was built through selection, training, repetition, testing and thousands of successful interactions. What once appeared impossible became routine. What once appeared dangerous became normal. The same process may ultimately govern autonomous warfare.
A soldier operating alongside an autonomous vehicle or a pilot flying with an autonomous wingman may initially be sceptical. Then familiarity develops. Then confidence. Then dependence. Trust is rarely a technological problem. More often, it is a familiarity problem.
𝗛𝗼𝘄 𝗠𝗶𝗹𝗶𝘁𝗮𝗿𝘆 𝗧𝗿𝘂𝘀𝘁 𝗜𝘀 𝗕𝘂𝗶𝗹𝘁
Military organisations have spent centuries learning how to build trust. It begins with selection, continues through training and testing, grows through shared experience and ultimately matures through operational performance. Soldiers do not trust one another because they are instructed to do so. They trust one another because repeated interactions create a sense of predictability. Over time, behaviour becomes understood. Confidence grows. Teams become cohesive.
The same process is likely to apply to autonomous systems. Soldiers will not trust AI because a manufacturer claims it is reliable. They will trust it because they have seen it perform thousands of times correctly. They will trust it because it behaves consistently under difficult conditions. They will trust it because it repeatedly demonstrates competence when uncertainty is highest. Transparency will also matter. Humans often tolerate mistakes from other humans because they understand why those mistakes occurred. Autonomous systems may therefore need to explain their reasoning, communicate confidence levels and identify uncertainty. The ability of a machine to explain itself may become almost as important as its ability to make decisions.
𝗧𝗿𝘂𝘀𝘁 𝗜𝘀 𝗔𝗹𝘀𝗼 𝗖𝘂𝗹𝘁𝘂𝗿𝗮𝗹
Trust is not determined solely by technology or military doctrine. It is also shaped by culture. Different societies have different relationships with authority, risk, institutions and decision-making. As a result, they may have very different attitudes towards autonomous systems even when the technology itself is identical.
Military organisations that emphasise decentralised decision-making and mission command may find it easier to delegate certain functions to machines because delegation is already embedded within their culture. Organisations built around tighter control may be more cautious because even delegating authority to humans has traditionally been limited. In some societies, technology is viewed as a source of reliability and objectivity. In others, human judgement remains the preferred foundation of trust.
This cultural dimension may prove strategically significant. Consider China’s apparent willingness to explore highly autonomous drone swarm concepts and AI-enabled battlefield decision systems. The emphasis often appears to be on operational effectiveness and system-level optimisation. Many Western discussions, by contrast, begin with questions of governance, safeguards and meaningful human control. Neither approach is inherently correct. They simply reflect different strategic cultures.
The future of autonomous warfare will not be determined solely by what machines can do. It will also be determined by what societies and militaries are willing to allow them to do.
The implications are profound. Two nations may possess equally capable AI systems. Yet the nation whose culture is more comfortable with delegating trust may realise operational advantages years before the other nation. The difference may not be technological. It may be psychological and cultural.
𝗧𝗵𝗲 𝗘𝗺𝗲𝗿𝗴𝗶𝗻𝗴 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗰 𝗥𝗶𝘀𝗸
For years, analysts focused on the dangers of trusting AI too much. Those risks remain real. Autonomous systems will fail. Algorithms will make mistakes. Technology will occasionally produce unintended consequences. Yet another risk is now emerging.
What if the greater danger is trusting AI too little?
Imagine two opposing militaries. One successfully integrates AI into intelligence fusion, targeting, logistics, communications and combat orchestration. The other insists upon preserving entirely human-centred decision cycles. One force operates at machine speed. The other operates at human speed. One coordinates thousands of autonomous systems simultaneously – the other struggles to coordinate hundreds. The resulting advantage may not be technological. It may be cognitive.
Recent reports regarding Chinese research into autonomous drone-swarm algorithms capable of identifying, prioritising and engaging targets under highly contested battlefield conditions illustrate how rapidly this debate is moving from theory to reality. The significance of such developments lies not merely in their ability to destroy targets. Their significance lies in the possibility that they may eventually perform certain battlefield functions better than humans.
Future military disadvantage may arise not because a force adopted AI too aggressively, but because it adopted AI too cautiously while its adversary moved faster.
The future of warfare is unlikely to be a contest between humans and machines. It will be a partnership between them. Humans will continue to define objectives, establish ethical boundaries and retain responsibility for military action. But machines may increasingly perform functions that require speed, scale, and information processing beyond human cognitive capacity.
It brings us back to the original question. Can we trust the machine?
Perhaps that is the wrong question.
History suggests that every revolutionary military capability begins as something people do not trust. Aircraft, radar, computers and drones all faced scepticism. Yet the technologies that ultimately transformed warfare became trusted so completely that future generations struggled to understand why anyone ever doubted them.
The future of autonomous warfare may not be determined by what machines are capable of doing. It may be determined by the moment a soldier decides whether he trusts the autonomous weapon beside him as much as the human standing next to him.
And if that moment arrives, military organisations may discover that the most important question is no longer whether AI can be trusted.
The most important question may be whether they can afford not to trust it.
Brigadier Narendra Pratap Singh (Retd) served as the director of Artificial Intelligence in the Indian Army and is currently the CEO of Mindsprintz Consulting
