"The battlefield is a great place for liars," Stonewall Jackson once said on viewing the aftermath of a battle in the American civil war.

The great general meant that the confusion of battle is such that anybody can claim anything during a war and hope to get away with it. But even by the standards of other conflicts, Iraq has been particularly fertile in lies. Going by the claims of President George Bush, the war should long be over since his infamous "Mission Accomplished" speech on 1 May 2003. In fact most of the 1,600 US dead and 12,000 wounded have become casualties in the following two years.

The ferocious resistance encountered last week by the 1,000-strong US marine task force trying to fight its way into villages around the towns of Qaim and Obeidi in western Iraq shows that the war is far from over. So far nine marines have been killed in the week-long campaign, while another US soldier was killed and four wounded in central Iraq on Friday. Meanwhile, a car bomb targeting a police patrol exploded in central Baghdad yesterday, killing at least five Iraqis and injuring 12.

Iraqi President Jalal Talabani, the leader of one of the Kurdish parties, confidently told a meeting in Brasilia last week that there is war in only three or four out of 18 Iraqi provinces. Back in Baghdad Mr Talabani, an experienced guerrilla leader, has deployed no fewer than 3,000 Kurdish soldiers or peshmerga around his residence in case of attack. One visitor was amused to hear the newly elected President interrupt his own relentlessly upbeat account of government achievements to snap orders to his aides on the correct positioning of troops and heavy weapons around his house.

There is no doubt that the US has failed to win the war. Much of Iraq is a bloody no man's land. The army has not been able to secure the short highway to the airport, though it is the most important road in the country, linking the US civil headquarters in the Green Zone with its military HQ at Camp Victory.

Ironically, the extent of US failure to control Iraq is masked by the fact that it is too dangerous for the foreign media to venture out of central Baghdad. Some have retreated to the supposed safety of the Green Zone. Mr Bush can claim that no news is good news, though in fact the precise opposite is true.

Embedded journalism fosters false optimism. It means reporters are only present where American troops are active, though US forces seldom venture into much of Iraq. Embedded correspondents bravely covered the storming of Fallujah by US marines last November and rightly portrayed it as a US military success. But the outside world remained largely unaware, because no reporters were present with US forces, that at the same moment an insurgent offensive had captured most of Mosul, a city five times larger than Fallujah.

Why has the vastly expensive and heavily equipped US army failed militarily in Iraq? After the crescendo of violence over the past month there should be no doubts that the US has not quashed the insurgents whom for two years American military spokesmen have portrayed as a hunted remnant of Saddam Hussein's regime assisted by foreign fighters.

The failure was in part political. Immediately after the fall of Saddam Hussein polls showed that Iraqis were evenly divided on whether they had been liberated or occupied. Eighteen months later the great majority both of Sunni and Shia said they had been occupied, and they did not like it. Every time I visited a spot where an American soldier had been killed or a US vehicle destroyed there were crowds of young men and children screaming their delight. "I am a poor man but I am going home to cook a chicken to celebrate," said one man as he stood by the spot marked with the blood of an American soldier who had just been shot to death.

Many of the resistance groups are bigoted Sunni Arab fanatics who see Shia as well as US soldiers as infidels whom it is a religious duty to kill. Others are led by officers from Saddam's brutal security forces. But Washington never appreciated the fact that the US occupation was so unpopular that even the most unsavoury groups received popular support.

From the start, there was something dysfunctional about the American armed forces. They could not adapt themselves to Iraq. Their massive firepower meant they won any set-piece battle, but it also meant that they accidentally killed so many Iraqi civilians that they were the recruiting sergeants of the resistance. The army denied counting Iraqi civilian dead, which might be helpful in dealing with American public opinion. But Iraqis knew how many of their people were dying.

The US war machine was over-armed. I once saw a unit trying to restore order at a petrol station where there was a fist fight between Iraqi drivers over queue-jumping (given that people sometimes sleep two nights in their cars waiting to fill a tank, tempers were understandably frayed). In one corner was a massive howitzer, its barrel capable of hurling a shell 30km, which the soldiers had brought along for this minor policing exercise.

The US army was designed to fight a high-technology blitzkrieg, but not much else. It required large quantities of supplies and its supply lines were vulnerable to roadside bombs. Combat engineers, essentially sappers, lamented that they had received absolutely no training in doing this. Even conventional mine detectors did not work. Roadsides in Iraq are full of metal because Iraqi drivers normally dispose of soft drink cans out the window. Sappers were reduced to prodding the soil nervously with titanium rods like wizards' wands. Because of poor intelligence and excessive firepower, American operations all became exercises in collective punishment. At first the US did not realise that all Iraqi men have guns and they considered possession of a weapon a sign of hostile intention towards the occupation. They confiscated as suspicious large quantities of cash in farmers' houses, not realising that Iraqis often keep the family fortune at home in $100 bills ever since Saddam Hussein closed the banks before the Gulf war and, when they reopened, Iraqi dinar deposits were almost worthless.

The US army was also too thin on the ground. It has 145,000 men in Iraq, but reportedly only half of these are combat troops. During the heavily publicised assault on Fallujah the US forces drained the rest of Iraq of its soldiers. "We discovered the US troops had suddenly abandoned the main road between Kirkuk and Baghdad without telling anybody," said one indignant observer. "It promptly fell under the control of the insurgents."

The army acts as a sort of fire brigade, briefly effective in dousing the flames, but always moving on before they are fully extinguished. There are only about 6,000 US soldiers in Nineveh province, of which Mosul is the capital and which has a population of three million. For the election on 30 January, US reserves arriving in Iraq were all sent to Mosul to raise the level to 15,000 to prevent any uprising in the city. They succeeded in doing so but were then promptly withdrawn.

The shortage of US forces has a political explanation. Before the war Donald Rumsfeld, the Secretary of Defence, and his neo-conservative allies derided generals who said an occupation force numbering hundreds of thousands would be necessary to hold Iraq. When they were proved wrong they dealt with failure by denying it had taken place.

There is a sense of bitterness among many US National Guardsmen that they have been shanghaied into fighting in a dangerous war. I was leaving the Green Zone one day when one came up to me and said he noticed that I had a limp and kindly offered to show me a quicker way to the main gate. As we walked along he politely asked the cause of my disability. I explained I had had polio many years ago. He sighed and said he too had had his share of bad luck. Since he looked hale and hearty this surprised me. "Yes," he said bitterly. "My bad luck was that I joined the Washington State National Guard which had not been called up since 1945. Two months later they sent me here where I stand good chance of being killed."

The solution for the White House has been to build up an Iraqi force to take the place of US soldiers. This has been the policy since the autumn of 2003 and it has repeatedly failed. In April 2004, during the first fight for Fallujah, the Iraqi army battalions either mutinied before going to the city or refused to fight against fellow Iraqis once there. In Mosul in November 2004 the 14,000 police force melted away during the insurgent offensive, abandoning 30 police stations and $40m in equipment. Now the US is trying again. By the end of next year an Iraqi army and police force totalling 300,000 should be trained and ready to fight. Already they are much more evident in the streets of Baghdad and other cities.

The problem is that the troops are often based on militias which have a sectarian or ethnic base. The best troops are Kurdish peshmerga. Shia units are often connected with the Badr Brigade which fought on the side of Iran in the Iran-Iraq war. When 14 Sunni farmers from the Dulaimi tribe were found executed in Baghdad a week ago the Interior Ministry had to deny what was widely believed, that they had been killed by a Shia police unit.

The greatest failure of the US in Iraq is not that mistakes were made but that its political system has proved incapable of redressing them. Neither Mr Rumsfeld nor his lieutenants have been sacked. Paul Wolfowitz, under-secretary of defence and architect of the war, has been promoted to the World Bank.

Almost exactly a century ago the Russian empire fought a war with Japan in the belief that a swift victory would strengthen the powers-that-be in St Petersburg. Instead the Tsar's armies met defeat. Russian generals, who said that their tactic of charging Japanese machine guns with sabre-wielding cavalry had failed only because their men had attacked with insufficient brio, held their jobs. In Iraq, American generals and their political masters of demonstrable incompetence are not fired. The US is turning out to be much less of a military and political superpower than the rest of the world had supposed.