The United States spends more than $70 billion a year on the gathering and assessment of information about its enemiesand friends. Other nations lavish proportionate amounts, which can only increase now that cyberwarfare and information games have become inextricably entangled with intelligence and counterterrorism. China is estimated to employ some two million people on electronic data collection and surveillance, much of this directed at its own people.

The prodigious sales of books about intelligence and especially huminthuman intelligence, or spycraftreflect the popular fascination with the subject. Many of these focus on betrayers: Aldrich Ames and Kim Philby, Klaus Fuchs and the Rosenbergs, Bradleynow ChelseaManning and Edward Snowden. The Secret State: A History of Intelligence and Espionage, the latest book from the former British military intelligence officer Colonel John Hughes-Wilson, also emphasizes famous coups and notorious failures by one belligerent or another: Pearl Harbor in 1941; the US Navys 1942 signals intelligence, or sigint, triumph before Midway; the German failure ahead of D-Day in 1944; Israels blindness in advance of the 1973 Yom Kippur offensive by Egypt and Syria.

Intelligence possesses merit only when it assists statesmen and military commanders to adopt wise policies in peace or to defeat enemies in wars. Unless such empowerment can be demonstrated to the satisfaction of historians, all claims about espionage coups must be suspect or meaningless. Many spies enjoy exotic careers that excite authors and their readers, while making scant impact on their employers fortunes. The British petty crook Eddie Chapman, Agent Zigzag, had extraordinary World War II experiences as the plaything of British and German intelligence. At different times he put himself at the mercy of both, and afterward became the hero of a movie and several books. It seems nonetheless unlikely that his activities did much good to either side, serving only to keep Chapman himself in girls and shoe leather. He was an intriguing but wholly uninfluential figure, one among countless loose cannons on the secret battlefield.

As Hughes-Wilson notes, the foremost requirement of a good intelligence officer is that he should Speak Truth unto Power. This is least likely to happen, as Loch K. Johnson observes in Spy Watching: Intelligence Accountability in the United States, his deeply informed study of political oversight of US intelligence services, when governments enlist spymasters to pursue ideological agendas. Ronald Reagan sent his election campaign manager, William J. Casey, to run the CIA, and his tenure as director (19811987) was neither happy nor successful. George Tenet (19972004) did well enough in the same position until he joined the neocons campaign to topple Saddam Hussein, by endorsing the WMD fantasy.

Tenets counterpart Sir Richard Dearlove, director of the British Secret Intelligence Service, inflicted lasting injury on its image and credibility by being complicit in Tony Blairs deceit of his own people about the alleged threat from Iraq in 2002. Donald McLachlan, a British naval officer in World War II, observed: Intelligence has much in common with scholarship, and the standards which are demanded in scholarship are those which should be applied to intelligence. This they certainly were not on Dearloves watch, nor on that of his successor, Sir John Scarlett, who was also irredeemably tainted by the WMD scandal.

Truth should be respected by decision-makers and those who inform them not as a matter of morality, but instead as an indispensable navigational aid in every field of endeavor. Statesmen and corporate chiefs, as well as generals and spymasters, must sometimes lie to others, but court disaster when they delude themselves. This makes it all more droll how often they contrive to do so.

No responsible historian accepts the conspiracy theory that President Franklin Roosevelt deliberately permitted the Japanese attack on Pearl Harbor. It is nonetheless easy to understand the enduring fascination of this myth, because the negligence of all those responsible for defending Americaprominently including General George C. Marshall, the US Army chief of staffwas astounding. Yet those of us who write about such matters cease to be surprised by command blindness, whether in Eisenhowers headquarters before Hitlers 1944 Ardennes offensive; at Westmorelands Saigon command center in 1968 before and during the Tet Offensive; or in Israel ahead of the Yom Kippur War in 1973. A common denominator is cultural conceit: if a prospective enemy initiative makes no sense by ones own playbook, it is deemed implausible or impossible.

The nations that gathered and used information best in World War II, and arguably during the cold war as well, were those committed to intellectual honesty and the pursuit of truth, while those that failed were the dictatorships to which truth was inherently alien, unacceptable, antipathetic. Stalins spies garnered treasure troves of intelligence, including overwhelmingly convincing evidence of Hitlers intention to invade the Soviet Union in June 1941. Yet all their sacrificesand almost without exception they paid with their lives for their labors in the cause of international communismwent for naught, because in the relentlessly oppressive, black fantasy world of the Kremlin, unwelcome truths were no more acceptable than they were at Hitlers headquarters. Compare Winston Churchill: though Britains wartime prime minister was often angered by unpalatable viewpoints, never did he question the duty as well as the right of his subordinates, and especially of intelligence officers, to say what they thought.

Almost all historical studies of intelligence focus on things that did happen rather than those that did not. Authors often fail to explain to readers the muddle of information and misinformation that confronts political leaders and military commanders. There is a cacophony of noise, from which signalstruths large and smallmust be winnowed. In August 1939, on the eve of the NaziSoviet pact, a senior British official wrung his hands over the confused messages reaching the Foreign Office about relations between Berlin and Moscow: We find ourselves, wrote Sir Alexander Cadogan in phrases applicable to most intelligence,

when attempting to assess the value of these secret reports, somewhat in the position of the Captain of the Forty Thieves when, having put a chalk mark on Ali Babas door, he found that Morgana had put similar marks on all the doors in the street and had no indication which was the true one.

Wireless interception and codebreaking, which achieved maturity in Western Allied hands between 1942 and 1945, transformed the very nature of espionage. The British and Americans possessed no agents of influence in wartime Germany, but the Japanese ambassadors wireless dispatches from Berlin to Tokyo provided Washington and London with more useful insights into the Nazi hierarchys thinking than any spy could have done. By 1945 the old cloak and dagger, as MI5s Guy Liddell and a Bletchley Park friend described pre-war espionage in a fit of nostalgia, seemed almost redundant.

Not quite, however: highly placed agents in the enemys corridors of power, such as the Russian colonel Oleg Penkovsky, proved important Western assets during the cold war. In countering modern terrorism, humint penetration can be almost as important as phone and Internet surveillance. Hughes-Wilson describes how British intelligence during the Northern Ireland Troubles photographed IRA boss Joe Cahill in compromising positions with underage girls and used the threat of exposure to recruit him as a toutan informant for the security forces.

Any nations successes must be viewed against the background of hundreds of thousands of pages of trivia or outright nonsense that cross the desks of analysts, statesmen, and commanders. Diplomats and intelligence agents, in my experience, are even bigger liars than journalists, wrote the British wartime spy Malcolm Muggeridge, who was familiar with all three.

The futility of much spookery was nicely illustrated by Frantiek Moravec of Czech intelligence. Once, in 1936, he presented his commanding officer with a report on a new piece of German military equipment, for which he had paid an informant handsomely. The general skimmed it, then tossed across his desk the magazine Die Wehrmacht, pointed out an article on the same weapon, and said, The subscription is only twenty crowns.

This story reveals a vice to which most national leaders prove vulnerable: that of supposing information gained from secret sources, especially code-breaking, to be inherently superior to diplomatic and journalistic dispatches. It seems a historic blunder that both the US and British governments have today placed their diplomatic services in receivership, so that the demoralized personnel of both the State Department and the Foreign Office are fast shrinking in quality as well as quantity.

A prudently conducted intelligence analysis meshes both open and secret sources. During World War II the US State Departments regular bulletins on world affairs, circulated throughout the Roosevelt administration, were as informative as and often more sensible about geostrategic issues than any material on offer from the Western Allied secret services, though it deserves notice that the Research and Analysis Division of Wild Bill Donovans OSS did better work than most of its agents in the field.

The reporting of good media correspondents deserves at least as much respect as that from most CIA, SIS, or military intelligence officers. In a devastating passage of Public Affairs: The Military and the Media, 19621968, a US Army study of the media in Vietnam, William Hammond writes, Press reports wereoften more accurate than the public statements of the administration. He may have been thinking of the dispatches of _The New York Times_s Gene Roberts on the Tet Offensive in 1968. For much of February the newsman, only recently arrived in Vietnam, assessed the condition of American arms in Hue far more accurately than any intelligence officer or higher commander in the battle zone.

While the US State Department and the British Foreign Office can no longer attract their societies brightest and best, Western intelligence services also face difficulties in matching the brilliant recruits that become available only in times of national crisis. When the first volume of the British official history of wartime intelligence was published in 1979, I suggested to its principal author, Professor Harry Hinsley, himself a Bletchley Park veteran, that it seemed to show the amateursmany recruited from the academic communitycontributing much more to the secret war than career Secret Service professionals. Hinsley replied impatiently, Of course they did. You wouldnt want to suppose, would you, that in peacetime the best brains of our society wasted their lives in intelligence?

Hughes-Wilson provides a proficient survey of intelligence uses and abuses through the ages, but one of his assertions seems importantly mistaken: It is hard to lose a game of cards when you can see the other fellows cards. Not so. Kibitzing an opposing player does nothing to diminish the strength of his hand or, in intelligence, of his forces. The British and Americans in World War II were assisted by unprecedented knowledge of their enemies, secured through the work of the codebreakers at Bletchley Park, Arlington Hall, and the US Navys OP-20-G. But they could not secure victory until their armies, navies, and air forces were powerful enough to overwhelm those of the Axis.

Bentley Archive/Popperfoto/Getty Images

Kim Philby, London, circa 1955

In December 1941 codebreakers provided a vivid picture of Japanese intentions toward British imperial possessions in the Far East, but Churchills armed forces there were too weak effectively to exploit the forewarning. The British Air Ministry and MI6, in which a notable part was played by the great scientific intelligence officer Dr. R.V. Jones, by 1942 achieved a superbly accurate picture of Germanys linear and electronic defenses amid the RAF and USAAF strategic bomber offensive. But the Luftwaffe crumbled only in the spring and summer of 1944, when USAAF P-51 Mustang long-range fighters became available to shoot its aircraft out of the skies.

The Western Powers are unlikely to participate again in clashes of armies numbered in millions. Yet todays transfer of NATO resources from soldiers and warships toward intelligence and special forces, a redeployment driven by political lassitude about almost all threats save that from terrorists and a nuclear North Korea, seems recklessly overdone. While terrorists possess the power to cause our societies much grief, they do not pose an existential threat. By contrast, the risk of a 1914-style miscalculation by one or another government could precipitate a clash of the most terrifying kind. Yet both in Europe and the US, sub-nuclear hard power is increasingly insufficient to address, for instance, a Moscow-generated crisis in the Baltic states.

The significance of intelligence discoveries often hinges upon their timing: not merely who knew what, but when. Thus the copious archives of decrypted wartime Ultra and Magic messages can convey a misleading impression of Allied omniscience, without careful examination of dates on which signals were brokenoften too late in real time, behind the pace of events on the battlefield.

In the Enemys House, Howard Blums account of how Arlington Hall codebreaker Meredith Gardner and FBI supervisor Bob Lamphere exposed important portions of Soviet spy rings in the United States, might have laid more emphasis on the whens. Gardner got seriously to grips with the Russians operations only in 1946, and the FBI did not pull the plugs on most of them, including Julius and Ethel Rosenberg, until 19491950.

Yet for almost two decades previously, Moscows puppeteers had been directing an astonishing array of agents and traitorous American and British informants, who not only provided the data to build Stalins atomic bomb, but a torrent of other technological intelligence, especially about aviation. The memoirs of Soviet residents in the US comment with contempt on the FBIs efforts to monitor them. Such sources are heavily tainted, of course, and J. Edgar Hoovers men had a few successes, but the record speaks for itself. Until at least the end of the 1940s, Communist spy networks enjoyed almost free range across the US.

Blum, a former investigative reporter, spins an excellent yarn, and what he writes seems entirely convincing as far as it goes. But US counterintelligence operations against the Soviets were scarcely the triumph his book implies. The best case that can be made for the FBIs institutionalized bungling, together with the sustained leakage of Western state and commercial secrets, is that the Russians access represented part of the price that democracies must pay for indulging their freedoms.

Spy chiefs frequently deplore the constraints imposed upon their operations by meddling politicians: intelligence omelettes cannot be made without breaking some rotten eggs, they argue. Most intelligence officers are conscientious, honorable, reasonably smart people. Unfortunately, however, and partly because their world is shielded from outside scrutiny and bracing cold doses of common sense, a significant minority of the secret worlds practitioners go mad. Few lose their marbles as conspicuously and embarrassingly as the CIAs James Jesus Angleton did amid what he himself described as the intelligence wilderness of mirrors.

Angleton, born in 1917, was the son of a senior executive with the National Cash Register Corporation. He was educated at Britains Malvern public school, then at Yale. He spent formative time in Italy, where he sat at the feet of Ezra Pound and became imbued with more of the old mans fascism than his poetry. A witty young cosmopolitan, he gravitated naturally into the wartime OSS, and thereafter the CIA. Its long-serving director Richard Helms described him as the dominant counterintelligence figure in the non-communist world.

The Ghost, Jefferson Morleys shrewd account of Angletons career as Langleys counterintelligence chief from 1954 to 1975, shows the harm that can be done by an energetic spook who is permitted grossly excessive latitude. The Ghost focuses on two manifestations of this. First, Angleton became so close to the Israelis that he provided them with assistance that Morley believes was ill-judged, especially in establishing their nuclear weapons program with fissile material almost certainly illegally shipped from a plant in Pennsylvania. He writes:

If he learned anything of the secret program at Dimona, he reported very little of it. If he didnt ask questions about Israels actions, he wasnt doing his job. Instead of supporting US nuclear security policy, he ignored it.

Morley deplores the manner in which Angletons Zionism, in his view, distorted US strategy in the Middle East and bequeathed a nuclear legacy to the region of which effects will be felt for decades, if not centuries.

The second and even more notorious aspect of Angletons career was his belief in traitors within the US and British security worlds, which destroyed scores of careers unjustly, and extended to a witch hunt against British prime minister Harold Wilson, who he became convinced was a Soviet agent of influence. Angletons obsession with alleged enemies within Western societies infected a faction of Britains MI5, with equally pernicious consequences. Much of it started with the treason of the Cambridge Five, most notably Kim Philby, who became Angletons inseparable buddy during his years in Washington in the early 1950s. Philbys belated exposure wrecked Angletons judgment and equilibrium. If dear, lovely, boozy, wisecracking Kim was a traitor, then anybody could beand probably was.

Angleton did untold harm to public trust in US intelligence services when his excesses were revealed, most of them by a 1974 New York Times expos. He also gave evidence to the Senate Intelligence Committee, in which he acknowledged presiding over a program of mail interception focused on civil rights activists and antiVietnam War protesters. He shared Richard Nixons view that hundreds, perhaps thousands, of Americans, mostly under 30are determined to destroy our society.

The public revelation of Angletons paranoia and his wholly nonaccountable abuse of the powers of Americas secret state inflicted damage on the reputation of the US intelligence community that has never been wholly repaired. A consequence of the apparently endless Western intelligence blunders and misdeedsthe Cambridge Five, Tenets and Dearloves espousal of the Iraqi WMD fantasy, Angletons assault on the civil rights of law-abiding Americansis that it has become hard to persuade either the American or British people to take their intelligence services seriously, far less to trust their judgment.

Morley concludes his book damningly: Angletons most significant and enduring legacy was to legitimize mass surveillance of Americans. Both John Hughes-Wilson and Loch Johnson argue that the leakers Chelsea Manning and Edward Snowden are unconvincing crusaders for liberty, seeming instead to be mischief-makers who have done much harm to Western security interests. Yet the two still have apologists on both sides of the Atlantic, civil libertarians who decline to acknowledge that in a world in which terrorism has become endemic, some loss of personal privacy is a price we must pay for protection: electronic eavesdropping is almost the only effective prophylactic against those within our society who wish us harm.

Even ideologically driven career agents of foreign powers do not rouse much public indignation. The network of Russian sleepers within the US became stars of a fictionalized and not unsympathetic TV series, The Americans, after their exposure and deportation, assisted by the fact that one of them was a conspicuously glamorous young woman. The British suburban grandmother Melita Norwood was revealed in 1999 as a lifelong Communist agent, who through her job as a clerk at the British Non-Ferrous Metals Research Association had been able to pass to Russia as many vital nuclear secrets as Klaus Fuchs had. Yet the media and public reacted to her belated exposure with laughter, prompted by this eighty-seven-year-old renegades success in running rings around the establishment for half a century. The British authorities declared a prosecutionand implicitly also a prison sentenceinappropriate. While popular sentiment today demands that pedophiles should be imprisoned for life and sexual harassers of women become social outcasts, the betrayal of state secrets is viewed as a much less heinous sin.

Loch Johnson was foremost among the founders of the University of Georgias School of Public and International Affairs in 2001, and was previously engaged for most of his career in the congressional monitoring of intelligence. His Spy Watching will be read by professional spook-watchers rather than many members of the lay public. It nonetheless deserves close attention, because of the personal experience that underpins his judgments, together with their evenhandedness and common sense. He considers the past only to address the future: How can intelligence services that have been granted unprecedented powers since President George W. Bush launched his ill-named War on Terror be subjected to democratic scrutiny? How can the loss of public trust be restored? How can citizens be taught to recognize that the US, like all nations, possesses secrets that must be preserved for the common good; that absolute openness on the part of government and its institutions is the enemy of national security?

Johnson is scrupulously fair, observing, for instance:

There is at least a whiff of hypocrisy in the revulsion expressed in Washington about Moscows active measuresagainst the United States and its presidential election. The CIA has been aggressively engaged worldwide in the use of covert actions, including electoral and other political meddlingeven against fellow democracies.

He argues that while covert action is often justifiable against ISIS and other terrorist organizations, it should have no place in US policies toward other states. He shares with Hughes-Wilson a conviction that the torture of suspected terrorists approved by President Bush and Vice President Cheney, and since applauded by President Trump, is not merely morally indefensible, but also ineffectual. He urges that the ever-more promiscuous deployments of CIA drones to kill suspected terrorists cry out for a warrant process. He repeatedly expresses concerns about the partisan and reckless fashion in which Trump meddles with national security institutions and indeed treats them with contempt. The president has become, in his own fashion, at least as dangerous and anarchic a force as WikiLeaks in undermining confidence in essential arms of government.

Johnson urges the creation by statute of a broadly recruited Citizens Intelligence Advisory Board, with nine members chosen by the White House, the Supreme Court, and deans of schools of international affairs. There must be a recognized channel, he says, through which public-spirited whistle-blowers can vent concerns without embarking on a rampage of revelations such as Chelsea Manning indulged. His concluding pages are infused with passion:

It is incumbent upon all Americans to take a more active role in demanding the protection of this nations fundamental constitutional freedoms, electing only those presidents, senators, and representatives who vow to take intelligence accountability seriously. The citizens of this great nation must either fight for their right of privacy or lose it.

In a peroration that might well serve to conclude all of these volumes, Johnson writes, The intelligence community must always remember that its most important possession is its honor and reputation. When this is besmirched by torture, rendition, and unrestrained surveillance policies, the peoples of democracies implicitly or explicitly withdraw support for the legitimate, admirable, and indeed essential activities of the intelligence services in pursuit of the security of us all. If wars are too important to be left to the generals, intelligence is too important to be left to the spymasters.

The Secret State: A History of Intelligence and Espionage

by John Hughes-Wilson

Pegasus, 510 pp., $19.95 (paper)

Spy Watching: Intelligence Accountability in the United States

by Loch K. Johnson

Oxford University Press, 615 pp., $34.95

In the Enemys House: The Secret Saga of the FBI Agent and the Code Breaker Who Caught the Russian Spies

by Howard Blum

Harper, 317 pp., $29.99

The Ghost: The Secret Life of CIA Spymaster James Jesus Angleton

by Jefferson Morley

St. Martins, 328 pp., $27.99