In the wake of The Bulletin's damning exposi of Australia's intelligence services and the attempted character assassination of Lieutenant Colonel Lance Collins, come further explosive charges against the military and its political masters. John Lyons reports.

Something very odd is happening at the top of Australia's defence forces. The men and women who have no fear of going into Iraq or East Timor on sensitive missions appear to be fearful of a different enemy – federal parliament.

Hearings on Australia's military justice system will begin this week. The Defence Department has set up a special strike force – an army "Tiger Team" – to deal with the Senate hearing. It wants to take up to 20 people around the country as a rapid response unit. But there's a problem. It has indicated to the committee that it wants them to provide security, which prompted one committee member to respond: "You're meant to be protecting us, not the other way around."

Australia's defence and intelligence services are in crisis. While Prime Minister John Howard has resisted a royal commission, it is becoming increasingly clear that for the sake of a long-term cleansing of the system, the best thing he can do for the country is call one to allow a thorough, careful examination of why the system keeps going bad.

There are deep, systemic problems involving culture and accountability, but the most immediate problem is the determined 49-year-old Lieutenant Colonel Lance Collins. The biggest problem for the chief of the defence forces, General Peter Cosgrove, is that Collins is not known as a stirrer. His colleagues say he's one of the best they've worked with: loyal, smart, someone to trust in tight spots. And sprinkled throughout the defence and intelligence system, Collins has important allies.

At the top management level, however, he is distinctly unpopular. As The Bulletin revealed last week, Collins upset some in Canberra in July 1998 when he wrote an intelligence assessment (later proved accurate) warning that the Indonesian military (the TNI) were preparing to wreak havoc in East Timor during any proposed vote for independence from Indonesia.

Collins, who knew East Timor well, stated the TNI and militia were effectively the same brutal unit. While the Howard government, under pressure from Jakarta, was trying to argue that only "rogue elements" were the problem, Collins warned of the Indonesian military.

But the real damage came from his criticism of a "pro-Jakarta lobby" in the Defence Intelligence Organisation (DIO), which he argued prevented accurate assessments being sent to government. Despite Collins' unpopularity, when Howard sent 4500 Australian troops to East Timor to rebuild after the destruction by the TNI and militia, Cosgrove hand-picked Collins.

Success in Timor was Cosgrove's big chance. His nightly television appearances made him a household name, but it was Collins' daily assessments of where the troops should and should not go that ensured Interfet lost not a single soldier. Cosgrove became the TV star, Collins the anonymous intelligence chief.

On his return to Australia, Collins was told the "knives" were out and very soon he felt the biggest knife of all in his back. Someone in Canberra put his name on a Federal Police warrant, and it was quickly leaked to the media. For 25 years, Collins had been trusted with the most sensitive of Australia's (and the United States) defence secrets. He wanted an apology from his boss and former friend, Peter Cosgrove.

The report into the incident by naval barrister Captain Martin Toohey, published in full in The Bulletin last week, shows that soon after the leak, the people who signed Collins' name to the warrant knew he had been cleared by the Federal Police investigation. But four years later, no one in an official position has bothered to tell him the investigation is over and he has been cleared. In fact, three weeks ago, they downgraded his security clearance to stop him seeing the Toohey report.

Toohey, who has top-secret clearance and is trusted with extremely sensitive Defence investigations, concluded: "I find as a fact the Defence Security Branch (activated, on the balance of probabilities) by malice, at the material time failed to inform Lt Col Collins as soon as practicable after the execution of the AFP search warrant of the fact that he was not, and never had been, under investigation ... I find as a fact that the incident could have been prevented by the Assistant-Secretary – Defence Secretary Mr Jason Browne advising Lt Col Collins, in a timely manner, of the complainant's complete lack of involvement in the security investigation."

The Toohey report made another devastating finding which neither the PM nor the chief of the DIO, Frank Lewincamp, have addressed: that because of his battle with Collins, Lewincamp "caused the flow of intelligence to East Timor to be suspended for approximately 24 hours". Until Howard reveals why Australian soldiers were endangered in this way, the issue will not go away.

After The Bulletin's revelations of the Toohey report, the PM, senior ministers and defence bureaucrats spent 24 hours planning how to react.

But a backlash over John Howard's attack on Federal Police Commissioner Mick Keelty convinced the government not to attack Collins personally.

Instead, they decided to "shoot the messenger", Captain Toohey. After receiving Toohey's report on September 7 last year, those around Cosgrove went looking for a legal opinion with which to counter it, claiming Toohey had gone outside his terms of reference. They had two problems: the devastating nature of the report, and the fact it had already been signed off by Lt Col Tina Mathewson from army headquarters.

Nonetheless, they sent it to Colonel Roger Brown, a Cambridge-educated PhD in law and Sydney magistrate. There was more bad news: on September 22, Brown's report arrived: "Captain Toohey's inquiry was in accordance with his terms of reference ... It should be noted it is a vital element of both legal and intelligence work that advisers be free to tender their advice, whether popular or not, without fear of repercussions for failing 'to toe the Party line'. Captain Toohey's findings clearly demonstrate that Lt Col Collins was denied this freedom." Brown even raised the possibility of disciplinary action against a serving officer over his treatment of Collins.

For Cosgrove, it was going from bad to worse. Defence insiders say his inner circle was devastated. They needed another opinion – this time the Toohey report was dispatched to Colonel Richard Tracey, a Melbourne QC.

He found "there can be no doubt there have been shortcomings in his career management since his return from East Timor", but the Toohey report had "miscarried" as it had led to an investigation of "bodies external to the ADF and insofar as it has led to recommendations for action by you which you could not, lawfully, take".

Last week, Defence Minister Robert Hill released the Tracey report in an effort to discredit the Toohey report, but his media release did not mention the Brown report.

It was Tony Jones on the ABC's Lateline who, in a masterful interview, derailed the attack on the Toohey report. The government wanted to bury it, and Hill was appearing on the program to talk about the Tracey report.

He was wrong-footed when Jones asked about the Brown report. Hill was all over the place – the best he could do was say the Brown report was only "a process matter" while the Tracey report was "the detailed analysis". This was simply untrue.

Forty-eight hours later, in media dead-time (Friday night after the TV news and newspapers had gone to press), Hill's office released the Brown report.

But a bad week was about to get worse. An email from Colonel Gary Hogan, the army's liaison to the current Senate inquiry into military justice, was leaked. It revealed that the cover-up mentality that marked the treatment of Collins was flourishing.

Hogan – whom Cosgrove has appointed to help the Senate inquiry gain information from the army – was in effect coaching senior Defence people on how to get around the Senate inquiry. Hogan advised those on his email list to write "Internal Working Document" on documents. The danger for Cosgrove is whether he knew of Hogan's instruction, and whether this advice constitutes a contempt of the Senate.

The email said: "All inquiry-related correspondence should be headed 'Internal Working Document' in order that the correspondence be exempt from tabling before the Committee under the Freedom of Information Act."

For John Howard, the crisis is deepening. He has got it badly wrong by appointing a former ambassador to Jakarta, Philip Flood, to run the inquiry. Flood is a classic insider, and accepted practice is that a royal commission be run by someone who has not been a key player in the very system the inquiry is examining.

For the sake of the nation's physical security, appointing a royal commission is one of the most important decisions the PM can make.