A group of visiting Indonesian MPs say they are not concerned that Papuan refugees recently granted asylum in Australia don't want to meet them.

The 42 refugees had been frightened by a request to meet the Indonesian parliamentary delegation accompanying Indonesia's ambassador to Australia, Hamzah Thayeb, and had refused, their lawyer said on Thursday.

Mr Thayeb returned to Australia last weekend after being recalled to Indonesia in protest against the Australian government's decision to grant protection visas to the refugees.

Delegation head Muhammad Hikam said the MPs were not concerned by the refusal, as a meeting with the refugees was not the focus of their visit. However, he revealed members of the delegation met on Thursday with representatives of the refugees.

"No (it's not a concern) because we are not actually (coming here with) the intention to meet them," Dr Hikam told AAP after a seminar in Melbourne on the state of relations between Australia and Indonesia.

"We welcome (it) if they want to see us, if they want to express their aspirations, something like that. But our intention here is not limited to that case, but to broader aspects. So it's up to them – we don't really have a special agenda."

David Manne, from the Refugee and Immigration Legal Centre, earlier on Thursday said an intermediary had approached him requesting a meeting between the Indonesian parliamentary delegation and the refugees, who are now living in Melbourne.

Mr Manne said the refugees were frightened by the request and had sent a statement to both Foreign Minister Alexander Downer and Immigration Minister Amanda Vanstone saying they did not want to meet the MPs.

"They're shocked and scared by the suggestion of meeting with Indonesian officials," Mr Manne said.

"They've just been found recently by Australian immigration officials to have a well-founded fear of being persecuted by Indonesian authorities if returned. They are really wanting to be left alone to live a life in the Australian community in safety and security without approaches from the Indonesian officials."

Dr Hikam earlier on Thursday said it would be impolite of the Papuans to refuse to meet him and his fellow Indon politicians."Usually, in Indonesia if you got an invitation like that, you cannot explain why (you refuse it)... you reject a good friendship," he told ABC radio.