The little boy is now receiving treatment at a nursing home in New Delhi, the grandfather explained.
"We don't want to keep our baby at Max hospital. We don't trust the doctors there," he said. "They betrayed us."
It is not known if the boy will survive.
In a statement, Max Hospital said the child had been "handed over without any sign of life" to the parents.
"We are shaken and concerned at this rare incident. We have initiated a detailed inquiry, pending which, the concerned doctor has been asked to proceed on leave immediately. We are in constant touch with the parents and are providing all the support needed," the statement said.
Delhi Chief Minister Arvind Kejriwa confirmed on Twitter that an investigation is underway while Delhi police are also looking into the incident.