Bowe Bergdahl’s hometown of Hailey, Idaho will not be holding a lavish homecoming celebration planned for the soldier’s return.

NPR reports the town is canceling the celebration after reports of Bergdahl deserting his fellow soldiers in Afghanistan.

The town has held the Bring Back Bowe Rally for the last several years, attracting crowds of more than 3000 people. Organizers of the rally had originally planned to turn this year’s ceremony into a welcome home celebration, but decided to cancel it because of the possibility of protests involving the circumstances of his return.

Hailey’s Chamber of Commerce president, Jane Drussel, told the AP her group has received hate mail and numerous phone calls about the rally.

“The joy has all of a sudden become not so joyful,” Drussel told NPR.

Bergdahl was released by the Taliban Saturday after five years of captivity in a prisoner exchange for five senior Taliban members. Since his release, controversy has surrounded the soldier’s return, including the circumstances around his disappearance and those angry the Obama administration did not give Congress 30 days notice of the exchange per law.

Tom Bowman of NPR said the military conducted an “exhaustive” review of the Bergdahl case. While the contents of the review remain classified, Bowman said he knows of its contents through people who have read it.

The report says Bergdahl grew disillusioned with the military and left on his own volition, but left the question of desertion open as they are unsure of whether he intended to return or not. Bowman was told by military officials Bergdahl left his post two times prior as well, once in California and another time in Afghanistan.