Edgar Fritz Raupach was seized in the city of Kano.
In March, a British and an Italian hostage were also killed during a similar attempt to free them by British and Nigerian forces.
The latest developments came as the authorities said an Italian engineer had been kidnapped western Nigeria.
Kidnappings are uncommon in the western state of Kwara. But oil workers are often targeted for ransom in the south, and Islamist militants are active in northern Nigeria.
BBC Nigeria correspondent Will Ross says that worryingly the Nigerian security forces do not seem able to prevent the abductions and have once again shown the danger of trying to intervene to rescue the hostages.
Details about the circumstances of Thursday’s raid are still sketchy – police in Kano are due to brief the media later.
“The German abducted in January has been killed by his abductors early this morning,” a military official told the AFP news agency on condition of anonymity.
“Following intelligence reports, men of [a military task force] raided a hideout where he was being held by his abductors.”
Residents in the west of the city say they heard gunfire during the morning.
A medical doctor has said he has seen the body, our correspondent reports.
A crisis team set up by Germany’s foreign ministry is trying to find out further information, German news website Die Welt reports.
It is not clear who was behind Mr Raupach’s abduction or if he was ever moved out of the country, but in March a video purported to be from al-Qaeda’s north African wing demanded that Germany free a woman jailed on terror charges in return for his release.
It was obtained by a private Mauritanian news agency, ANI, which frequently publishes statements by al-Qaeda in the Islamic Maghreb, which has previously kidnapped Europeans elsewhere in West Africa.
The Italian man abducted in Kwara reportedly works for Borini Prono, an Italian building and civil engineering firm that focuses mainly on roads.
A government spokesman in Kwara state confirmed to the BBC that the kidnapping had occurred and said the engineer was at a work site when he was seized on Monday.
An Italian foreign ministry spokeswoman said Italian officials were in contact with the Nigerian authorities and the man’s family.
“We can confirm an Italian engineer has been kidnapped. We don’t yet know why or by whom,” she told AFP.
A year ago, British national Chris McManus and Italian Franco Lamolinara were seized by gunmen in the north-western town of Birnin Kebbi.
Ten months later they were killed in a failed rescue attempt in the northern city of Sokoto, where officials said a splinter group of the Boko Haram Islamists were holding them.
Boko Haram denied any involvement.
News Source:BBC News