In Paris, the head of Electricite de France, the biggest nuclear power generator in the world, said that "financial conditions" had not yet been met for acquisition of British Energy.
A source told Dow Jones Newswires that the two companies may decide to restart talks but added that EDF did not intend increasing its offer.
The BBC reported that major shareholders in British Energy, Invesco and Prudential, were holding out for a higher price owing to the surging cost of fuels.
Meanwhile Britain's Business Secretary John Hutton told the public broadcaster that the government was "disappointed" at the failure to reach a deal.
Britain's Labour government owns more than a third of British Energy but wants to sell its stake.
"We are disappointed by the failure to reach an agreement overnight," Hutton told BBC radio. "I think it would have been a good fit."
He added: "It's up to the board now and EDF to see if there is any way that this gap can be bridged."
British Energy said in a short statement to the London Stock Exchange that "advanced discussions with a party have continued but without agreement to date," leading to a sharp drop in its share price early Friday.
Speaking at a press conference in Paris, EDF chairman and chief executive Pierre Gadonneix said that talks had ended overnight without an agreement on a takeover by the group, in which the French state is the majority shareholder.
"Following in-depth discussions, and after hearing the demands of all sides, we deem that the financial conditions allowing a major development in Britain by EDF have not currently been met," he said.
Gadonneix added that, in light of a trend towards nuclear power in several markets, EDF -- which Friday declared a 12.2-percent drop in half-year profits to 3.08 billion euros -- intended to play a leading role.
EDF, as the world's biggest nuclear power supplier and already Europe's number one electricity producer, had been widely expected to confirm that a deal had been struck to buy British Energy, which has a market value of 12 billion pounds (15.5 billion euros, 24 billion dollars).
A source close to talks with British Energy had told AFP on Thursday that the French group's directors had backed a takeover.
The bid was said to have been in the form of either a cash-only deal worth 765 pence per share or a cash option worth 700 pence a share plus shares.
The offer was aimed at the British government's 35.2 percent stake in British Energy. Under takeover rules, any bid for the government stake would trigger an offer for the rest of group.
In European stock market trading on Friday morning, British Energy's share price was showing a fall of 4.18 percent to 699 pence on London's FTSE 100, which was down 0.31 percent.
EDF dropped by 0.89 percent to 55.45 euros on the Paris CAC 40, which was 0.41 percent lower.
British Energy provides almost one fifth of Britain's electricity and owns and operates eight nuclear power stations.
Last month, British Energy said that a series of takeover proposals received from unnamed parties had undervalued the company.
The government, meanwhile, wants to see a renewal of its nuclear-power generating capacity and the assets owned by the group are seen as a springboard for any such effort.
"The future of new nuclear in the UK doesn't depend on this particular (EDF) deal," Hutton added during his interview with the BBC.
"I think we are absolutely committed to new nuclear power and if this deal is not able to go through for whatever reason, we will be looking at plans to make sure we can continue with our foot on the floor because Britain needs these new nuclear power stations."
Britain's governing Labour Party called nuclear power an "unattractive" option as late as 2003, but has changed its mind as energy costs soar, initiatives against climate change intensify and North Sea oil and gas stocks dwindle.
It also follows an increasing global trend towards atomic power, which currently provides about 16 percent of the world's electricity.