More than 350 years ago, a French mathematician wrote a deceptively simple theorem in the margins of a book, adding that he had discovered a marvelous proof of it but lacked space to include it in the margin. He died without ever offering his proof, and mathematicians have been trying ever since to supply it.

Now, after thousands of claims of success that proved untrue, mathematicians say the daunting challenge, perhaps the most famous of unsolved mathematical problems, has at last been surmounted.

The problem is Fermat's last theorem, and its apparent conqueror is Dr. Andrew Wiles, a 40-year-old English mathematician who works at Princeton University. Dr. Wiles announced the result yesterday at the last of three lectures given over three days at Cambridge University in England.

Within a few minutes of the conclusion of his final lecture, computer mail messages were winging around the world as mathematicians alerted each other to the startling and almost wholly unexpected result.

Dr. Leonard Adelman of the University of Southern California said he received a message about an hour after Dr. Wiles's announcement. The frenzy is justified, he said. "It's the most exciting thing that's happened in -- geez -- maybe ever, in mathematics." Impossible Is Possible

Mathematicians present at the lecture said they felt "an elation," said Dr. Kenneth Ribet of the University of California at Berkeley, in a telephone interview from Cambridge.

The theorem, an overarching statement about what solutions are possible for certain simple equations, was stated in 1637 by Pierre de Fermat, a 17th- century French mathematician and physicist. Many of the brightest minds in mathematics have struggled to find the proof ever since, and many have concluded that Fermat, contrary to his tantalizing claim, had probably failed to develop one despite his considerable mathematical ability.

With Dr. Wiles' result, Dr. Ribet said, "the mathematical landscape has changed." He explained: "You discover that things that seemed completely impossible are more of a reality. This changes the way you approach problems, what you think is possible."

Dr. Barry Mazur, a Harvard University mathematician, also reached by telephone in Cambridge, said: "A lot more is proved than Fermat's last theorem. One could envision a proof of a problem, no matter how celebrated, that had no implications. But this is just the reverse. This is the emergence of a technique that is visibly powerful. It's going to prove a lot more." Remember Pythagoras?

Fermat's last theorem has to do with equations of the form xn + yn = zn . The case where n is 2 is familiar as the Pythagorean theorem, which says that the squares of the lengths of two sides of a right-angled triangle equal the square of the length of the hypotenuse. One such equation is 32 + 42= 52 , since 9 + 16 = 25.

Fermat's last theorem states that there are no solutions to such equations when n is a whole number greater than 2. This means, for instance, that it would be impossible to find any whole numbers x, y and z such that x3 + y3 = z3 . Thus 33 + 43 (27 + 64) = 91, which is not the cube of any whole number.

Mathematicians in the United States said that the stature of Dr. Wiles and the imprimatur of the experts who heard his lectures, especially Dr. Ribet and Dr. Mazur, convinced them that the new proof was very likely to be right. In addition, they said, the logic of the proof is persuasive because it is is built on a carefully developed edifice of mathematics that goes back more than 30 years and is widely accepted.

Experts cautioned that Dr. Wiles could, of course, have made some subtle misstep. Dr. Harold M. Edwards, a mathematician at the Courant Institute of Mathematical Sciences in New York, said that until the proof was published in a mathematical journal, which could take a year, and until it is checked many times, there is always a chance it is wrong. The author of a book on Fermat's last theorem, Dr. Edwards noted that "even good mathematicians have had false proofs." Luring the World's 'Cranks'

But even he said that Dr. Wiles's proof sounded like the real thing and "has to be taken very seriously."

Despite the apparent simplicity of the theorem, proving it was so hard that in 1815 and in again 1860, the French Academy of Sciences offered a gold medal and 300 francs to anyone who could solve it. In 1908, the German Academy of Sciences offered a prize of 100,000 marks for a proof that the theorem was correct. The prize, which still stands but has been reduced to 7,500 marks, about $4,400, has attracted the world's "cranks," Dr. Edwards said. When the Germans said the proof had to be published, "the cranks began publishing their solutions in the vanity press," he said, yielding thousands of booklets. The Germans told him they would even award the prize for a proof that the theorem was not true, Dr. Edwards added, saying that they "would be so overjoyed that they wouldn't have to read through these submissions."

But it was not just amateurs whose imagination was captured by the enigma. Famous mathematicians, too, spent years on it. Others avoided it for fear of being sucked into a quagmire. One mathematical genius, David Hilbert, said in 1920 that he would not work on it because "before beginning I should put in three years of intensive study, and I haven't that much time to spend on a probable failure."

Mathematicians armed with computers have shown that Fermat's theorem holds true up to very high numbers. But that falls well short of a general proof. Tortuous Path to Proof