Wall Street smarts – mathematicians and physicists try to predict market characteristics with varying degrees of success

Wall Street smarts – mathematicians and physicists try to predict market characteristics with varying degrees of success – includes related articles on electronic trading; using cowrie shells as cash; portability of money; devaluation of Thai currency; and breaking bank codes

Gary Taubes

IN 1946, WHEN MY FATHER WAS IN HIS FORMATIVE YEARS, HE purchased 500 shares of stock in a company then known as Haloid. As my father remembers it, the stock cost $9 a share, which meant my father was out $4,500, a significant sum at the time. He held the shares for two years while the stock climbed slowly to $16. In October 1948, Haloid announced the existence of a photocopying process that would later become known colloquially as Xeroxing after Haloid changed its name to Xerox. Contrary to all reasonable expectations (including my father’s), the announcement did not send the price of Haloid stock sky ward. Indeed, by January 1949 it had dropped to $14 a share and my father sold out. He made a tidy profit of $2,500, but had he held those 500 shares until, say, 1969, they would have been worth about $10 million.

My father’s failure to predict the behavior of the market, or even of a single stock, cannot be held against him, however I might feel about currently having to work for a living. Indeed, the question of whether the prices of stocks, bonds, and other financial commodities can ever be predicted is one of the great scientific controversies of our time. According to orthodox wisdom, financial markets behave randomly. For each given trade, a stock’s price goes up and down with no more predictability than a coin toss, and no amount of knowledge of the stock’s history, no amount of intuition, tips, or analysis, will tell you with any certainty what will happen today, next month, or next year. This paradigm goes by the name of the Efficient Market Hypothesis (EMH) or the Random Walk Hypothesis, a name derived from the suggestion that the price of a stock stumbles around a given value like a drunk around a lamppost.

The controversy rages because of the unavoidable implication of the hypothesis: if it’s true, then traders who make money speculating in stock movements–George Soros, for instance, or Warren Buffett–are neither brilliant nor intuitively gifted, but only lucky. Devotees of the hypothesis, such as Burton Malkiel, a Princeton economist who wrote the classic investing text A Random Walk Down Wall Street, like to point out that if the EMH holds, a blindfolded chimpanzee throwing darts at the stocks pages of the Wall Street Journal could pick stocks as well as any expert. Financiers who make their living on Wall Street, needless to say, have never been enamored of the theory.

While researchers have collected evidence supporting the hypothesis and its blindfolded chimps, there is also copious evidence to the contrary–in particular, everyone who has ever made a fortune in the market. What’s more, the last decade has seen a wave of physicists, mathematicians, and computer scientists migrating to Wall Street and apparently making large sums of money once there. These scientists have been at the forefront of a revolution in finance, applying powerful computers and sophisticated mathematical and statistical research to the stream of data pouring out of the world’s financial markets. And they have founded a handful of companies–Renaissance Technologies, for instance, run by mathematician Jim Simons, or D. E. Shaw & Company, run by computer scientist David Shaw–that are doing remarkably well placing wagers on the behavior of stocks, bonds, currencies, and other financial commodities.

But why? If the market is unpredictable, then these scientists have just been lucky. If they’re more than just lucky, then the market is indeed predictable. Choosing the correct scenario is difficult, partly because the players who are actually making money on Wall Street tend to be obsessively secretive about what they’re doing and how they’re doing it, while those who aren’t making money may not know anything valuable. And there’s another reason, which is the kicker, the heart of the controversy. Simons, who spent eight years heading the mathematics department at the State University of New York at Stony Brook, puts it best. “In this business,” he says, “it’s easy to confuse luck with brains.”

CREDIT DOESN’T always come to scientists while they’re young enough to enjoy it. In his doctoral thesis in 1900 at the University of Paris, “The Theory of Speculation,” an aspiring French mathematician named Louis Bachelier created much of the future of market theory. But neither the great Jules-Henri Poincare, Bachelier’s adviser, nor anyone else found the thesis compelling. Poincare called it “somewhat remote from those our other candidates are in the habit of treating.” Bachelier never managed to get a job in mathematics, and his thesis was ignored for half a century. So it goes.

The experience of Andrew Lo, a professor of finance who directs MIT’s Laboratory for Financial Engineering, shows how far the theory of speculation has come. In 1986, when Lo was a precocious 26-year-old assistant professor at the Wharton School of Finance in Philadelphia, he and Ills Wharton colleague Craig MacKinlay were among the first to provide demonstrable evidence of Wall Street’s predictability. As Lo explains it, the best way to think about a random walk is as a kind of gambling game where the odds are exactly 50-50. If the market is fair–that is, if none of the speculators has an unfair advantage–then the best prediction for how much money you’re likely to make is zero. “If you start out with a pile of money, you bet each time, and you have 50-50 odds of winning,” Lo says, “then your wealth today is an excellent forecast of your wealth tomorrow, because you can’t predict whether you’re going to win or lose.”

This idea evolved into the Efficient Market Hypothesis in the 1960s and 1970s, with the help of MIT economist Paul Samuelson, who won the Nobel Prize in 1970, and University of Chicago economist Eugene Fama. They pointed out that if countless traders are competing to buy and sell stocks at what they think, based on rational considerations, are their proper values, then the market will always instantaneously adjust the price of each stock to what it’s worth. Moreover, any information relevant to the real value of the stock–news, or insider information, or anything else–will also be instantaneously absorbed into the price. When the market moves in response to any news, so the theory goes, it will move so quickly that no one will be able to make money reliably.

To Lo, the EMH has a “Zen-like”, beauty. “What it says is that the more people who try to forecast markets, the less forecastable markets tend to be,” he explains. “If there is any kind of fore-castability in prices, somebody is going to take advantage of it, so if there are enough people competing with each other to try to take advantage of small forecastabilities, then all of those forecastabilities will be dissipated.”

After the efficient market does its work, according to theory, what’s left is the random motion–the drunkard’s walk–of the day-to-day or month-to-month fluctuations. Whether these fluctuations are truly random and unpredictable, of course, is the kind of question that depends on whether you buy the hypothesis. Those who don’t buy it usually point out that some people certainly have made money in the markets, and the in: vestment profession is stocked with managers, analysts, newsletter publishers, and gadflies who recommend stocks to buy or sell.

Yet despite decades of effort, academic researchers have been unable to demonstrate that these expert prognosticators do better than the blindfolded chimps. When the market is going up, as it has for the past decade, the values of the stock portfolios recommended by these experts go up–although, on average, perhaps not as fast. When the market goes down, the portfolios go down. In fact, studies of how well investment managers do from decade to decade show that, with a few rare exceptions, those who do very well in one decade often bomb in others. The possibility that success is all luck is one reason the Securities and Exchange Commission requires mutual fund prospectuses to carry a disclaimer warning investors that past performance is no guarantee of future results.

Fortunately, the Efficient Market Hypothesis comes with the kind of paradox that offers scientists an opportunity to find a new truth. It works like this: If the market is truly efficient, then any information relating to a stock’s value is instantaneously absorbed into its price. That, however, leaves no incentive for anyone to learn new information bemuse they can’t possibly profit from it. And if no one is out there learning it, then how does the information make it into the market to be instantaneously absorbed?

“Who the hell has any motive to learn anything?” asks Lo, somewhat rhetorically. “If the markets are always informationally efficient, then nobody is trying to gather information, because they can’t make money off it. Clearly that’s an absurd premise. So no new information is getting into the market and that’s the paradox. The only way you can get out of the paradox is if the markets are not informationally efficient all the time and it takes a lot of effort to make them efficient. You get paid for that effort.”

In a 1988 paper, Lo and MacKinlay took weekly stock market data from 1962 through 1985 and demonstrated statistically that the movement of prices was not random. Stock prices of small firms would lag behind those of larger firms in the same business. For those 23 years at least, the market exhibited predictable behavior. Any trader with the right insight and sufficiently sophisticated computers (which didn’t exist at the time) could have detected this lag, or inefficiency. He could then have used the movement of the larger firms’ stocks to bet on the movement of stocks from the smaller firms. Such a trader, says MacKinlay, “would have ended up very wealthy.”

This violation of the Random Walk Hypothesis seemed so sacrilegious that when Lo first presented the results publicly, an eminent economist–Lo would rather not disclose his name as they’re on good terms now–suggested that the two Wharton researchers must have made a programming error. The attack, says Lo, left little room for rational discussion. “We said, `No, we didn’t make an error.’ He said, `Yes, you did.’ It then degenerated into a shouting match.” But afterward, enough researchers replicated Lo and MacKinlay’s results to rule out error.

Curiously, the predictability Lo and MacKinlay spotted a decade ago has nearly vanished. This suggests two possibilities: either it wasn’t real to begin with or the market has since adjusted. “In the early nineties,” MacKinlay says, “the pattern was not nearly as strong. That may be expected. As patterns become known, people trying to exploit them will make them go away. In some sense they will be competed away.”

Since their original work, Lo and MacKinlay have spent much of their professional life studying violations of the market’s random walk. In effect, says Lo, the market is efficient, but not instantaneously so. Small inefficiencies can exist long enough for traders to take advantage of them, at which point they will vanish.

“There are in fact people who make a good living coming up with new ideas, good ideas, and executing them in a profitable manner,” says Lo. “Nine out often times, the guy who says he can beat the market is probably wrong because he doesn’t have anything new. All he’s observed is a statistical fluctuation, pure luck. But every once in a while you find somebody that has come up with a genuinely new idea for either forecasting the market or assessing information better, or being able to put all the pieces together in a more cost-efficient manner. He will make money.”

Because every new technology also makes the market more efficient, Lo explains, “traders have to continuously adapt to stay ahead.” They do it by developing or enlisting new technologies, most of which come from various fields of scientific research: financial engineering, mathematics, statistics, computer science. Only recently, Lo adds, “has anyone had the vision of applying very, very quantitative and computationally intensive techniques to financial markets.”

DAVID SHAW AND Jim Simons are among the pioneers of this particular vision. Both left academic research for Wall Street because they were intrigued by the challenge of beating the market. When Shaw took a job with the investment banking firm Morgan Stanley in 1986, he already had a Ph.D. from Stanford in computer science and was designing massively parallel supercomputers as an associate professor at Columbia. Morgan Stanley offered him a salary six times what he was making at Columbia. The firm also offered him an equally compelling intellectual challenge. “It seemed like what they were doing,” says Shaw, “was clearly impossible.” Until then Shaw had been a believer in the Efficient Market Hypothesis–a belief he inherited from his stepfather, a finance professor at UCLA. But now Morgan Stanley seemed to be using sophisticated mathematics to systematically beat the market.

Shaw, however, had something to offer besides his intellect and ambition. He was hired, he says, to bring Morgan Stanley the kind of technology and environment needed to do serious research. At the time, there was a tremendous lag between the computational abilities of Wall Street and those of academia. (Now, a dozen years later, the lag has inverted. Wall Street firms have the computing muscle, while the academic labs, in funding slumps, struggle to keep up with the newest technology.) Shaw’s secondary task at Morgan Stanley was to explore new computational techniques for finding profitable patterns and inefficiencies in the market beyond those the firm had already discovered.

After two years Shaw decided to go it on his own. He raised $28 million and founded D. E. Shaw & Company to specialize in what is now known as quantitative trading: using sophisticated computing and mathematics to analyze the market and make trades. Today D. E. Shaw has more than a thousand employees and is considered the most technologically advanced trading company on Wall Street. On a “big day,” says Shaw, his firm alone may be responsible for more than 5 percent of the trading volume on the New York Stock Exchange.

Simons has a slightly different story. With a background in pure mathematics, he believed his expertise could be useful in predicting market behavior. Simons, who is now 60, obtained his doctorate in mathematics from Berkeley and taught at MIT and Harvard before spending four years as a code breaker at the Department of Defense during the Vietnam years. (He was fired for taking a public stand against the war.) He was then hired to run the math department at Stony Brook. After eight years he left to found Renaissance Technologies.

Simons went to Wall Street because he liked the idea of making money and because he wanted to be “a little more free.” He knew he would leave academia from the time he got his first faculty job. He remembers sitting in the MIT library at age 23, thinking, “Well, now I’m on the faculty. Let’s see what happens here: I’m an instructor, I become an assistant professor, then an associate professor, then a full professor, then professor emeritus, and then I die. Everything was mapped out, and it felt too constraining to me.”

As early as 1961, Simons had started investing in companies run by his friends. After cashing out of one such endeavor in 1973, he recruited a friend who was trading commodities to invest the proceeds for him. The result was his first fortune–“at least it seemed to me a for tune”–and Ills first exposure to the luck-brains paradox on Wall Street. In eight months of trading, his friend increased his investment tenfold. Simons considered him “the smartest guy in the world.” Years later he realized his friend had been outrageously lucky: he could just as easily have lost everything if the dice had rolled a different way. “Thank goodness we stopped. He never again made money like that.”

Simons took his fortune and founded Renaissance Technologies in 1977 with the idea of using mathematics to create models of currency trading that could tell when and what to buy and sell. The firm now has some 90 employees, 40 of whom are Ph.D.’s. According to Financial World, one fund run by Simons, known as the Medallion offshore futures fund, averaged a 44 percent return on the dollar through the first half of this decade. This means that if you had invested $10,000 with Simons in January 1990, it would have been worth $95,000 by the end of 1995. Simons is now the personal benefactor of mathematics institutes from MIT to the University of California at Berkeley. Mathematicians are quite fond of him.

Although Simons preceded Shaw into finance by a decade, in effect they play the same game: both figure out how best to mathematically model the behavior of financial markets so that a computer can predict the direction of stocks, currencies, or other financial commodities at least long enough for them to bet on the knowledge. Minus the financial incentive, this is the same challenge confronting, for example, climate researchers who want to create accurate computer models of Earth’s atmosphere, oceans, and landmasses to predict what will hap pen to the global temperature when they add, say, a little extra carbon dioxide to the atmospheric mix. Both cases require the computers to process enormous amounts of real-life data, which is why researchers refer to them as computationally intensive. And both cases require the researchers to create equations that realistically capture the relationship between the variables of interest.

The catch in both is that while you can create models that might mimic everything that’s happened in the real world until now, you can never know for certain that the future will play along. Your predictions and reality might diverge in utterly unexpected directions. This is why climate modelers try to suppress their personal pride when their prediction of a bad El Nino year actually comes true (as it did this year). While finding repeatable patterns that fail to repeat predictably could be embarrassing in the climate-modeling business, it could be financially disastrous on Wall Street.

This is the problem of statistical inference, and it’s at the heart of the Wall Street luck-brains question. In the hard sciences, such as physics, the key to knowing whether a phenomenon is real is to replicate, more than once and in a variety of ways, the experiment that produced the observation. Replication minimizes the likelihood that you simply made the same mistake twice. On Wall Street–or in climate modeling, for that matter–such replication is impossible. “You have only one realization of history,” says Lo. “And from that realization you have to infer what’s going on with the underlying phenomena. Since you have only one observation, there is a fairly large margin of error, and so you have to think more carefully about the probabilities and statistics involved in what you’re trying to determine.”

Simons says that the advantage scientists bring to the game is less their mathematical or computational skills than their ability to think scientifically. They are less likely to accept an apparent winning strategy that might be a mere statistical fluke. The math involved is “not terrifically deep, a lot of linear algebra, a lot of statistics,” so the salient talent is how well his researchers use that math to differentiate between a real predictable pattern and a chance fluctuation. Dome level of mathematical sophistication is required,” Simons says, “but also sort of a love–a fascination with trying to know how things work. It’s not really mathematical–I don’t know what you’d call it. Anyway, it’s science.”

To Shaw, the key is not just to create models of how financial markets work but to test them rigorously. “What we’re applying is essentially the scientific method,” he says. “We don’t just take a bunch of data and grind through it and find patterns. Rather, we formulate hypotheses and build up models of a lot of mechanisms that we believe are operating out there. And then we validate those models in the same way any scientist would. We will do experiments of various sorts looking at historical data, and in some cases we actually have to move large amounts of money through the system and see what happens. In a given experiment we might pump billions of dollars through the market.” While they hope to make some money on these experiments, Shaw adds, they also budget for significant losses–perhaps millions of dollars.

Neither Shaw nor Simons will talk about the kinds of inefficiencies they find to trade on or the details of their models, because those are what make them money. Indeed, Shaw won’t even tell what’s not useful, because he doesn’t want to give his competitors an edge by allowing them to skip blind alleys. (Simons simply shuns publicity. The quotations for this story came from a taped interview Simons granted two mathematicians at Berkeley’s Mathematical Sciences Research Institute–Hugo Rossi and David Eisenbud.)

All Shaw will say about what doesn’t work are “the most obvious things” that might come to a mathematician’s mind if he or she is up on the literature. “If someone has a quantitative strategy that they believe is capable of beating the market,” he says, “there’s a good chance that they’ll eventually come to us to see if we’d be willing to provide the capital necessary to trade on it. Almost always, though, we find that they’ve fallen into one of a standard set of pitfalls that we’ve learned to recognize very quickly.”

Among the less obvious approaches that Shaw will discuss is chaos theory, which began receiving unrestrained press in the 1980s as the new way to beat the market. Chaos theory is the study of apparently random phenomena that derive from surprisingly simple relationships, which in mm can be modeled with relatively simple equations. Wall Street’s behavior looked like the kind of apparently random behavior that might really be chaotic, yielding up its underlying patterns to mathematics. The reason Shaw is willing to discuss his belief that chaos theory hasn’t helped traders make money from the market may be that most economists and financial researchers who have studied the problem share his view.

BACK IN THE EIGHTIES, researchers hoped that the market would be shown to demonstrate “low dimensional” chaos, which means that the market’s behavior could be described by just a few mathematical equations showing the relationships between key variables. They learned, however, that the financial market is described not by 2 or 3 equations but by maybe 50, “which is as good as random,” as Lo puts it. Even if fewer equations had been necessary, says Shaw, that would not mean anyone could make money off chaos. Even when chaotic systems are simple–that is, low dimensional–their behavior can still be awesomely unpredictable. “Over a fairly short time frame,” explains Shaw, “very small differences in the starting conditions, or in anything that interferes with the system over time, can cause enormous changes in the ultimate outcome. So even though you can detect evidence of nonrandomness in the market, that doesn’t mean you can actually use it to predict anything.”

The other thing Shaw will say (which may be a ploy to throw his competitors off, although it seems unlikely) is that this business of quantitative trading is not the future of finance. It is not, he says, “a growth industry.” This is the result of efficient market theory exerting itself. If everybody gets into the game, the game will vanish. In fact, both Shaw and Simons see a steady degradation just from their own actions. “The system is always leaking,” says Simons, which is why they are constantly recruiting the brightest young people to turn up undiscovered inefficiencies.

When Shaw started in the business ten years ago, he says, he could find one or two inefficiencies in the market and make money by trading on them. This is no longer possible. It’s as though the veins of the strategy have been mined until what’s left costs almost more to extract than it’s worth. “It’s not just the commissions that we pay,” he says, “because we own our own broker-dealer firms and so our trading costs are extremely low.” But there’s a hidden cost of trading known as “market impact” or “slippage,” which means that just the act of buying a stock or currency raises the price slightly. If the expected profit depends on the price being a smidgen lower, then the profit will vanish as soon as the investor tries to trade on it. The only way one can make money now, he says, is by finding 15 to 20 tiny inefficiencies and trading them in a combined strategy that makes slightly more money than trading them separately. “It is extremely difficult to find these things anymore,” he says.

This difficulty has an upside for Shaw and Simons, because mining market inefficiencies is now so difficult that if anyone else wanted to get into the business he’d probably have to spend several hundred million dollars getting the computers, the data, and the understanding to capitalize on them. “If we were to start now,” says Shaw, “we would hemorrhage money before we did enough research to break even.”

When Shaw talks about the future of his business, he begins to sound less like a computer scientist and more like a Wall Street sage. The best way to make money, he believes, is still the old-fashioned way. Even among the players who try to cash in on inefficiencies and those who try to find financial commodities that are undervalued or overvalued, says Shaw, the most important factor is not describing the behavior of the market in mathematical terms. “It’s the people who can analyze companies,” he says, “who understand what products the companies make: whether the chief executive officer seems to be a good chief executive officer; whether their business plan makes sense; whether it’s a growth industry or a shrinking industry; and so forth. Those are the kinds of things we do not believe are amenable to mathematical or computational analysis. And we also believe that they are actually far more important and should occupy a much larger fraction of the total number of people in the business.” He is not beyond imagining that someday a truly intelligent computer may help with these jobs, but now, he says, the grunt work of making money in the market is still “exclusively within the province of human thought, and we don’t have any pretensions about our ability to change that.”

Along these lines, Simons says that his first move when he started Renaissance Technologies was to hire Leonard Baum, a mathematician and former colleague from his Pentagon days. Simons considered Baum the best code breaker and maybe the best mathematical modeler in the country. As Simons tells it, “He could see what was going on under the pattern of all these O’s and l’s.” So Simons put Baum to work modeling the currency markets, which he did for three months. Then Baum said he was fired of doing research and strongly suggested they buy British currency. “Margaret Thatcher has been holding it down, and there is no way she can hold it much longer,” Baum said. “It’s going to go through the roof–you better buy it.” So Simons bought pounds and made a killing. “From that day on, Lennie stopped modeling,” Simons says. “He felt he could predict these markets better than any computer model we’re ever going to build. There was no way I could persuade him differently, but that was fine: he stayed with us, and in the subsequent years we made a lot of money.”


Swap Meet

The ease of electronic trading encourages stock swaps, and on-line traders are said to make deals three to four times more often than traders not online. But that may not be good for the bank account. According to a five-year study of 60,000 households by economists at the University of California at Davis, the most active traders earned returns of 10 percent; the least active, returns of 15 percent.


Slavers and Shells

Until the 1880s, cowrie shells were one of the world’s most abundant currencies. These small shells were harvested from the Maldives and traded around the world–from Indian traders to European merchants, who exchanged them for African slaves. After Britain and the United States banned the legal slave trade in 1807, cowries were used to trade for palm oil. In the mid-1800s, several European companies shipped some 35,000 tons of shells to West Africa. The flood of “cash” swamped the market, prices crashed, and the Cowrie trade from the East ended. Merchants began using cowries harvested from the waters of Zanzibar, but by the turn of the century trade in cowries was finished.


Size Matters

What makes money useful is its portability–or so dictates conventional European economics. Yet on Yap, in the Caroline Islands of the west Pacific, natives had a less convenient currency–large, round slices of limestone 12 feet wide, cut from rock on islands some 400 miles away. This was flashy, status currency, used only by men in ritual affairs. Women used small strings of mussel shells.


Thai Drop

The devaluation of the Thai currency in July 1997 devastated Asian economies–perhaps none more so than Indonesia’s. In 1997, 11 percent of Indonesians lived in poverty. In 1998, that percentage may climb to 48, says Indonesia’s chief of central statistics.


Code Breaking

In a privately sponsored code-cracking contest, hackers extraordinaire John Gilmore and Paul Kocher broke the nation’s Data Encryption Standard (DES) in just 56 hours. The team’s supercomputer, built for less than $250,000 and carrying chips designed to test millions of keys a second, beat competitors who used processing power distributed among several thousand computers. DES is used by banks and other financial institutions and encodes information using 56-bit keys. The message decoded by Gilmore and Kocher read, “It’s time for those 128-, 192-, and 256-bit keys”–which are more difficult to crack.

COPYRIGHT 1998 Discover

COPYRIGHT 2000 Gale Group