Is U.S. innovation on the skids?

It would be hard to exaggerate the angst that has gripped the U.S. in recent weeks as markets have continued to churn and assets have melted. But the headlines that have made us dread picking up the newspaper mask a long-term problem that may shape the nation’s future even more than Uncle Sam’s unprecedented efforts to rescue the economy.

By most measures, the U.S. has been in a decade-long decline in global technological competitiveness. The reasons are many and complex, but central among them is the country’s retreat from long-term basic research in science and technology, coupled with a surge in R&D in countries such as China.

[ Learn more about how the financial crisis is affecting IT and the high-tech industry, plus what IT can do to help, in InfoWorld’s special report. And for the latest in government IT news and issues subscribe to InfoWorld’s Government IT newsletter. ]

R&D has two components, of course, and published figures showing a rise in “research and development” hide a troubling trend. Companies still spend billions annually on development, typically aimed at the next product cycle or two. But the kind of pure research that led to the invention of the transistor and the Internet is declining as companies bow to the pressure to improve quarterly and annual financial results.

To take but one example, Bell Labs , which was founded as AT&T Bell Laboratories in 1925, helped “weave the technological fabric of modern society,” as its Web site claims. Its “top 10 innovations,” according to parent company Alcatel-Lucent, include the transistor, data networking, cellular telephony, digital switching, communications satellites and Unix. Although Bell Labs continues to innovate in most of those areas, all of the top 10 had their origins in the 1970s or earlier.

In January 1982, Time magazine reported, “With 22,500 people on its payroll (3,000 of them Ph.D.s), 19,000 patents and an annual budget of $1.6 billion, Bell Laboratories is a mighty engine of research and development. It is possibly the finest, and certainly the largest, private operation of its kind anywhere.”

But beginning with its reorganization during the 1984 breakup of AT&T, Bell Labs has become steadily more focused on advanced development rather than pure research. On Sept. 4, New Jersey’s Star-Ledger newspaper reported that Bell Labs was disbanding a group of scientists doing basic research in areas such as material science and device physics. The paper reported that Research Director Gee Rittenhouse had said that “the team was going to have a hard time integrating its research into product development.”

Washington Watch

The change in focus from long-term research to shorter-term development in the private sector has been mirrored in the government. While federal funding for “R&D” has not declined overall and has in fact increased since the early 1990s, it has been more and more focused on the short-term needs of government.

In particular, critics say, under the administration of George W. Bush, the Defense Advanced Research Projects Agency — which gave birth to the Internet, computer time-sharing, computer graphics, LANs, and much more — has concentrated its research on short-term needs for warfare and homeland security. Today, DARPA funding tends to go to those who can promise measurable results in a year or two.

DARPA funding is now “short-term, applications-oriented, highly competitive, with small amounts of money and lots of reporting requirements,” says Leonard Kleinrock, a professor of computer science at the University of California, Los Angeles, and an Internet pioneer in the 1960s. “That does not engender quality research.”

In a recent bulletin to its members about the federal budget, the American Association for the Advancement of Science (AAAS) said, “Although high-priority investments in physical sciences research, weapons development and human space exploration help to keep the federal R&D outlook brighter than the bleak outlook for domestic programs overall, the FY 2009 budget continues the recent trends of declining federal support for research.”

The AAAS said that the federal investment in basic and applied research would fall in real terms for the fifth year in a row under the FY 2009 budget proposal. Meanwhile, it said, other countries, including China and Korea, are boosting government research spending by 10 percent or more annually.

The AAAS also presented data that shows that despite a big surge in health research funding for the National Institutes of Health between 1998 and 2003, total federal R&D spending as a percentage of gross domestic product has been in decline since 1976. “Federal research investments are shrinking as a share of the U.S. economy just as other nations are increasing their investments,” the AAAS observed.

The Technology Policy and Assessment Center at the Georgia Institute of Technology recently completed a study that compares the technological progress of 33 countries between 1993 and 2007. It concluded that China has progressed more, and more rapidly, than the other 32 countries, while the U.S. and Japan have slowly declined.

“The pattern is inexorable,” says Alan Porter, one of the authors of the study. “China is coming up strongly, and it’s in high-tech areas, not just cheap consumer goods.” China’s rise is aided by an authoritarian government, low wages and a good manufacturing base, he says, but that isn’t all. “You see tremendous effort in research in China,” Porter says. “The U.S. and China are neck and neck in basic science.”

“We have kind of lost our way in some respects,” says Vinton Cerf , chief Internet evangelist at Google and another Internet pioneer. “We have a significant diminution of industrial long-term research in IT, and we have seen one of the major federal sources of IT research — DARPA — essentially withdraw from a lot of that.”

Cerf says he doesn’t like the word competitiveness because it suggests an adversarial relationship. He’d prefer that scientists and engineers work across borders to collaborate openly and publish their results.

Cerf suggests that the new administration should encourage immigration by the most talented science and engineering students. “They are the creme de la creme, because they can’t get in otherwise,” he says. “Maybe they [ultimately] go home, and maybe they stay, but they contribute mightily to the health of research and add a great deal of value to U.S. research initiatives.”

Henry Chesbrough shares that goal. “We are losing our ability to attract the best and brightest at the graduate level to come to the U.S.,” says the executive director of the Center for Open Innovation at the University of California, Berkeley. “There are two reasons,” he explains. “Our concerns about security and immigration have caused us to be perceived as less welcoming. And the options back in the home countries are better than they have ever been. So at precisely the time we need to be more competitive to attract and keep these people, we are pushing them away.”

But attracting bright Ph.D. students is just one challenge; funding their work is another. According to the AAAS, total federal funding for R&D at universities has risen slightly recently, but, adjusted for inflation, that amount has declined in each of the past two years.

Kleinrock says he is troubled by how campus researchers are forced to take a pragmatic view to win short-term funding from DARPA. “They don’t stop to ask what’s behind the results they get,” he says. “They are not being pushed to get a fundamental understanding; they are looking for the answers now, for this system, for today.”

He says he worries that this short-term view of science will propagate from professor to student in a way that weakens subsequent generations of researchers.

Of course, given the current economic turmoil, most everyone sees other critical needs pushing R&D even further down the list of federal priorities. “Yes,” Chesbrough admits, “that’s going to make this deferred gratification even more difficult to accomplish.”

This is a print version of a Computerworld story that originally ran online. Computerworld is an InfoWorld affiliate.