Thaindian News

Washington, March 29 (ANI): The day is not far when an online search engine would be giving netizens direct answers to their queries rather than web pages containing them, if researchers are to be believed.

Weiyi Meng, a professor of computer science at State University of New York at Binghamton (SUNY Binghamton), reckons that such a search engine would even be able to answer nuanced questions like “What do Americans think of universal health care?”

He says that the search engine would create a report indicating trends in opinion based on what has been posted to the web.

The researcher even believes that search engines may eventually be used to conduct polling, and even help sort fact from fiction.

Meng, who is helping make such possibilities a reality through his own research as well as president of a company called Webscalers, says that big search engines such as Google and Yahoo are fundamentally flawed.

He points out that the web has two parts-the surface web, which is made up of perhaps 60 billion pages; and the deep, which, at some 900 billion pages, is about 15 times larger.

The researcher says that Google, which relies on a “crawler” to examine pages and catalog them for future searches, can search about 20 billion pages.

He adds that web crawlers follow links to reach pages and often miss content that isn’t linked to any other page or is in some way “hidden”.

Working in collaboration with researchers at the University of Illinois at Chicago and the University of Louisiana at Lafayette, Meng has pioneered large-scale metasearch-engine technology that harnesses the power of small search engines to come up with results that are more accurate and more complete.

“Most of the pages on the deep Web aren’t directly ‘crawlable.’ We want to connect to small search engines and reach the deep Web. That’s the idea. Many people have the misconception that Google can search everything, and if it’s not there it doesn’t exist. But we should be able to retrieve many times more than what Google can search,” he said.

According to him, not only can a metasearch engine probe deeper, but it can also offer the latest information.

“In principle, small guys are much better able to maintain the freshness of their data. Google has a program to ‘crawl’ all over the world. Depending on when the crawler has last visited your server, there’s a delay of days or weeks before a new page will show up in that search. We can get fresher results,” Meng said.

This is not the first time that this concept has been discussed, for the first metasearch engine was built in 1994.

Meng, however, said: “The big difference between our technology and the ones pursued by other people is that most of the other technologies do the metasearching on top of a small number of general-purpose search engines, such as Yahoo, Google or MSN. We have a completely different perspective. We want to build large-scale metasearch engines on top of many small search engines.”

Webscalers has developed a prototype that would allow a search of all 64 campuses in the State University of New York system as well as SUNY’s central administration.

“People can use it to find collaborators. It could also help prospective students find programs they’re interested in,” Meng said, adding that the technology could be adapted to large companies or even the government.

He hopes to build a grand metasearch engine one day that would integrate all of the one million small search engines into a single system.

“There are still a lot of significant challenges in creating a system of such magnitude, but I am optimistic that such a metasearch engine can be built,” he said. (ANI)