Building Encyclopedia Galactica with Wolfram Alpha

Fans of Douglas Adams The Hitchhikers Guide to the Galaxy will be happy to know that the evolution in internet search technology is now one step closer to creating the Encyclopedia Galactica (Encyclopedia Galactica was first used by Isaac Asimov in Foundation in 1942; building the Hitchhikers Guide is a different animal altogether). Wolfram Alpha is the newest entrant to internet search and has led some to believe that this will be the biggest technological breakthrough on the internet so far and has the potential to change the internet as we know it. Wolfram Alpha was showcased at Harvard recently and will be released later this month (May 2009).

The hype around Wolfram Alpha rivals that of Segway when it was unveiled. It was said that the whole neighborhoods will be reconfigured for Segway, such a marvel it was. It hasn’t happened. Nevertheless, Natural Language Processing is the holy grail of internet search and has proved to be difficult to implement. Google has been working on this for some time and early implementations of similar systems (AskJeeves) have not captured the imagination of the internet users in the way it was anticipated. To properly respond to a natural language query, the system not only needs to have access to vast amounts of data (that a search engine has) but also needs to be able to parse the query, understand the intent, collect disjointed sets of information, determine relevance and credibility and ultimately collate the information in a manner that makes sense.

So how will Wolfram Alpha change the internet?

Wolfram Alpha introduces a new paradigm in internet search; one that pushes ‘information’ to the forefront. Hitherto, the search engines have been more concerned about presenting data to the users. Regardless of how smart the search engines have become in uncovering ‘credibility’, ‘authority’ and ‘relevancy’ in the search results, they still present a set of links to pages that the user needs to visit to determine if they answer the basic question that the user is trying to ask. Wolfram Alpha will allow the user to ask the question as they naturally would and than try to find an answer to the question based on the data on the internet. It would also present a list of resources relevant to the user’s question to help user explore more. So if you have a question on how many golf balls can be fit into a Boeing 747; Wolfram Alpha may be able to compute that answer for you (sorry I haven’t tested this yet but will do so as soon as the system is available). Answers to questions like this can be found today through many Answer sites (WikiAnswers, Yahoo Answers, etc) but the problem is that these are user submitted responses with very little vetting. Even the current font of all knowledge on the internet, Wikipedia, is often criticized for lack of vetting and the potential for incorrect information. Wolfram Alpha will have a staff (currently estimated at 1000) who will source and vett the sites that the system will access. Additionally, search engines and answering services can only deliver what has already been asked before. Wolfram Alpha promises to be able to answer questions that have never been asked before.

Uncluttering the web, less information overload and understanding the user will ultimately make the internet more useful and accessible. Hopefully as the time goes by, the system will learn and become even more relevant. I am sure a farmer in China and a farmer in North America asking the same question ‘Should I plant Soybeans this season’ probably expect and should get different answers.

Applications to build localized intelligent systems

The increased relevancy and the ability to present direct answers means that the users using a mobile device to access information will find this much more attractive option as they will not need to go through pages of search results to get what they want. I am also interested in looking at possible applications of Wolfram Alpha engine for localized systems. Think of an intelligent home in the future that is well connected. You wonder if you should water your outdoor plants today (or more likely, the control sensors in your watering system makes this decision). You would like to determine the soil moisture levels, the types of plants you have and the amount of water they need, what is the weather forecast, how much sun do your plants get where they are located, etc. The answer to this question will depend on collating the data that is local to your home as well as from the external sources (e.g. weather report). If the Wolfram Alpha system works as it is advertised, it can become the core search engine for your home, tapping into your local data sources (all the devices in a connected home), and the internet as needed to drive many of the decisions in the future. Sure such devices are built today but they require specialized programming at each device level. Having a common engine that the devices could refer to make their decisions will greatly simplify these gadgets and their use. Of course, quite a bit of this is speculation but is certainly within the realms of possibility.

The creator of Wolfram Alpha, Stephen Wolfram, doesn’t expect this to challenge Google or Wikipedia or anyone else in the traditional sense and fully expects the system to work in tandem with many of these existing services. As Mr Wolfram points out on his blog, Wolfram Alpha takes a completely different approach to handling information.

Will it be able to find the answer to Life, the Universe, and Everything? We shall see.

If you would like to test drive the system when it goes live, you can do so at the Wolfram Alpha site. Be sure to report back your impressions of the service in your comments here.