After all the hype that has occurred prior to launch of the new WolframAlpha knowledge engine I checked it out after it went live a couple of hours ago. Here is my personal evaluation of this new site. Having seen many search engines and technologies rise and fall within the last decade I was curious to test the site as soon as possible. As I am not a scientist I will try to provide a view which focuses on the normal internet user.

First Tests

As I had recently been trying to find out how many castles there are in Scotland I typed „castles scotland” into the input field. WolframAlpha returned the not very helpful message „Wolfram|Alpha isn’t sure what to do with your input.” so I changed the query to „number of castles in scotland” which still returned not a single result. Maybe WolframAlpha just does not know that so I tried „number of rivers in scotland” which was interpreted as “is Rivers, Manitoba, Canada in Scotland, Connecticut, United States” and returned „Result: no”.

So I changed the query to „number of rivers in Germany” which worked out fine - great! The first correct result was returned!

In the minutes that followed I went on to enter different queries and analyzed the results to come up with the following list of inconveniences and problems from my point of view.

Where Are All The Links?

The fundamental elements of the web that keep it all glued together are links yet I didn’t get any result which allowed me to dive deeper into the information. Surely after clicking on some values a layer occurred that allowed me to copy the values but only sometimes there were links contained that I could click on directly. Links from one page to another page is what sometimes keeps me for half an hour or even longer on Wikipedia traversing through the links from one article to the next although I only have been looking for a simple answer to a problem. WolframAlpha doesn’t seem to offer that kind of linking most of the time.

Just searching for Inverness which is a city in the Scottish Highlands near the famous Loch Ness returns some useful information on the city. It also lists cities nearby. Good idea but I’d like to click on any city name directly to perform a search for that city, e.g. Edinburgh. What I need to do is either click on the name of the city, find the link to the city and click on it or copy the city name or enter it manually into the input field and perform a new search.

Another example: searching for „Walt Disney” does not return a result on the person Walt Disney but on the company. I need to select „Use as a person instead” which by the way is in a very small font which I personally regard as bad usability. From the Walt Disney (person) result there is no link back to the Walt Disney Company he founded. Why not? Wikipedia has it.

Weird Incompleteness

WolframAlpha doesn’t return anything on “london underground” which is the oldest subway system of the world. Yet “new york subway” returns a result. Likewise you can search for “longest subway system” which returns the New York City subway system but don’t search for “oldest subway system” (which would be the London Underground) which will return nothing at all. This is even the more fascinating if you keep in mind that Stephen Wolfram was born in London.

Google’s top results for “oldest subway system” show that it’s the London Underground - and I don’t even need to click on the results to get that information as it is contained in the snippets Google provides.

If you enter “liberty island” WolframAlpha doesn’t find anything, however Google does. Yet WolframAlpha knows the Brooklyn Bridge. Seems to be more important to know the Brooklyn Bridge than to know the island where the Statue Of Liberty is located.

There Often Is More Than One Answer To A Question!

Most of the time you are stuck with the result without any helpful links whatsoever. Searching for “toons” shows “Interpreting “toons” as “towns”" which for me seems very far-fetched. It also does not return a result at all but allows you to select a city. I would have expected to receive the result for “toon” instead which is the correct singular for “toons”.

The word “simpsons” is interpreted as “sum formula” yet a search for “bugs bunny” returns information on a Warner Brothers movie entitled “Bugs Bunny’s 3rd Movie: 1001 Rabbit Tales (movie)” with data from the IMDB. I would rather have seen some historical info on Bugs Bunny as on Wikipedia.

Likewise a search for “james bond” returns no info on the fictional character but returns information on the movie “A View To A Kill” from 1985 which is a somehow matching result but not really what I had expected. Why has exactly this movie been selected? Luckily WolframAlpha also shows a list of all of the other James Bond movies. But, again, something unexpected happens. If you select “Casino Royale” from the list you won’t see any information on the movie from 2006 but instead on the 1967 TV version of the book. There seems to be no ranking. And there is no way for you to find out that there is another movie from 2006. No link, no info. If you only depend on this information you’re doomed to fail. If you search for “Casino Royale” manually info on the 1967 movie is shown but you can select the movie from 2006 directly.

Using Google searching for “Casino Royale” shows the IMDB entry for the 2006 movie as the first result which is what I would have expected. The second result from Google shows the 1967 version. Great!

If you query WolframAlpha for “Wolfram” you’ll be shown info on Stephen Wolfram - the creator of WolframAlpha.

Yet if you try searching for the Google Founder’s last names “Brin” and “Page” on Google that kind of bias doesn’t exist there. For me WolframAlpha’s result in this case is not an objective result.

Some results seem to be very blatant errors. Just searching for my surname “Kimmel” as a single word without any spaces is interpreted as the distance between “Kim, Sughd, Tajikistan” and “Mel, Veneto, Italy”. Ouch! However searching for “Jimmy Kimmel” returns information on the talk show host.

Searching for “Illuminati” returns no result (conspiracy theorists: here we go) yet searching for “Adam Weishaupt” which has founded the Order of the Illuminati returns a result.

I could literally go on for hours but you should just try it for yourself but forget searching for “Mickey Mouse” and “Seinfeld” as no results are returned for these terms currently.

Bottom line: contrary to mathematics there is not always only one solution to a problem. Just imagine Google would only show you the one result it thinks is the best match. You wouldn’t like that either I suppose.

Diving Into The Scientific World

Although I am not a scientist I just tried some searches with some unexpected results as well.

I have learned not only from Google but also from the previous WolframAlpha searches performed above that queries seem to be case-insensitive. That was an error. I searched for “h2o” all in lowercase to get info on the water molecule. Yet WolframAlpha interpreted this as a degree value. No water here. Searching with H2O in uppercase works though returning the expected result.

Second try: I entered “au” which is what I believe to be the chemical abbreviation for gold (from Latin “aurum”) but this has been interpreted as “astronomical unit”. Although there are many links at the top of the page there is no link for “as a chemical element”. Searching for “Au” returns the expected result however. To get the correct results you obviously need to know the correct capitalization of the word you are looking for.

I don’t think the normal web user knows that.

Third try: searching for “fly genome” returns no result. Google shows the expected results with the Berkeley Drosophila Genome Project first. I then searched for “drosophila genome” on WolframAlpha but got no result either just a reference to “Animals: drosophila”.

Some More Searches

Here are some searches I performed which really give great results and bad results, respectively:

Good:

• how old is barack obama
• 100 EUR + 8%
• salinity dead sea
• chocolate vs. pears

Bad:

• inauguration of barack obama
• when was george w. bush born (Ouch, returns his father’s birthdate: George H.W. Bush)
• “telephone inventor” and “who invented the phone” (but “who invented the telephone” works)
• who discovered america
• “ufo” does not work, but “UFO” does, “hiv” and “HIV” work both

One-Fits-All Approach Is Wrong - Case-Sensitivity Is A Problem

In my opinion WolframAlpha should not return only one result or if it does it should offer a better disambiguation to the user. The one-fits-all approach is wrong. Currently it still doesn’t return the correct result quite often. Google on the other hand shows not only one result but (most of the time, apparently) the results that it believes are the ones the user has been searching for but does not decide for the user what he seems to have intended. Searching for “flytrap” on WolframAlpha returns a word definition as “a trap for catching flies”, not even the botanical definition of the “Venus Fly Trap” or anything else. If you search Google for “flytrap” the results contain completely different entries allowing for a manual disambiguation. There is a company named “Flytrap Technologies”, an eZine named “Flytrap”, a Wikipedia article on the Venus Flytrap and much more. Therefore you can refine your search and search for “venus flytrap” on Google to get more information on that.

Bottom line: If you know exactly what you are looking for and know the complete correct term and capitalization you will most of the time get the results you are looking for. If you don’t you’re lost quite often. WolframAlpha knows only one “John Smith”, Wikipedia knows more than 80 people with that name.

Let’s hope WolframAlpha gets better for every one of us, not just for scientists - I’m sure they’ll love it. For now I’ll try WolframAlpha often but I’ll stick with Google and Wikipedia for most of the searches. What about you?

Anways, it still contains a huge amount of knowledge and definetly is something to thank the creators for. Surely it will develop over time. Let’s hope it doesn’t go where WikiaSearch has gone before.

Tags: search, searchengine, wolframalpha

This entry was posted on Saturday, May 16th, 2009 at 12:21 pm and is filed under Search Engines, Security. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.