After more research I finally managed to find an existing app that’s close to Bubble Fish. It’s a Windows-only app called “Babylon”. (http://www.babylon.com/) If you have access to a Windows box, I really suggest trying it out using their free trial. It’s not bad (as Windows apps go.) It comes with some free dictionaries that are stored locally, but does have some online options, including Wikipedia.
Despite the initial deflating feeling of “Aw! It’s *kind of* been done before”, I think the existence of this app is very helpful for Bubble Fish:
1. You can get a feel for what it would be like to use Bubble Fish, and also get ideas for how it could be much better.
2. Proof of feasibility - if they can do it, it can be done. I spent a bit of space addressing the problem of grabbing the text. For text rendered in standard text boxes, Babylon has no problems. Although it’s a Windows app, I can’t help but think there must be an equivalent method on OSX, so now I’m not too worried about that potential problem. One interesting note, if the text is inside graphics or some non-standard text control, Babylon will still try to grab it, most like using OCR. However the results suck! So OCR probably doesn’t work for this application. That’s minor though since most words you will search for won’t be inside graphics.
3. Indicates probable performance
In the voting judges notes, Jason raised the concern that it would just take too long to grab results from the web. Since Babylon has a Wikipedia option, you can test that directly. On a system with standard DSL, it took Babylon 1-2 seconds to grab the Wikipedia results. It felt more than fast enough for me during my testing. Certainly way way faster than switching to a browser, and fast enough that I didn’t get annoyed at it.
Regarding selecting the appropriate data source (e.g. Spanish slang), you would still need to direct Bubble Fish as I discussed, but with a smart GUI, I think the selection process could be made fast enough to be worthwhile.
In the event that you do check out Babylon, you might ask “well, will Bubble Fish be just like that, but on a Mac?” I think the important differences (i.e. improvements) between Babylon and Bubble Fish (aside from being on a Mac) are the following:
1. Being on a Mac, Bubble Fish would be much prettier, with cool effects, possibly transparency, and whatever else the developers can think up.
2. Babylon displays the results from all of your active data sources in the same window. You can expand or contract them with the mouse. I found this incredibly annoying. I only want to see the information I’m after, and quickly. If the default data source happens to be wrong, I want to instruct Bubble Fish what to use using a few keystrokes; I don’t want to go collapsing things with the mouse.
3. I still think the real advantage of Bubble Fish is the plug-in architecture for data sources. Babylon has a lot of free dictionaries, some you can buy, and some other sources like Wikipedia. But you’re still stuck with only what they decide to provide and in their formatting. Even though Bubble Fish itself would be a commercial app, if the plug-ins were open and free, a real community could develop around it. It takes that kind of community to realize the full potential of an idea like this.
Since technical feasibility is one of the requirements for the winner of this competition, I’d like to offer some thoughts on implementation issues affecting Bubble Fish. In particular, the major implementation problem is grabbing the text on which to search. The main value of the Bubble Fish app is in the convenience it offers, so implementation difficulties that restrict that convenience become very important, even if it’s just saving a keystroke here or there. I unfortunately haven’t done any GUI development for the Mac, so my comments will be limited by my ignorance of exactly what’s possible and what’s not.
First, I’d like to thank Stephen for pointing out the dictionary search feature from Tiger (which I didn’t know about.) I looked into that, but unfortunately this feature only seems to work with Apple apps. (If this is not correct, someone please correct me.) So evidently it uses front-end integration with those apps to get the text. Still, if Bubble Fish could use the same methods it could work with maximal convenience with at least Apple apps, which on a Mac goes a long way. Whether this is possible or not depends on how that feature is implemented and if it’s documented or not.
So assuming for the moment that the functionality used by the Tiger dictionary search is not available to us, what are the options?
1. Worst case scenario - copy
In this scenario you would a) select word/phrase with the mouse, b) copy with Apple-C, c) Bubble Fish control key. In the case of step c, if Bubble Fish were active, it could constantly monitor the clipboard and do a search on whatever showed up there. Thus, you could trade step c for the initial activation control key. However, if might be annoying if you forgot to turn it off (plus the extra key to deactivate Bubble Fish.)
This approach has the advantage that it will definitely work, and the disadvantage that it’s pretty yucky. Note however that it’s still far ahead of the manual alternative: c) switch to or launch browser, d) navigate to appropriate web page, e) paste word/phrase into page, f) click some button.
2. Better - automated copy (possible?)
I’m not sure if this is possible or not, but it would work something like this: a) move cursor over word and hover, Bubble Fish figures out which app contains the text, sends the app a fake double-click to select the word, then sends a fake Apple-C to copy the word, then grabs the word from the clipboard. In cases in which this worked, this would almost be the best case scenario - that is, after activating Bubble Fish, you could just move the mouse around and not have to do anything else. On the downside, you never know what you’re going to select from a double-click. If it’s in English and you only want to search on one word, it might be ok. However, some foreign languages don’t use breaks between words, you might want to search on a whole phrase, or the app might just do something weird with the double-click.
3. OCR - great but hard
Another option that comes to mind is to grab the final graphics around the area of the cursor (possible?) and do optical character recognition (OCR) on the graphics to get the text. This has the advantage that you just need to move the mouse to the right location, and that’s it. For phrase selection, you could just highlight the phrase and have the OCR look for inverted text. On the downside, it’s hard - you’d have to find an OCR package with the appropriate rights, and you’d probably need one for each language. I noticed however that Google recently open sourced an OCR package.
4. Any ideas out there?
Wow! My very own picture on a web site that more than 10 people will see.
Now that I’ve had a few days to think about this in more detail, I want to post up more thoughts on how Bubble Fish might work. I’d love to here other thoughts and criticisms, and I’ll try to incorporate all good ideas.
First, I want to describe my vision for how things might work assuming all the technical problems can be solved adequately. Again, these ideas are not necessarily the best ones; if you have better ideas, please contribute. Go Bubble Fish!
Bubble Fish Manager
We’ll need some app to handle BubbleFish configuration. Following are some things it would handle:
- install/uninstall data source plug-ins
- select which data sources are active
- select which data source is the default
- set general options, things like bubble transparency level, launching on startup, mouse-hover delay, hotkey selection, other bubble pop-up behaviors, etc.
General Operation
Bubble Fish would run in the background and most of the time wouldn’t do anything; you obviously can’t have things popping up while you’re working. When you come across a word or phrase on which you want to search, a control key would activate Bubble Fish. In the best of all possible worlds, simply hovering the mouse cursor over a word would be enough for Bubble Fish to figure out what the word is. Assuming for a minute that that worked, for as long as Bubble Fish remained active, you would get an information pop-up for anything you hovered over. There are a bunch of options for when pop-ups go away. They could go away as soon as you move the mouse away (but not into the pop-up), stay until another pop-up appears, stick around based on some control key, etc.
Searches would be done using the default data source. If you had multiple sources active, that might not be the right one for a particular word. In that case, a control key would give a selection bar in which you could tab through the active data sources to the one you need. They could be identified by name and a cool graphic that visually identifies them. You’d probably want the new one you select to become the new default.
It would also be nice to have an automatic mode in which if a search yielded no results, it would automatically try the next data source in the active source list. For example, if you selected a French word, but the default source was a Japanese dictionary, there would be no results. If you had a French dictionary in the source list, it would eventually get a match there. The effectiveness of this mode would obviously depend on the speed of the searching. During such a process, Bubble Fish could give you a status pop-up telling you which source it was currently trying.
Another control key would deactivate Bubble Fish. However, it might be nice to be able to make certain pop-ups sticky so they stay around until you close them manually. You’d need to be able to move them around on the screen in that case.
Information content
Bubble Fish would handle the display of the information. The best format for data source plug-ins to provide to Bubble Fish might be HTML. Whatever the format, it would need to support graphics and formatting so we can display rich and organized information. Also, it’s important that the plug-ins be able to provide the display format since that will depend on the type of information. It would also need hyper-links. Normal links might launch the page in your default browser. We might also have special links that would trigger another Bubble Fish search using the same pop-up.
Potential data sources
The sky’s the limit here, and there are more potentially useful sources than just I could think of. Most would probably be on the web, but for some cases where there are free downloadable databases for things you use heavily (e.g. foreign language dictionaries), it would be nice to have the data local to speed up the response time. Among the sources I can thing of:
- language X to language X dictionaries
- language X to language Y dictionaries
- special purpose technical dictionaries
- encyclopedias
- standard search facilities
— web keyword (i.e. Google)
— image search (Google, Flickr, etc.)
— video search (Google, YouTube, etc.)
- local system searches
— your email
— your documents
— i.e. Spotlight!
Ok, so that’s my basic vision. Next time, the tough part - what will make or break Bubble Fish - some thoughts on how Bubble Fish can grab the text.
Go Bubble Fish!!
The basic idea for the Bubble Fish program is not really novel, and has been implemented before in limited ways. Essentially, a mouse hover (or other event) triggers pop-up windows containing information about the text located under or near the cursor. For a very good example related to translating Japanese web pages, point the following at any Japanese page:
http://www.popjisyo.com/WebHint/Portal_e.aspx
Another example is how recent versions of Trillian will give you pop-ups with Wikipedia content for selected words in your messages.
The Bubble Fish program would perform this function with the following improvements:
1) It would function on any visible text in any application on the desktop. (hot-keys to turn it on/off.) The major technical challenge of the project is of course to grab the text without the cooperation of the containing applications. Obviously, it’d be better to give info with just a mouse hover, so one approach is to grab the screen graphics around the cursor and use optical character recognition to get the text.
2) It would function with multiple data sources, either local or on the web. For example, English dictionary, language X to language Y dictionary, encyclopedia, etc. The main app handling the text grabbing and pop-up displays would have a plug-in architecture to allow custom data sources.
I haven’t found any apps that do this. (If anyone knows of one, please tell me and I’ll just use that!)
My original motivation for this project was Japanese to English translation. The solution I linked above works great for the web, but what about other Japanese text that may be on my systems? Personally, I believe that we’re currently living in the dark ages of computer software. Usually, people probably don’t know that they’re living in a dark age; everything probably looks pretty modern to them. But in this case, I think it’s obvious. (I mean, Windows is the dominant operating system!) There’s some text on my screen - I can see the text - there are databases out there with all kinds of information about that text - I don’t know that information. Why the hell should I have to highlight the text, copy it, open some browser or app, paste the text, then click some button just to get that information? Isn’t the computer supposed to be working for me? Anyway, the bad thing about dark ages is that simply realizing you’re in one doesn’t get you out of it. You can just make little steps in that direction. I think this would be a pretty nifty step. I hope you think so too.
More details of the app are to come. (Assuming someone doesn’t say “Heh, X does that already!”)
