In the beginning, Portal was born from a simple idea: Keep my two Macs in harmony. From the start, we all knew that Portal was a difficult idea, with a limited market (people with two or more Macs), and a nightmare of a development path. J Allard called it the “hardest problem chosen by the finalists.” Steve Wozniak said “there are tremendous challenges here… it seems that there isn’t an easy way to get multiple machines just to sync the simple stuff…” Jason Harris was the most direct: “Ease of Programming: low, Potential Earnings: moderate… Don’t vote for Portal…”
The need for a simple solution to this problem is well known, but how to get to such a solution is not. To that end, and in the hopes that Portal lives to fight another day, I submit the following Plan of Action.
Facing the totality of the synchronization problem is overly intimidating, so I suggest breaking the task into parts, ordered by necessity to a final, as yet unwritten application. This process can be best visualized as a three stage development: Foundation, Frameworks, and Finishing.
Foundation
As an application, Portal needs to first be installed on both systems, and then be granted permission to use the machine’s network interfaces to sniff out other Portal installations. Once they find each other, the two Portals need to connect in such a way as to facilitate file transfer between the systems, both for simple communication between the applications themselves and for the eventual transfer of the user’s files. Furthermore, each Portal app needs to be able to plug into the FSEvents system to be able to track file changes on their systems. For each file Portal is designated to sync, the app also needs to be able to identify the corresponding file on the other Mac so that it can identify conflicts. During a live connection between systems, file changes are automatically synced between the systems as they occur, so no dedicated “Sync Now” process is necessary. When systems connect after an extended period apart, the sync should launch automatically in the background.
Conflicts are the meat of the syncing problem, so lets pause for a moment on them. The zero-order, brute force solution would be to simply overwrite older files with newer ones. This is the option presented in the Mac OS X Finder whenever you try to add a file into a directory where a file with the same name already exists. For database driven applications, a partial merger is often available, such as when syncing a Contact or Calendar event from a portable device to a main system (like the iPhone to a Mac). These options should be available as appropriate within Portal’s conflict resolution system (see below). For safety purposes, older files should be backed up in either case (overwrite or merge), in case the network connection is lost in the middle of a transfer. My idea would be for all files to be moved to a temporary directory on the destination Mac first, then copied over the existing files later on. The file transfers should be checked for completeness and authenticity before the overwrite, and the older files should be moved to a safe location (either on the local drive, to a Time Machine backup, or elsewhere as designated by the user).
Network communication, file selection, file transfer, and file backup form the Foundation of a good syncing application and are the necessary first steps for Portal’s development.
Frameworks
The best way of handling conflicts is not at all clear cut. For some files, Apple has made the process simple (Sync Services), but for others you are left wanting. Thankfully, there are several independent initiatives working towards better solutions. Unison was the original favorite out of this category thanks to its robust feature set, but we shouldn’t stop there. To improve upon any one of these individual frameworks, Portal needs to take a “Best of Breed” approach to conflict resolution and use them all. The brute force overwrite method will be in place for all files and types at the beginning. Later on, specific improvements for a given file type (such as .HTML, .PLIST, .ICS, or .KEY) will be put in place on a case by case basis. If Unison is really good at DOCX mergers, but not so good at TXT, then Portal should limit Unison’s use to DOCX and find something else for TXT. Necessarily this complicates the mechanics of the conflict resolution and means that Portal will have a long development cycle. From a user point of view, the complications should be entirely hidden. From a developer point of view, one should realize that these improvements will be made incrementally and are not intended to all be ready for the 1.0 product. Version 1.0 is not the finish line!
Portal can move from a basic “smart file pusher” to a real, full featured syncing app by taking advantage of many of these existing frameworks (and improving them when possible).
Finishing
The last step is to hide all the difficult work done so far underneath a clean, simple user interface. John Casasanta said he’d “like it to be as simple as SuperDuper is at backing-up your computer.” J Allard said he’d “like to have seen less intrusive ui and for it to be directly integrated with os x“. I myself have started going in the direction of a “Time Machine”-like app where the core preferences are stored in a Preference Pane inside the System Preferences, and yet the app’s “main” user interface can be launched by a Dock icon. The goal is to make the act of syncing transparent and the process of conflict resolution quick and simple. The keyword here is subtlety. As much of the work as possible should be done in the background in such a way as to minimize visibility to the user (this means keeping processor usage down as well). When the user must be contacted, it must be in such a way as to minimize distraction from the user’s actual work. Popup boxes and dialogs should fade in and out quickly, information should be clearly presented, and the choice to be made should be explained properly. If the user wants to trust Portal to make all the decisions itself and have no user input whatsoever, that should be an option as well. If the file backups are robust, then this level of trust should be well warranted (possibly to the point of making it the default setting). File restore should be handled as efficiently as possible, taking cues from Time Machine (or even using Time Machine itself) when possible. The user shouldn’t have to learn an entirely new set of UI metaphors just for Portal.
If we let Apple dictate the user experience themes, then Portal can sink even further into the background and be that much more transparent. Extravagance and panache will be held in reserve for the occasional sync animations (user activated only, since the default background sync is invisible).
—-
In the end, Portal cannot be the end-all, be-all solution to the syncing dilemma. All Portal needs to do in order to be a success is be a solid 90% solution. (For the sake of the UI, that last 10% would be hidden anyway, so new additions wouldn’t even be visible to the typical user.) The Foundation level alone should get Portal to a 50% solution, with brute force overwrites as the ONLY “file merge” option. More options can come online with version updates. Integration into existing frameworks like Unison will also come later. All the while, the UI will evolve to match the new features, adding or removing controls as necessary.
When we started this whole My Dream App adventure, there was a stated goal that the ideas should take advantage of as many Leopard technologies as possible. At the time, we knew a little about Leopard, but since I for one was not a developer, there was really very little for me to go on. Jump forward a year, and Leopard is out (and installed on all of my systems), and all the NDA’s are lifted at long last. In that year very little was said about the applications under development for Leopard, thanks to Apple’s own embargo. Again, as I am not a developer, I was never privy to the specifics of those agreements, but nonetheless I was aware of their existence.
All that is water under the bridge now. Leopard is in the wild. The ADC has a full suite of Leopard developer notes. Ars Technica has even posted a magnificent breakdown of the core technologies at work. Reading all this material has given me pause for thought about how Portal will work and integrate into this new ecosystem.
Finder: File and Screen Sharing
The Macs on a local area network see each other much more easily now, and are much more talkative. It took very little time for me to set up file shares between my systems, so little in fact that I’ve stopped using DropCopy entirely. I have a standing link on each desktop to the drives in all my other machines, making file transfer insanely simple. The hardest part is deciding which files to send over, and, of course, dealing with conflicts. Screen sharing is another magnificent feature, and the “Back to My Mac” addition to .Mac is a powerful draw. The $99 price tag Portal was set to rally against has just gotten justifiable. I was even able to set up a screen sharing session with a Windows XP system in my house (the file sharing was one-click simple, the screen sharing was not. It took five-clicks).
The result is that machines on a network are all at your fingertips and the need for a file to be on your local system is diminished. Why sync your laptop with a bunch of files that just hog space on your hard drive, when you can just go “Back to My Mac” via the web from wherever you are? The only flaw in the argument is the issue of off-network access (the same flaw in the iPhone/Web Apps debate). The issue of “keeping two books” is also still present. Why do I want to risk having my iCal calendars on the laptop disagree with the desktop? With Leopard, .Mac becomes that much more powerful, much to the chagrin of Portal.
Time Machine, Spotlight, and FSEvents
I’m not a developer, and I’m not a computer science guy per se, so what I’m about to talk about may not be 100% technically correct. Just bear that in mind. Leopard has two headliner applications that take advantage of a low-level, system-wide event tracker called FSEvents. This API allows applications to track any and all file system activity, and it does so in such a way that the application that is doing the request doesn’t have to run all the time. Log files are created that keep track of all the events, so when an application comes online after being turned off for a while, it can take a quick look at the event history and know what’s changed about the system. Spotlight uses this to keep its search indexes up to date. Time Machine uses it to keep its backup directory up to date. Portal could use it to keep track of files that have gotten out of sync. The beauty of the FSEvents solution is that it is the user-level application doesn’t have to handle the task of looking for changes and tracking events. That sort of activity, which is potentially very time consuming, is handled by the OS itself.
The ideas I had a few days ago about Portal acting a lot like Time Machine were more prescient that I had thought. I had intended that Portal simply act like Time Machine in the sense that it was a tightly integrated System Preference that also had a full-screen “WOW-factor” interface. In truth, Portal would be well served if it could tap into the same system event tracker that Time Machine uses, and now it appears that such a thing is actually possible!
An added benefit of having a system-wide event tracker and backup system is that there is added pressure for high granularity in files. Massive databases are no longer practical in the post-Time Machine era since they take up unnecessarily large amounts of space (each change to a 50MB file means a completely new 50MB backup, even if the change only affected 50KB of real data). As granularity increases, so does the possibility of better and faster syncing. Apple has also taken to the use of XML in many application and file settings. Merging XML files is far from trivial, but it is a potentially simple file to read and to track changes in. This makes it possible to go in even tighter than the “per file” level for the syncs. This means less data needing to be sent between systems and more applications being candidates for syncing. Its a win-win situation for Portal.
Sync Services
Apple defines the “Five Phases of Sync” as:
Create a Sync Session (clients are alerted that a sync is coming),
Negotiate How You Will Sync (set modes for sync engine: fast, slow, etc),
Push Changes (either whole files or just the changes),
Resolve Changes (conflict resolution, either automatic or user-defined),
Pull Changes (all clients pull down the best file, after conflicts have been fixed).
Leopard is better suited than ever to communicate with other machines (both Mac and Windows), so setting up a Sync Session shouldn’t be difficult. Changes can be identified by calling on FSEvents, which will be running in the background on all Leopard machines (even without Time Machine turned on, Spotlight still uses the daemon). Conflicts can be resolved using the Unison framework, as has already been stated. Apple also includes a few handy services in their default Sync package for dealing with contacts, calendars, etc. that might be useful to integrate as well. The act of pushing and pulling the files from system to system could be handled by the OS and its handling of local servers (for standard file sharing, with the same privileges, etc), or it could be handled directly by the Unison layer. That sort of decision is entirely up to Martin, since I have no idea about how to do either one. (Martin, if I’m totally off base here, I apologize!)
The point I’m trying to make here is that all the pieces of the puzzle for a great syncing application are in place. Changes can be easily tracked. Conflicts can be handled by the OS for some cases (contacts, calendars, etc.), and by Unison in others. The Macs can all talk to each other very easily, so local networking won’t be an issue at all. I can even see it being possible to have Portal push itself out to all your machines so you only have to go through the installation process one time.
Let me make this clear: What I am about to talk about is pure speculation. I don’t have Leopard, I don’t know what the most recent version of Portal looks like, and I don’t know when Martin is going to make any future announcements. Martin is working on his ideas for the UI, and I’m working on mine. There are more pressing issues with the development right now than the interface. You’ll hear about that end of things later on.
Okay then. Today I want to share a few ideas I’ve had about Portal’s user interface. The more I look at Time Machine, the more I like how Apple has managed to make it both accessible and hidden. There is an icon in the Dock and there is a keystroke for quick activation, but for the most part Time Machine just runs quietly in the background. When you do need to make changes to the settings, you interact with Time Machine through the System Preferences, just like for the Dock, Expose, Spaces, etc.
This got me thinking. If Portal is going to work the way it needs to, then it has to be seamlessly integrated into the OS. Logically, that would put its interface into the System Preferences with the rest of OS X’s settings. Also, the user needs to be able to get at Portal quickly and directly, so a Dock icon would be handy. The icon could serve a double purpose as a Drop Icon for adding files to be synced.
I took a few minutes this morning and put together an idea for what I’m talking about:
I intentionally removed the “Sync” button, since I don’t want the user to have to do that sort of thing manually. Syncs will take place periodically in the background whenever possible (again, just like Time Machine). If the user feels the compulsion to force a manual sync, that option can be put in the pop-up over the Dock icon or in the Settings tab . Keeping track of file histories (Time Machine) between two systems may be impossible, but that would be a cool feature to have too.
Leopard is upon us and miraculously I had a brief lull in my workload. If there is anyone left on this site, I’d appreciate your comments!
This is how things looked from my point of view:
Tuesday, towards the end, I had realized the possibility that Portal was the Purple bar in the voting results page. That meant, at the time, that I was in second. That alone made me have doubts. Then Kevin (Hijack) figured out he was Red and went into overdrive promoting the idea. I didn’t feel too bad, after all I considered Hijack one of my favorites.
Then, towards the last two hours, the Red bar began to make serious progress. Purple had slipped to third and every time I refreshed my browser, Red was that much closer. Then it happened. Purple fell to fourth. Red then passed Blue and claimed second place all to itself. I was now 40+ votes back.
After frantic calls to everyone I knew, plus cross-continental iChat sessions with college buddies, I thought I had secured about 10-15 more votes. Not enough, and I knew it. Going into the final minutes I had all my friends scared I had blown a fuse or something. I was a mess, so I went for a walk.
When I got back, everyone was in a holding pattern in the “Final Round Voting Closed” forum. 425 posts in under three hours or so. It was chaos. So I went and watched LOST. Chaos was still there when I got back.
I can’t remember exactly when I saw that the results where in, but I do know I was suspicious right away because it looked like Portal had made it through. “That’s impossible”, I thought. I was wrong. Voting irregularities had propelled me from the brink of elimination into second place. I still can’t believe it. Portal placed higher than Cookbook AND contributed to Hijack being pushed out of the top three.
To say I’m surprised would be an understatement. I’m in shock. I’m not a designer. I’m certainly not a programmer. I don’t even own a copy of Photoshop. Cookbook looks almost perfect. The GUI is clean and refined. Hijack had full-screen animated mockups for crying out loud!
—–
This brings me to my next point. I love Portal, it was always been my favorite app in this contest. I know I’m biased since its my idea, but what else do you honestly expect me to say? Still, there have been three other ideas I really liked: Ground Control, Hijack, and Cookbook. Cookbook is safe, so I don’t have to worry about it anymore. GC and Hijack are in a different category: They still need developers to sign on! GC had a website going for it for a while and now Russell has added in a Blog to keep people on top of any progress. Hijack always had a following and now they seem to be self-organizing , even as Kevin gets some well deserved sleep. If either of these two ideas has any need for my modest skills, I’d gladly offer them, and I encourage the same from any diehard Portal supporters out there. Portal is in the loving, caring hands of MDA now, so attention can be turned to these other great app ideas.
That said, I’ll be taking some time off from this contest. Probably about a week. I’m on empty right now, and really, really need a vacation so I’m off to join Kevin on his secluded island of solitude in the Pacific. I’ll drop you a postcard. Thanks again!
