Lessons Learned and Progress Forward
So, here are some of the things I’ve been working on since the last update:

1. XULRunner Development — I’ve got a prototype desktop client running, based on Mozilla’s XULRunner platform. Dealing with that code base has been, one might say, a challenge. There’s a steep learning curve, mountains of documentation of varying obsolescence, and as Steve Yegge observed, “well over a decade of ugly packed in there.” On the other hand, it’s got a full suite of state-of-the-art web technology, a powerful Javascript engine with access to native functionality, and built-in Mac/Windows/Linux cross-platform capability. There should be smoother sailing now that I have a better handle on the XULRunner landscape, and things will get better over time as Mozilla coders make progress toward Mozilla 2 — there are a lot of cool features being worked on there.

2. Two Ways to Connect: Desktop and Web – Last time, I discussed the idea of making a web-accessible version, as well as a native desktop app. More and more, I’m thinking of this “dual access” as a key feature of the product, and have been constructing the prototype to make it possible. Users will be able to use a single mobile client which connects to either the desktop (via WiFi, Bluetooth or USB) or to the Internet (via WiFi or cellular network). They’ll be able to sync their data whichever way they like, and switch between them as their circumstances, schedules or preferences dictate. Furthermore, sparklet authors will be able to develop once, and have their code work in both places on the same Javascript engine (modulo differences between what data is available in the environment — desktop files obviously won’t be available to the web version, without special provisions). I’ve got some architecture in mind for the web version, I’ll see if I can get a demo operational soon.

3. Sparklet Development – For those just joining us, “sparklet” is the name I’ve chosen for the plug-ins that let users sync their data between their mobile devices and whatever PC or Internet services they’d like to sync with. I’ve made progress toward specifying the structure and operation of sparklets, though it’s still in flux as I try different things out. Basically, I’m looking at each sparklet needing to have two pieces: a) an event handler, written in Javascript, which gets notifications of sync requests, connectivity events, etc, and performs the sync using a bunch of Candyspark-supplied APIs; and b) an Ajaxy web interface, written in HTML/CSS/Javascript, for configuration and realtime display. All this will be zipped up with a specific structure for easy deployment, just like Java does with JARs.

I’m trying to make sparklet development as simple and low-effort as possible, by making the APIs easy to use and smoothing over the asynchronous stuff. It would be great to work out ways of running sparklet functionality directly off of live web pages, either modeled after Microformats or Greasemonkey, but there are significant security and usability issues with that and it needs more thought.

4. Mobile Client Dev – I’ve got prototype clients working on S60 (in Python) and the Android emulator (in Java). Their functionality is limited, but enough to prove that the basic idea works, and should be easy to extend as needed. The plan is to keep the mobile client as simple as possible, since that will make it easier to port to additional devices going forward. We will eventually need to develop a more sophisticated solution for detecting device differences (as some will allow access to resources that others lock down, for instance) and exposing that to sparklet developers in a clean way.

5. Documentation, Marketing and Website Dev – I rolled out a new website design yesterday, with a cleaner look and much increased explanation of what Candyspark is and why people should care. I’ll be putting up more info in the coming days, and expect to have a fairly complete pitch up there by the end of the week.

I also set up a Beta Signup script to start gathering interested future users. The signup asks a few questions about mobile synchronization tools, habits and desires to get a bit a feedback as well.

6. CTIA Meetings – I will be heading to Vegas in a couple of weeks to attend CTIA. If you’re interested in meeting to see a demo and learn more about what we’re up to, please get in touch!

Over the past weeks, I’ve been doing a lot of reading, brainstorming, whiteboarding, experimenting, diagramming, idea bouncing, mocking-up and proof-of-concept hacking, and now I feel I’ve got a much better idea for how to build Candyspark sync, and a more concrete list of the parts and components that will be part of it.

So here’s a rundown of the current plan, citing the technologies and techniques that will be included. All of this is, of course, subject to change as we make progress.

Overview: How Candyspark Works

I put together this diagram back in November 2007, showing the main pieces of the Candyspark system as I envisioned them. At the time, I hadn’t dug too deeply into how to actually build this, so a few things have changed and a lot more details have been filled in, but it’s still basically the same idea.

Decisions and Updates
Here are some of the things I’ve been working on, decisions made, and other updates to my plan for Candyspark:

1. XMPP for connectivity to mobile devices — We’re going to use XMPP (aka, the Jabber protocol) to communicate between mobile devices and the rest of the system. If you’re interested in Internet technology and haven’t investigated XMPP, I’d really suggest taking a look — it’s pretty cool. XMPP is to real-time communication what HTTP is to document access, and it’s got the potential to explode over the next few years as it starts to catch on in more niches.

There will be XMPP client code on the phone, which connects to an XMPP server either on desktop or internet. This brings us a well-designed and well-tested protocol with powerful extension capabilities, and will ease integration with other technologies. Plus, it should work over any TCP/IP connection (WiFi and cellular natively, BT and USB w/ a helper) as long as ports are open, and it makes it clear that there’s minimal difference between desktop and internet syncing.

2. XULRunner as a desktop dev platform — That’s the open source Rich Internet Applications environment that’s been built by the Mozilla Foundation. It’s the basis of a lot of popular apps, including Mozilla’s own like Firefox and Thunderbird, but also third-party efforts like Joost, Songbird, and Miro. It’s open source, free to use and redistribute and cross-platform on Windows/Mac/Linux.

XULRunner should make it easy(ish) to build out the Candyspark desktop UI using HTML/CSS/Javascript without the need for heavy native development in C++/Java/whatever. On the other hand, it can be extended with native code using XPCOM; thus can integrate native libraries like ffmpeg, opensync, Bluetooth/USB helpers, etc.

For prototyping, we’ll build out some simple XUL user interface screens, either on the standalone XULRunner executable, or as Firefox plugins. Unfortunately, there doesn’t seem to be a very well-trodden path into XUL development, though there are some efforts to improve that, like Mozpad and OpenKomodo.

However, we’re still keeping an eye on Adobe AIR as an alternative. Adobe’s got a lot of experience building good developer tools and making development a more pleasant experience, and they’re making it clear that they plan to push hard to support development on the AIR platform. AIR also has a much better installer story, making it easy for new users to download, install and configure in just a few clicks from the web.

3. Candyspark Central: Server-Based Sync — With the use of XMPP, it’s clear how the same mobile client can sync with both the desktop and the Internet. It’s also clear, from talking to potential users and investigating competing products, that server-based syncing is appealing to a lot of people. Thus, we’ll design and build a server version alongside the desktop version. (This isn’t in the diagram above.)

From a business angle, the idea of Candyspark being both desktop- and server-based complicates the story somewhat, and leaves some issues to be sorted through regarding market positioning and business model. On one hand, having a server-based component makes for a more comprehensive solution and can lower the hurdle for new users to begin using the product.

From a technology angle, however, the server-based solution is an excellent fit with the other parts of the system. We’ll use server-side Javascript (running in SpiderMonkey or Rhino) to maintain a common code base with the desktop client/RIA. I’ve also spared a bit of thought for how this would eventually scale (horizontally, simply, and cheaply, of course…).

4. Mobile Client — A client will be installed on the mobile device to enable communication with the rest of the system (via XMPP as discussed above), and access to the device’s files, data and services. To begin, this will need to be a user-installed app, which means Candyspark will only work with smartphones that make that possible; eventually, we hope to get pre-installed and/or integrated with the device operating systems, so that won’t be a long-term requirement.

For prototype purposes, I’ve been hacking using Python on S60 (PyS60) on Nokia smartphones. When we get to the beta, we’ll probably launch using the Python version compiled into a SIS file; we may eventually port to C++ to improve size and performance, but only after we get the basic structure down. Symbian Platform Security will definitely be an issue for some Sparklet applications, so it’s good to get some exposure to those issues early. Also, S60 has the largest installed base among smartphone OSes, so is a very important platform for finding beta users.

We’re planning versions for Windows Mobile and Android as well. Android, in particular, should be interesting, as it already encourages peer-to-peer communications with other Android devices using built-in XMPP client code. So far, they only expose a few API methods and assume connection to the GTalk server, but it will be interesting to watch that evolve as they flesh out the platform details. [edit: look like people are figuring out how to get around Google's artificial limitations] It could be a very interesting platform for Candyspark.

We’ll probably open source the mobile client code, primarily to make it possible for device builders (including open-source device hackers like those from OpenMoko) to integrate with the Candyspark system.

5. The Candyspark Platform: Sparklets — One big area of the Candyspark plan is to expose a developer platform for third parties to build sync applications, which we call ’sparklets’. These are client applications which run in the desktop client or on the Candyspark server, which manage information flow between the mobile device, the desktop, and the Internet. Within the Candyspark framework, all user data syncing is handled by sparklets, and each individual sparklet is responsible for one kind of syncing.

The Candyspark product will come with a set of pre-installed sparklets for handling various tasks: contact sync, desktop file sync, podcast subscriptions, etc. Third parties will be able to build their own sparklets, using tools we provide, and make them available to users via the Candyspark Catalog (for free, or perhaps for sale; imagine the iTunes Music Store, but for sync plugins). Power users will be able to build their own sparklets, for private use or for publication.

This is shaping up to be the aspect of Candyspark with the most open questions. In Marc Andreessen’s terms, we’re building a Level 3 Internet platform. Issues of security and privacy will be very challenging, since we’re dealing with users’ most important data. Developers can be highly demanding users to build tools for; as Marc writes: “lots and lots of people have opinions about platforms, but the people whose opinions matter are programmers, and people who can make decisions about what programmers program”; we need to make a platform worth building on.

This is probably the toughest part of the plan to pull off, but I think it will be worth it.

6. Business Issues — The basic plan is to start small and cheap, and get a rough prototype built as soon as possible while self-funded. With that, we’ll pursue two avenues forward in parallel: building awareness with companies who could license our technology (like handset manufacturers, carriers, specialty service providers, etc.), and launching a public beta to get users to try out our product. I’m not going to go into budgets, revenue model, funding and all the rest here, because, a) none of your business, and b) largely speculative at this point. So best to leave that out.

Onwards

From here, the next big step is to build out these components enough to get the system as a whole operational. There are still plenty of open questions, but there are also several well-defined components ready to be built, at least in prototype form.

If you’re interested in hearing more about what I’m thinking, if you have questions or suggestions, if you want to be a beta (or alpha) tester, or if you think you might want to be part of this project in some way, please do let me know.

These legacy sync applications were designed largely in the pre-Internet era, and largely fail to take advantage of the features and applications available on the web. Further, many of these apps are bloated and fragile, having been incrementally upgraded over the years, even as the devices they sync to have advanced in leaps and bounds.

As an individual user, my requirements for a sync system can be summed up rather succinctly: it must work on my phone and PC, it must properly sync my data and applications to the proper locations and services, and it must be convenient to use. To satisfy a large body of users and to fit within the constraints and tendencies of the larger mobile ecosystem, these requirements will need a certain amount of translation, however.

I propose that to meet the needs of current and future mobile device users, a modern sync solution should have these properties:

• It must access Internet services as easily as local files. Today, web services provide us with email, calendar, photo sharing, bookmark storage, IM, backup, document collaboration, reference and too many more applications to list. A modern sync solution should be able to easily link in these services alongside local applications and storage, and allow users to hook up the services they care about with a minimum of pain and stress.
• It must accommodate the capabilities of modern mobile devices. Our phones have hundreds of features and contain dozens of different types of data. They support multiple types of downloadable application and connect through half-a-dozen radios and network interfaces. It’s a tough job, but a modern sync solution must tackle this complexity.
• It must be a platform that can be extended with new services and features. No one company can anticipate all the possible permutations of applications and services to be synced, so instead of a single application, we need a platform for developing applications. Furthermore, it should be an open platform with low barriers to getting started with development.
• It should use standard networking protocols. Modern mobile phones are, at their heart, small networked computers. Syncing should work across whatever connection is available, whether across the desk (USB cable or Bluetooth) to across the room (same WiFi network) all the way to around the world (across the Internet, via cellular or not). Some aspects of syncing may work differently depending on the capabilities of the available connection mode, but the base functionality should be available regardless. Further, by sticking to standards, we can take advantage of the existing infrastructure and knowledge base, while being confident of continued advances in the future.
• It should be platform-agnostic at the core. The PC side should work on both Windows and Mac, and Linux would be real nice too. The mobile side should be manageably simple to port between different phone models and even OS platforms.

This, in summary, is what I hope to build here at Candyspark. Over the coming months, we will work to design and prototype a new sync application platform, which (I hope) meets these requirements.

I’ll be posting ideas, updates, related news and random thought here in this blog. If you’re interested in this space, or if you’ve got ideas/opinions/disagreements/whatever, please post in the comments or get in touch!

Mobile, desktop, Internet: I’ve come to see these as three legs of the stool that support an information-rich life. Between them, they take up probably half of my waking days and provide the majority of my communication, reading material, and (increasingly) media consumption. And so, it’s disappointing and annoying when they don’t work very well together, and to me it seems that one of the biggest gaps is between the mobile (specifically, my phone) and the desktop (specifically, my laptop).

So, lately I’ve been pondering this issue, and I’ve come to believe that while it’s a complicated problem, it’s one worth diving into, and potentially rewarding both personally and commercially. Thus, I’ve formed Candspark, a start-up company which will develop products for desktop-to-mobile sync and communication from an Internet-centric point of view

What’s a Candyspark? It’s a “candy spark” — what you get when you crunch a piece of crystalline candy, and it makes a flash of light through triboluminescence. If you’ve never seen it yourself, try wintergreen flavor Life Savers for an especially bright spark. No connection with mobile, desktop or the Internet; I just like it as a name.

I’m launching Candyspark to develop mobile-to-desktop sync solutions for the Internet age.  That is, I plan to develop tools and applications to help people synchronize their mobile devices (especially phones) with their personal computers (like desktops and laptops), but in ways that fit with the Internet-enabled lives we now live.

Over the next few posts I’ll write more about what I mean by that, who I am, and what I hope to accomplish.

Cheers,

-cashman