On my last job, working mostly on our iOS app as a frontend developer, something struck me hard. So much effort went into building plumbing code, things like offline caching, pagination, connection error handling. Whenever I spend so much time on infrastructure code instead of implementing features, I’ve always started questioning the very paradigms I’m working with. Back in the days when most of my effort was spent on configuring Spring and Hibernate components, I discovered Model-Driven Architecture and the ability to generate most of this plumbing out of UML in order to focus on the functional aspects of the software I wrote. But I still had to sift through all this generated code, customize it, understand all the complexity of it, until I came across Groovy and Grails, and their amazing expressiveness. With less verbose code, the path from functional specification to working feature was shorter and shorter. But then I moved to the iOS frontend, a frontend where neither code generation nor language expressiveness would help me (for the latter, it all changed with Swift of course, but that’s for another story). The more I had to fight those mobile constraints, the more it became clear that we were trying to fit a round peg in a square hole.
The web model is based on a few assumptions. When a computer connects to the internet, it’s basically a relatively dumb terminal with very little processing power, at least inside the browser since you can’t access the entire processing power of the machine it’s running on, for both security and genericity reasons. It also has very limited storage space, mostly for security reasons I guess. But the one thing you can count on is permanent connectivity. And nowadays, you can even count on a pretty good bandwidth too. Based on my understanding, the very paradigms the web runs on are tailored to this state of things. The HTTP protocol is designed to let you access Resources, those resources being online always in reach, at a specific Unified Resource Locator (URL). If you need a resource, simply GET it from the network and it shall be returned to you. And every time you request it, you will get it consistently, in its latest known state, wherever you request it from. So no need to store it. No need to do some complex processing with it either. Actually, it’s better if you let the almighty Server to it for you. He has more power, he centralizes all of that. All you have to do is request and display. Now of course, for performance reasons, you might have to keep a hold of the resource and use tricks to know whether your version is up to date, but clearly the surest way is just to request the resource again, and since the network is always there with its wide bandwidth, why bother. The entire concept of RESTful API’s seems to be based on those assumptions. And that’s what we’ve been using to connect web frontends to their centralized backends for decades.
Now if you look at the mobile space, and especially at the native app space, it looks like all the constraints are all reversed. The terminal is certainly not dumb, you have access to quite some storage space and native processing power, but connectivity on the other end, is far from granted. It can be very slow, it can even drop on you and come back without prior notice. And yet, despite all those reversed constraints, for the most part, we are still using RESTful API’s to connect mobile native apps to server backends in order to share data amongst users. We’ve always done it that way, right? Why change?
You might be offline from time to time? OK, just build some local database to cache data then, and then serialize all your JSON coming from your RESTful API into your mobile database format, whatever it might be.
Your bandwidth might be narrow, and you can’t download too much information at a time without impacting the reactivity of your app (and believe it or not, mobile app users are far less enclined to wait than website consumers)? No problem: just build pagination into your API and into your frontend and only request the information you can display, even if that makes your caching much more complicated to implement.
At some point, you realize that maintaining all those different layers of data representation (database storage, backend objects, API resources, mobile database storage, etc.) and building pagination throughout your entire app is taking up more and more time. It’s increasing your time-to-market, crippling your user experience and decreasing the quality of your code (more code, more bugs).
For a time, you think there’s no way around it. You look at others and all mobile developers seem to be struggling with the same issues. Hundreds of Github frameworks and libraries are here to help you with them.
But I’ve always been a gotta-be-a-better-way kind of guy. So I started digging into alternatives. I first discovered Firebase (recently acquired by Google, how about that for a sign?), and this amazing idea of doing everything locally in the frontend, and then let the infrastructure handle the data sharing with all the instances of the app. It was appealing, but I didn’t like the fact that I had to hand over my data to a third party for eternity: I was OK to host it in the cloud for some time, but at some point I wanted to be able to take it back and host it on my own infrastructure. Plus, I never got it to work in my simple iOS testing apps.
And then I came across Couchbase Sync Gateway. Couchbase is mostly known as a NoSQL Big Data store, uniting the best of CouchDB and Membase. But about a year ago, they added another ingredient to the mix, in the form of 2 new components:
- Couchbase Lite lets you store data natively in an iOS or Android app in a NoSQL way, which is already amazing in and of itself. No more serializing JSON data into Core Data objects or SQLite records.
- Couchbase Sync Gateway, which lets you automagically synchronize a Couchbase Lite instance with a server-hosted Couchbase server database.
So all of a sudden, with a Couchbase architecture you can centralize and share data between remote clients without having to implement a complex RESTful API with caching and pagination and all that. All you have to do is store data in your mobile data store, and let Sync Gateway handle the synchronization, signal synchronization conflicts and let you handle them, and even deal with data access control in a clever way. Less time spent on writing code (or maintaining libraries) to handle network connections and API requests, more time spent on real value-adding features. And you even get some nice additions for free, like the ability for your users to use your app even when they’re offline, and then let sync gateway share it automatically with the rest of the world when the network comes back.
For months, I was amazed by this amazing prospect and the incredible powerful mind shift it creates: I finally get to use all the storage and processing power I have at my disposal, and I can finally embrace the fragility of my network connection instead of trying to fight it.
But over the past few weeks, I’ve discovered the catch: although amazing, the Couchbase technology is still very immature. Not technologically speaking. I have the utmost respect for the engineering teams there, who are doing an outstanding jobs coming up with such breakthrough clever ways to change the way we think our mobile architectures. But on the market side of things, I’m still struggling. Couchbase is obviously coming from a very special world, a world where big companies are spending (and making) billions by storing, processing, analyzing and repurposing huge amounts of data. And then they added data sync and mobile into the mix, but my impression is that they did that as an afterthought, as a bonus for big data companies. I’m a startup entrepreneur, I don’t have that kind of data to manage (not yet) and I certainly don’t have the funding that goes with it. I just want to build a mobile app in the easiest and most efficient way, and still think ahead in terms of performance, so that when I do start to have traction, my architecture can scale effortlessly. Technically, that’s exactly what Couchbase holds. It certainly has all the components to do it. But then when you start trying to set up a production-ready infrastructure to host that architecture, that’s when things get tough:
- There is no cloud-based offering to let you get started with Couchbase in a couple of hours and focus on the development part without having to worry about the operations part. There have been some attempts, like Kurobase and CouchSyncHQ, but for some reason they both disappeared overnight, and I haven’t been able to figure out why.
- So the only way is to set it up yourself. Couchbase provides you with some Amazon Web Services AMIs, but the Sync Gateway ones seem completely outdated, and finding documentation about these is hard. Plus, they make for a very expensive setup to run.
- Recently, I’ve been working a lot with Traun Leyden, trying to help him improve his AWS+Docker+CoreOS+Couchbase cluster setup and it is easy to deploy now, but not particularly easy to customize, especially when you’re not a ops guy and you don’t understand all these technologies very well.
- So I tried contacting Couchbase themselves, asking for a consultant to help me set up a basic cluster, and I got a quote for 4 days of work, at 2500 bucks a day. 10k is certainly not a reasonable amount to spend on infrastructure when you are a non-funded startup, let alone for a side-project. At the very least, that’s a very tough mountain to climb at first, and that kind of service offering is clearly more adapted to the usual big data customers of Couchbase than to me.
Today, I’m stuck on this tough choice: should I give up on this amazingly promising technology to switch back to a more pragmatic, easier to deploy and maintain architecture, at the potential long term cost of adding more complexity to my app? Or should I stick with it and find a way to customize it, possibly outspending what it would cost me to ask one of their consultants to do it for me? I really don’t know, and I wish Couchbase allocated more resource to making it all more accessible to small entrepreneurs who just want to ship a working app in no time, and improve it very fast. Maybe we should be the ones building that layer on top, I don’t know yet if I want to shift focus like that. Just thinking out loud here. If anyone from Couchbase wants to comment, I’d love to talk about it.