Archive for ‘Cloud Computing’

October 18, 2011

google app engine 1.5.5 sdk not a relief

task queues + backends + cron jobs don’t work well together on the development server. this has been the state for the last few months sucking up huge amount of dev + qa + release cycles.

checks like these continue to fail on the development server but work on the production (appspot) platform.

assert get_backend() == ‘consolidate_billing_units’

October 17, 2011

salesforce contact integration on google app engine

contact integration with salesforce is easy. here is how we did it recently on a Google App Engine project.

this works for enterprise salesforce accounts only (and ofcourse developer accounts)

step 1: get an authorization code from salesforce


def get(self, **kwargs):
code = request.args.get(“code”)
args = {
‘redirect_uri’: REDIRECT_URI,
‘state’:workspace # salesforce allows user # defined to be stored in the # state variable

if code is not None:

# call back from salesforce …

args[“client_secret”] = SALESFORCE_CONSUMER_SECRET args[“code”] = request.args.get(“code”)
args[‘grant_type’] = “authorization_code”

url = AUTHORIZE_URI + “/services/oauth2/token?” + urllib.urlencode(args)
response = urlfetch.fetch(url, method=urlfetch.POST, validate_certificate=False) content = json.loads(response.content)

access_token = content[‘access_token’]
instance_url = content[‘instance_url’]
refresh_token = content[‘refresh_token’]


# call salesforce

args[‘response_type’] = ‘code’
url = ‘%s%s’%(AUTHORIZE_URI, “/services/oauth2/authorize?%s”%urllib.urlencode(args)) return redirect(url)

step 2: use the token to call into salesforce

def salesforce_query(email, contact_fields, access_token, instance_url):

fields = ‘, ‘.join([f for f in contact_fields])

args = {
“q” :”SELECT %s from Contact where Email = ‘%s'” % (fields, email) }

headers = {
“Authorization”: “OAuth %s” % access_token

url = ‘%s%s%s’ % (instance_url, “/services/data/v20.0/query?”, urllib.urlencode(args))

response = urlfetch.fetch(url, headers=headers, method=urlfetch.GET, validate_certificate=False) return simplejson.loads(response.content)

October 17, 2011

a simple and elegant yes no dialog using dojo

a very common situation in ui design is to let the user confirm an action (especially one which is destructive) via a dialog. similar to microsoft windows yes/no or ok/cancel dialogs.

here is how do accomplish this using dojo, something I customized for our needs,

1. first the markup



2. next the handler

function yesno()
// assume that yesno is called when the user clicks on the object which is key // to the elements being destroyed.

var obj = this;

// my yesno is reusable, you could arbitrarily bind the same // dialog to more than one object. i store the title and the // description of the dialog in the object itself.

var description = obj.node.dialog_description;
var title = obj.node.dialog_title;

// tie the user action handlers – here if the user clicks on yes, // the object has to be destroyed.

dojo.byId(‘yesbutton’).onclick = dojo.hitch(obj, remove);

// no? simply hide the dialog.

dojo.byId(‘nobutton’).onclick = dojo.hitch(obj, function(){yndialog.hide()}); dojo.byId(‘yncontent’).innerHTML = description;
yndialog.set(‘title’, title);;

function remove()

// this is the remove handler. i associate this handler with the // object itself such that ‘this’ points to the object.

var obj = this;

dojo.destroy(obj); // or trigger a server side action and then destroy this node. }

June 13, 2011

Symantec.Cloud Review

I recently had an opportunity to attend a Symantec Conference (Enterprise 2011). I was mostly attracted by the 45  minute session on Cloud Computing – which was towards the end of the half day conference.

I waited with eager anticipation as I was ignorant of any of Symantec Cloud offerings. Being an ex Veritas File System developer with a long association at Symantec I was naturally curious.

As the hour approached, the presenter went on to unveil a set of Symantec Products offered via SaaS model (mostly concerning with Data Security/Availability – full list here

I was disappointed, to say the least, at this attempt to pass off SaaS as cloud computing. I was hoping to learn of some cool new storage/compute virtualization story (Symantec and Vware are buddies) tied together with utility computing and security thrown in the mix. Alas, no such thing.

Something did not add up in my own mind after that presentation. I was not sure why I should be disappointed that “SaaS is not the same as Cloud Computing” (A similar debate is going on concerning Apple’s iCloud).

SaaS – Cloud Computing? It is and It isn’t.

It is difficult to find a precise and widely accepted definition of what cloud computing is and what it isn’t. However, there are some Cloud Computing Guarantees (well, promises) that are generally well accepted. Some of which are –

  1. Availability – promises on service availability (five 9 availability for e.g.)
  2. Scalability  – promises on how well the service scales (horizontally and vertically)
  3. Utility Computing Based Billing – pay as you go
My own thoughts when someone mentions cloud computing is the cloud computing infrastructure – hardware and operating system software sans the (business logic) applications. Similar to Google App Engine, Amazon EC2, Microsoft’s Azure.
Applications written to run on such infrastructure are expected to exhibit availability and scalability properties.
It is interesting to note that Symantec promises 100% service availability and a guaranteed latency (from which you can draw inferences about the scalability). I suppose from a service consumer’s point of view this is all that matters.
It is anybody’s guess as to what kind of infrastructure powers Symantec.Cloud and consumers should not be unduly concerned. They should instead focus on the service availability and response times which are very very good indeed.
April 27, 2011

why we should avoid tipfy – a gae framework

as we recently found out, choosing a framework for GAE can be a costly and frustrating affair. for various reasons we decided to go with tipfy ( for one of our projects. at that time the tipfy framework version was 0.5.9.

GAE has evolved and i guess tipfy had to catch up with the new additions.

the current version of tipfy is 1.0b1 … the source layout has completely changed which means we have to spend effort to make it work with the latest form it has taken.

the maintainers of tipfy clearly don’t think backward compatibility or easy porting an issue worth consideration. we have since decided to do away with tipfy and port it back to vanilla GAE. from experience the only thing we will miss is the jinja templating which was superior to the django template engine a year or so ago. in fact more important (from our requirements point of view) things like the google map reduce functionality should work out of the box – it was a royal pain to get it to work on tipfy. the tipfy implementation appears quite weak.

our recommendation for anyone out there trying to choose a framework to start on GAE is to stick to the core GAE itself and at the most venture into django.

of course, we welcome tipfy experts to educate us on what we are missing. but for now, tipfy is not yet mature for serious development.

April 9, 2011

3 simple steps to adopt cloud computing

Cloud computing is now synonymous with Flexible Provisioning and Scale. Find out below if you are taking full advantage of cloud computing.

The As Is deployment – lowest adoption cost, reasonable benefits:

Move the server application “as is” to a cloud server. This is nothing but a co-located server, at Amazon for example. The provisioning and maintenance of the application is still a self driven task.

The win is in the dynamic on demand provisioning. Easy to compute the ROI here. Let us say that your application needs to be available all year round – but cater to seasonal demands. Say it costs $400 to host your application to cater to peak demand. You would end up paying 12*400 = $4800 per annum to keep your application up. Most of the time it would be under utilized. Cloud computing has made it really simple to change your compute capacity as easily as setting a reminder in your out look calender. With amazon or google, you could just log into the admin panel and say that you need additional resources only on certain dates. At the end of the month you get billed for the amount of resources you actually consume.

The Managed RDBMS deployment – reasonably low adoption cost, reasonable benefits:

A lot of work has to be done to ensure that the application is available. i.e. a replication strategy and policy to keep the database available. This is still a lot of effort and money. The alternative is a managed RDBMS, where the provider (amazon or google) manages the database. They worry about keeping the data safe from being lost. Much harder to do the ROI here – as the time spent in managing this would have to be offset against opportunity costs. Note that there would be some amount of code restructuring (not a lot) to get this going. An example of this is the Amazon MySQL RDS. At the time of writing, google is yet to announce the availability of their hosted sql service.

The Application Rewrite – highest adoption cost, highest benefits (arguably)

If your goal is to write an application which scales very well then you should consider a complete application rewrite to take advantage of the storage APIs. Hosted RDBMS is still a single machine (or a cluster) running a database server – with bottlenecks – be it memory, cpu, networ or disk.

Cloud computing offers storage APIs to access and manage data unlike traditional methods of file or rdbms storage. Because of the underlying architectural differences, cloud datastore offers better scalability –

April 8, 2011

The recent Google App Engine Outage: pseudo transparency


The app engine team in its RCA explanation stops at this point :

The Datastore relies on Bigtable to store data (read more about
Bigtable here:  One of
the components of the Bigtable is a repository for determining where a
specific entity is located in the distributed system.  Due to
instability in the cluster, this component became overloaded.


Instability in the cluster? Sounds like a bug in the system which Google does not want to talk about in detail. But why go through this whole ritual of trying to be transparent with the community? Being an engineer, this is not an acceptable RCA. Who are they kidding? We don’t know whether the problem is really fixed or it is still lurking out there. Knowing a thing or two about clustering, instabilities of any kind are not that easy to fix. Let us wait and watch.


Still a fan of GAE.


February 2, 2010

What exactly is cloud computing?

I get asked this question a lot by our clients. And I often ask myself – what is cloud computing. And when I’m lost – I refer to the wiki — according to which the current definition is …

Cloud computing is Internet– (“cloud-“) based development and use of computer technology (“computing“).[1] In concept, it is aparadigm shift whereby details are abstracted from the users who no longer have need of, expertise in, or control over the technology infrastructure “in the cloud” that supports them.[2] Cloud computing describes a new supplement, consumption and delivery model for IT services based on the Internet, and it typically involves the provision of dynamically scalable and oftenvirtualized resources as a service over the Internet.[3][4]

The Internet at the beginning of the definition is the key – and instead of development it should be  Deployment and Delivery. The rest is just fancy wording – it could have easily been summarized as “Outsourcing of a service via the Internet“.

For ex.  You can outsource the management of your emails to Google or Yahoo. Outsource the management of your CRM to Salesforce.

Again, what is cloud computing?

January 9, 2010

Cloudy combo Google App Engine and Amazon S3 combo pack:

Google App Engine provides you a ready to use environment to build and host web applications. The hosting environment is secure and made highly available by the google’s cloud computing infrastructure.

While GAE provides you such a strong platform to build and host web applications, it has it’s own limitations as well. While Google’s infrastructure gives you abundant computing power but not the  case with storage which you would expect if you are developing web applications which require a large amount of flexible secondary storage as well, for that even the applications requiring moderate amount of secondary storage to store the files.

Coming to the actual point, GAE has a limitation of allowing applications to store not more than 10MB of a single large file on it’s server. This may not be a problem for some of the web applications. This is a serious issue for many web applications which assume sufficiently large underlying storage, given these days secondary storage is available in abundance at cheap rates. Some of the developers and users are shying away from using GAE as a platform for their web application development and hosting only because of this reason. And to make life hard google does not have a paid service either to work around this issue.

What are these web application developers supposed to do? Switch to something else which offers sufficient storage along with computing power. The other better options available are Amazon EC2 and VPS. Give me a break. Why do we want to run away from the problem? Well, look around. Amazing Amazon is there to help you out. We need to think creatively and be selective when making decisions about technology these days. There are a bunch of technologies lying around. It is just a matter of selecting and integrating the two technologies and using them to satisfy needs. I am not trying to promote any specific cloud here, but to highlight the fact that GAE and Amazon S3 work really well. And it turns out to be a cost effective solution.

GAE provides the development and hosting platform for you web applications at free of cost while Amazon S3 provides you reliable, powerful and abundant storage at cheaper rates. GAE and Amazon S3 together can do wonders.

Enough of theory, let me explain it to you with a practical example.
We, at BYGSoft, had a requirement to develop a web application for one of our clients. It was a simple web application which will work as a download portal for it’s customers. We chose GAE as a development and hosting platform for web application and Amazon S3 as the back end storage.
Read More :