Some call it cloud computing ‘implementation’ and some call it Software-as-a-Service ‘deployment’ to manage application processing and data storage. Some even call it ‘migration’ to virtualisation platforms and tools. But the trouble is, not enough people call it cloud services ‘integration’, which is really the most crucial aspect of bringing cloud online in the first place.
But what principles, doctrines, processes and edicts govern the mechanics of cloud integration? We might quite productively make use of Conway’s law here, which states that: "Organisations which design systems ... are constrained to produce designs which are copies of the communication structures of these organisations.”
If we take Conway’s Law to be true, then company corporate culture has a tangible impact upon the way cloud projects are handled on both a day-to-day level and from a long-term perspective. The effect of this is that staff members, from developers to network specialists, right through to marketing, sales, finance and legal all have to bring their, often quite disparate, teams and systems under a new umbrella.
That’s what happens when the prevailing environment is a cloud-based one that now facilitates computing and storage power. It is this umbrella under which new business processes will all have to gather, coalesce and coexist: if the elements or entities under this single covering layer start to jostle and collide, then someone gets wet.
At a deeper more technical level, what this new environment also means is that if two separate software applications, codebases, architecture layers, input/output channels or application components are going to be able to talk to each other and get along, they need a mutually comprehensible interface structure.
What Melvin Conway was alluding to in his original proclamation was that software systems may very often (in his view) exhibit an interface that reflects the social structure or make up of the software development team or collected community that builds them. So, in a world where cloud computing implementation, migration and integration is heavily impacted by “management by committee” direction, what are the implications?
The truth is that implementing a new set of cloud-based applications (or a complete architecture) can have as much to do with business process management (BPM) and its redesign as it does with pure cloud theory.
Call it BPM or call it business process transformation (BPT) if you absolutely must, the problem here is that CIOs are often fighting a losing battle just to bring cloud efficiencies online. Making sure they present a Conway-agnostic integration face outwards throughout implementation is surely even harder.
Actian Corporation VP & GM Lance Speck argues that as more companies adopt hybrid cloud deployments, it is even more important to keep data (and the interfaces to it) consistent as it travels between numerous systems and servers.
“While data portability is a huge headache in cloud integration, I’m seeing that real problems pop up during the post-deployment stage, largely due to companies being lured in under the context that the existence of an API should negate any integration concerns. Leaving data integration as an afterthought makes the entire deployment process more difficult, so much so that many companies end up abandoning SaaS projects altogether midway through,” Speck says.
While figures suggest as many as 20 percent of attempted SaaS deployments fail because of data integration problems, it’s not just about having clean and accurate data to integrate.
If Conway’s law holds true, then bringing data streams together is almost like a dating game and it’s crucial to have that “difficult integration discussion” early. Team A needs to find out if development and operations team B likes to wear orange trousers and eat bananas while they work, ie you need some insight, any insight, into the working culture of the code and data coalface you are integrating to.
And by the way, that difficult integration discussion you need to have early - make that early and often. Cloud-based software systems are nothing if not dynamic and those that rest upon ‘Agile’ highly iterative software application development and delivery principles will be subject to more tangential sidestepping than most.
Vision Solutions director Ian Masters argues that migrating to cloud involves an awareness of what systems you are trying to replicate and why, so there will be an ongoing requirement to check that a cloud-centric approach fits with what the business requires as a whole.
“This evolution may then lead to organisations needing to ‘break’ Conway's Law and their cloud install. For smaller companies, this might be moving from smaller and disparate cloud projects that individuals have implemented, over to a more formal cloud strategy. While for larger companies, it might be driven by greater understanding of cost models and requirements,” said Masters.
What Masters is arguing is if we do accept Conway’s Law, but reach a point where we break its guiding principles, that this in and of itself necessitates a re-architecting of our cloud system. Should Conway’s Law then be used as a barometric measure of whether cloud integration programmes will succeed? We’re not quite saying that yet, but it’s a nice theory to work with perhaps.
Continual incremental feature evolution
Tibco CTO Matt Quinn warns that cloud integration will always be a tough job whatever the job in hand. This is in no small part down to the 'continual incremental feature evolution' of cloud-based services. Quinn describes a trade off between either having a stable well-known interface that everyone can rely upon - that doesn’t always reflect the full features of the service - OR you can have a changing interface that reflects the evolution of the underlying service. But essentially, both of these still cause issues for integration.
"We see time and time again that developers struggle trying to marry the needs of the business against shifting and changing internal and external governance. Where data can live as it crosses country boundaries, how data is moved, stored or cached - all these factors can trip various regulatory issues and make integration even more difficult. Customers need to demand high levels of visibility/communication as new features are rolled into services and specifically look at how it will impact the APIs to access service features," said Quinn.
Microsoft’s Chris Patterson agrees. In his role as senior program manager responsible for the build automation components of Microsoft’s ALM tools, including the cloud-hosted Team Foundation Service, he handle clients’ in cloud implementation and deployment issues every day.
“Moving to the cloud presents new challenges to the development and operations teams in the areas of deployment and troubleshooting. Overcoming these challenges requires careful planning, team collaboration and integration that can be streamlined immensely by the right agile planning and DevOps tools,” says Patterson. “The nature of production environments makes them inaccessible to developers by traditional debugging methods, and operations teams (rightly) need to keep those environments as clean as possible to maximise performance and uptime.”
“Whether Conway’s law holds or not, most organisations find that taking a measured approach to moving to the cloud, at a pace that is comfortable for the organisation itself, increases the likelihood of success with minimal disruption,” he adds.
Integrated cloudiness will not magically prevail
John Engates, the CTO of Rackspace , is under no misconceptions and says you simply cannot install a cloud into an enterprise and expect “integrated cloudiness will magically prevail” overnight. “Unless there are people within the organisation that are pulling or pushing for cloud computing, a private cloud will sit in the corner and idle. So making cloud integration happen does have a big human factor - and if this is Conway’s law in motion then so be it. Testing and development needs to be the first application of cloud, it’s that important,” says Engates.
"In terms of advice here, I would say that you should lower expectations on how fast cloud will be adopted and pilot ‘cloud in a pocket’ - an area with a high tolerance for change and a strong desire for automation and self-service. Possibly even create a ‘startup’ within the organisation to pilot cloud and iron out integration challenges," he adds.
The Rackspace CTO suggests that Eric Ries' book The Lean Startup is a great guide to transforming how products are built and launched and that “big organisations should think like a startup” when it comes to integration sensitivities.
Whether Conway’s law has a place in cloud integration theory of not, it’s for sure that developers will (or, in fact, most commonly won’t) leave annotations, notes and guidance through the structure of the applications and software integration middleware that they will have worked upon.
If this is the “personality of the cloud” that explains the human factor in the data and code to be interfaced to then maybe Conway had a point. Look, cloud isn’t easy; maybe Murphy’s Law would have been more appropriate.
