Welcome Mission Critical CloudArjan Eriks
Our enthusiasm in CloudStack and the Apache Software Foundation began in 2011, when we started working with CloudStack. Citrix had acquired cloud.com, followed by the very exciting decision to donate the software as open source to Apache. Schuberg Philis had had the desire to enter the Cloud and Infrastructure as a Service (IaaS) space for some time, but as an experienced integrator we dislike vendor lock in, and this bold move by Citrix gave us the extra option we needed: an alternative to OpenStack that would allow us enter this new time in an agnostic and open source way.
In late 2011, a team of 4 engineers from Schuberg Philis began building the first stage of what would become our dedicated Mission Critical Cloud. At first, as with all new technology, we started with a proof of concept, and we built both an OpenStack and CloudStack orchestration layer to test and explore the features each could provide. As with many before us, and as many after discovered, CloudStack provided a reliability and simplicity that OpenStack simply could not match, and while OpenStack offered more features, it could not deliver the quality or reliability we desired in our IaaS software. Our goal was to build a cloud environment for both our engineers and our customers, one controlled with a standard API, that would use new and exciting technology and would help steer us towards the future while empowering us to deliver our 100% goal in everything we do. To be really certain we made a technology shift, we promised ourselves to not use any tool or brand we ever used before.
After the decision had been made, we rapidly began developing. From easy implementations of Nicira networking (now known as VMWare NSX) to hours of pain coaxing Nexenta storage to work, we learned how to bring a cloud based architecture into our way of working. Some key points that we proved included:
- Enterprise application workloads can run in a cloud
- Being agnostic and using OSS means we can drive the feature roadmap because we develop it ourselves
- Running open source at scale requires dedicated software engineers devoting their time to improving Cloudstack
- Our customers trusted us to take care of both their functional and nonfunctional requirements, and moving into the cloud was not a blocker for them
Over time, our influence grew. We started by working closely with Shannon and Sameer from Citrix, and purchased CloudPlatform as we valued that partnership. After making noise in the open source community, we were invited to talk at the CloudStack events in Las Vegas in December of 2012. A group of Schuberg Philis engineers flew over, including Hugo, Harm Boertien, our Chief Community Officer and Jeroen de Korte, one of our Mission Critical Engineers. The stars of the party though were Funs Kessen and Roeland Kuipers, who delivered an entertaining talk on stage. A later performance in Budapest (CCCEU) earned them the nicknames of Cloud Pinky and Perky. We used this opportunity to meet the original founders of cloud.com and users and developers from the greater community, which was a fantastic feeling for us.
This was not only a turning point in our usage and vision of cloud technology, but also the start of our involvement in multiple open source communities. We started becoming more connected with the DevOps community, the Lisa conference in North America, and we spoke at the CloudExpo in Santa Clara too, a bridge not just to orchestration but also a discussion of enterprise level challenges, the security implications of a cloud and how this can be integrated into CloudStack. But for the CloudStack community, we made the conscious decision to give back to the project. We hired developers to write our own contributions, and invested in conferences and events supporting the community. We started, in collaboration with Citrix, running “Build your Cloud” days, designed to make CloudPlatform and CloudStack more visible in the market. The largest event we organised with friends from the community was the gigantic CloudStack Collaboration Conference Amsterdam in November 2013, where almost 500 users and developers attended for discussions and plans for the future, though we still have to give the record of best beer and pizza event to ShapeBlue near London West End.
The position of CloudStack at the Apache Software Foundation (ASF) itself moved rapidly as well. While it was first an incubator project, it very excitingly evolved into a full ASF project. Chip Childers became the first ASF CloudStack VP, later followed by our own Hugo Trippaers and today being looked after by Sebastien Goasguen. We continued to commit more code, expanded our cloud, expanded our development team and progressed to being the release managers for multiple releases. It was in this final stage that we experienced, for the first time, friction between the community and ourselves on how to do things.
In Spring 2015, several of the key collaborators in the project gathered together in London to discuss a roadmap for CloudStack, and agree on a new way of working. Following it, we all agreed to work more closely together going forwards. From a personal perspective, we wanted to increase test coverage, and at the CloudStack Collaboration Conference in Tokyo in June 2015 all of the key players in the community agreed that we needed to change our release process .
We took on the role of release manager started at 4.6, wrote documentation about our view of the release process and encouraged debate and discussion on the mailing list, followed by stabilizing the master branch of github. This was a big change from the previous method of creating a release branch and forming different QA teams to stabilize and release it, while development continued on master. This had two major benefits: we could release a new version at any time, and now that releases were built on top of each other we could guarantee that a fix would always be present in later releases of the software: if it was in 4.6, it would be in 4.7. This new way of working allowed a monthly release cycle, with i
ncreased code review and more automated integration tests. Over 500 changes were tested, reviewed and included into the source code between November 2015 and January 2016.
Acting as the release manager with so many volunteers, developers and company goals is a task that requires both tact and stamina. We had two key engineers attempt the task: first Daan Hoogland (release 4.4) and then Remi Bergsma (4.6 till 4.8) attempted to not only organise and complete a release,
but at the same time improve the speed, reliability and
quality of the software. At the same time as we pushed for this change, we continued to run our Mission Critical customers on CloudStack. Our customers and colleagues depended on the bug fixes and quality assurance we were trying to deliver, and we had to develop while maintaining our service.
In Fall 2015, during this process, we were hit by a nasty bug. Without diving too deep, a code snippet in the HyperV module of CloudStack started reporting non-HyperV hypervisors as down, resulting in the CloudStack management server to start trying to shutdown and migrate the VMs running on them. Thanks to a high availability design at a layer above the cloud we had no downtime, but it was a rem
inder of how code we do not use when not tested correctly could cause outages. In our retrospective, it became clear that these risks needed to be eliminated. We rely on CloudStackfor our mission critical workloads, and our minimum requirements for stability, performance and reliability were not being met. It was a wake up call for us: pushing for quality in an open source fashion is key, and we needed to redouble our efforts to insist upon it to prevent a recurrence of critical bugs.
It became clear that things had to change. We devoted more of our time with the community to make sure that new commits were properly tested, and encouraged stability and testing in the master branch, and a faster pace of developing bug fixes and feature releases. For us, this was almost the optimal way of working and we wanted to push it even further. But it became apparent that this was not so
This was not a decision we have made lightly, and we are keenly aware that this move has a number of implications for not only us as a company but also for the existing CloudStack community. Over time, both versions may start to deviate from each other. We have decided that our fork will remain open source to allow for as much collaboration as possible. We are aware that others in the past have chosen to close forks of CloudStack and develop internally, but we believe in the power of sharing. Being open, inclusive and transparent are values we see as a key driver in making the IT world a little better. An open fork, with the tools and procedures we feel are needed to ensure higher velocity, greater reliability and better quality is, in our opinion, the best option at this point in time.
Our future roadmap is mainly focused on our customer requirements. We are a Mission Critical company, and have a need to cater for High Performance Computing, support for containers and integrating with other new technologies such as Kubernetes, Mesos and Nomad. Our customers’ requirements change rapidly, and we need to be both agile in creating new components while also guaranteeing quality. Besides customer requirements we will also start working on improving the architecture of CloudStack. Items that spring to mind are the plug in model, removing dead code and refactoring of important items.
So there you have it. We have decided to create a fork. It will be open for all to use and contribute to, with new governance, procedures and tools all designed to deliver quality and velocity to the project. We would like to stress our dedication to being open and inclusive, and would like to welcome anyone who has ideas about our move, any who are interested in joining us on this journey, or perhaps would like additional information about our decision to please not hesitate in contacting us.
We truly hope you understand what we are doing, and that this is not an attempt to hurt the current community.