Last day of the year

Today is the last day of 2015. Something weird, that we still celebrate some date in half of the world because some hundred years ago it was decided this way (and could have been decided different)

A fine year has been, I would say, being on holidays.

This 2016 is going to be even better...  For now, we are trying to increase the number of people working on the project. We have offers from partners to participate or increase participation, and we have offers from customers that want to make this better.


I believe we have created something new, that adds value to all elements of the chain. Customers have overcome build vs. buy - and old IT mantra- with a build to buy, powered by open source. Partners have discovered a fine way to learn and get new hires in a market where finding trained cloud professionals is so difficult, and of course we have a new way of interacting and adding value to partners and customers, while enhancing the product.

The question remains: how to make this better?. Continuous improvement is needed, we need to refine, enhance and update the model so more value is given to all parties. Any ideas? We have 365 opportunities, starting tomorrow, to use them.

Meanwhile, rest in peace, Ian. We'll keep working on Open Source.

It's nearly Christmas

So Merry Christmas and Happy New Year! Enjoy the holidays if you want and enjoy my good will if you don't, or just don't feel identified with it.

Meanwhile we have advanced a lot. Yesterday we had a review of the things we are doing and it looked quite impresive.

- Rafa showed tier rating. Now you can use tiers (i.e. from 0 to 2 CPU --> x $/month, 2-4, another amount).... and also a new way to tariff: fixed+variable. Easier to say than to do, I should say. This is a great enhancement, as we know can make tariffs that covers a lot of new uses - with the example that now we can use the same schema that Red Hat uses in its subscription model.
- Amaury showed us the new interface to edit rates. You can now add rows in a table instead of having all the options already there. It will become quite important when we increase the number of options allowed.
- Tamara showed us how to put currencies into the rates, and the initial translation of CF4 into Spanish.

On the other hand, we have new offers to collaborate in the project from different companies. Too early to say, but I expect a significant increase of resources with the start of the new year... coming from different companies.

We also have decided to take a try of an open source project to follow our scrum process
https://tree.taiga.io/

We used Trello before and is great, but we were lacking some measurements that would be great to have, let's see how we adapt to the new platform.

Next steps are harder. We have done everything we can without modifying the models, and now we need to change configuration files, models and everything around.
Focus points: assignment of rates to users, resources and groups, and being able to charge at different levels, not only with the data coming from the C&U database. We are also starting to see how we can integrate with external billing systems.

We've had some contacts with opencell, an open source billing company and product, and we need to work hard on that.

Stay close, this is going to evolve quickly.

Attribution 2.0 Generic (CC BY 2.0) https://www.flickr.com/photos/07anycolouryoulike/6566897119/

Some useful addresses

This sprint we have really advanced quickly, now we are able to do tier rating (almost), choose currencies (almost), we have changed the way rating is selected, and prepared the interface for the next steps. So with this sprint we will have the interface in a status that will be the base line, what is done today, but some good tier rating and all done in a more sensible way.

Results are good, but we are still learning the craft, there are too many small details that change how things are seen and done.

We need to migrate to patternfly (https://www.patternfly.org/), and use haml
We need to improve our tests. For some extrange reasons we go straight to code instead of going through testing first, and then things break down... Perhaps because rspec and capybara are great, but defining your tests with them is not so obvious.

For instance, I am still working to know where are described the contexts of capybara, as you can't access all the methods from all they test types... And all of us need to revisit patternfly.

And we need to polish the way we work, basically.

We are getting there.


Next: we are now adding new functionality beyond rating, we will be focusing on adding some capabilites to the exiting paradigm, just before we change things internally.

It is being fun

References:
http://guides.rubyonrails.org/
http://railscasts.com/episodes/267-coffeescript-basics
http://www.bootply.com/new#
http://www.rubydoc.info/github/jnicklas/capybara
https://gomockingbird.com/
http://rspec.info/
http://guides.rubyonrails.org/
http://www.rubular.com/
http://haml.info/

Another sprint goes by and we have something to celebrate

Let's celebrate!

 

So... we've got the first functionality inside ManageIQ. On time for the next version of the tool... and it has costed us some blood, I have to say.



Let's cellebrate :D This week is Red Hat Week, and we are proud to reach a milestone in our little project.

https://github.com/ManageIQ/manageiq/pull/4887

It has taken 11 days to merge it.... And we have learnt a lot while we worked on it.

First:

You need to visually know what you intend to do... We had a revision web conference and discovered that we had agreed to deliver something... that was not what we wanted.

Solution: mockups of the final solution. Now we are doing it first, with the rspec: Tests and mockups to know what we want exactly the tool to do.
It is such a stupid thing  trying to use TDD and fail on doing this properly.
Consequence: some functionality won't be included in ManageIQ until we refactor it because we have learnt a lot - meaning: we have done it wrong.

Second:

We need to focus more in the basis, iteratively. Git, patternfly, ruby, rails, we can't take them for granted, we need to improve how we do things. We make mistakes, we don't understand why tools are used for... and one month later you know why that functionality was there...

Third:

Sometimes the business part of things is hard. Even being myself the product owner, I don't get specifications detailed enough to make everything understandable, and thus it is good we are using Scrum... Imaging if we discovered that the functionality was wrong after 3 months instead of 3 weeks.... We wouldn't be talking about learning, but other things.

Pull requests

Our first pull request is here... and the second one.

Last week we had a wonderful conference call with some senior developers in ManageIQ. Amazing how easy things can be if you have the right knowledge in front of you.

So today we have done a couple of pull requests:

One real:
https://github.com/ManageIQ/manageiq/pull/4669
One pathetic (mine):
https://github.com/ManageIQ/manageiq/pull/4668

So mine is just to add .ruby-version file, so rbenv does know what version to install, and anyone programming with the source code know which version of ruby he needs to set up the environment to.

The other one is quite more interesting: Tariffs in real life are not linear. They are based on tiers. So anything down 512MB has one price, and if you ask for 473 MB, you are still charged for 512MB. So we need to reflect that.

Interesting thing about this. It is not so easy to get a pull request, we want to collaborate, not being a burden:
1. We need to do it in smaller pieces. These pull requests are perhaps too big. Functionality is ok, everything works together, but reviewing perhaps it is a little too long. Next time, we will do smaller chunks. Hard to know whether is better to do a full functionality pull request or divide it...
2. git is a wonderful tool, so we are learning to stash, rebase, update, squash. Thanks god pro-git is free :D
3. manageiq does not work with postgres 9.4, the default version in Fedora 22, so now we need to use 9.3 or 9.2 (the one included in RHEL 7/CentOS 7)

rbenv

So you need to develop in Ruby.

I am working now with a couple of different environment:
- ManageIQ needs Ruby 2.2.3
- Openshift.com, where I am developing an application (PaaS is nice), needs to use 2.0.0p645.

So I started to have a look at how is the best way to do it.  In RubyonRails.org it is recommended to use rbenv, so I did give it a try.

I followed the instructions to set up the environment, compiled a new ruby, installed it for my application... and did a bundler install.... "ERROR"

Those things of installing with git clone. Dependencies where not there. It took me a while to search the web and see that I need to install cmake, and some libraries for development that were needed to compile the gems (mariadb-devel, etc).

After that
$ gem update --system
$ gem update
$ gem pristine --all
$ gem install

And then
$sudo dnf install npm  #for node.js needed to run jscript and rails generate with it

made the day.

Second sprint

We have closed the second sprint.

Wow! Things are more complex than we expected. This sprint we have focused on upgrading what is already there:
 - Change how rates are defined. Now we can define a tariff in different units (i.e. 1€/KHz/month or 10€/MHz/month, etc)
- Tiered rates. Now we can define tiers for rating (1-2 CPU 5€/mont, 3-4 CPU 9€/mont, etc)
- Currency management, we have starting adding definition for currencies.

And then we came to Rails... Coding an enterprise grade application is complex, and there is a bunch of controllers that we need to understand... and we couldn't.

So we have closed the sprint a little too soon and we will need to continue working on this. We needed to take the decission of whether to delay the sprint for a week or close it as it is... and we decided to close it. No demo to be shown, a lot of things learned.

Now we need to define the next sprint.

Cloud

What is a cloud?

And can we make a cloud out of things that are not cloud-oriented? Short answer: it depends, but mostly the answer is no.

One shoud be aware of the definitions already there for the cloud. In particular, the National Institute of Standards and Technology (NIST), has released a definition of Cloud that is the basis for many other definitions and clarifications worldwide. The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., host firewalls).

Five characteristics define a cloud:

• On demand self-service. .
• Broad network access.
• Resource pooling.
• Rapid elasticity.
• Measured service

Reference: http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
 So the questions remains. Can I get a cloud with whatever infrastructure I have. And I believe the answer is twofold:

• Yes. As you can provide those characteristics with any virtualization, if you can provide Internet access and create and destroy VM automatically, based on measures. That is basically what many traditional virtualization are selling. "Use our virtualization with a management layer and you will get a cloud.
• No. A big no here. A cloud is about applications, not about infrastructure, so creating a management layer on top of your infrastructure will look like a cloud, will behave like a cloud, but won't be a cloud, because your application won't be cloud ready and you won't benefit from the new paradigms.

And that is what I believe is the big wrong thing on cloud. You should be expecting to have improvements because you use a cloud. Those improvements are both in performance, and in TCO for the solution, and in order to get them you need several things, all together:

1.  A management layer. Something that can provide and manage automatically, the generation and access to your workloads, both internally and externally, and provide measures and elasticity.
2. A lifecycle management tool. Because your workloads won't be static, you need to configure, update and depoly content to them.
3. A cloud workload. Elasticity comes with a price, you need to have VM that are ready to join a cluster of other VM, automatically. And that means changes in your development, getting away from HA to cluster-apps. Perhaps a PaaS.
4. An infrastructure that is aligned with the previous points.

The only thing is that you don't need to implement all of them at the same time. Of course you can get a "cloud" on any infrastructure, including physical and/or traditional virtualization, using traditional apps, and you will have most of the features defined by NIST. But you just miss many of the improvements of using a real cloud.

Because the value is in the application! So using a lifecycle management tool, or a management tool, will help you automate your workloads, and thus increase productivity and TCO - we have been doing that for years- but you won't be using a full cloud, because your application won't be there yet. PaaS, SaaS, are already helping you to go that way.

Just change the way you develop and deploy applications, over an infrastructure optimized for cloud, and then you will see the benefit. Increased utilization of hardware, rapid adaptation to the workloads, and thus CI/CD and everything that leads to innovation.

And of course, remember that we are not here to rebuild you datacenter tomorrow, so be prepared to have an hybrid cloud, with different workloads over different infrastructures, for many years.

So, coming back to earth: you really need OpenStack, and containers, and a Cloud Management Platform, and automation, but it is quite possible that the reasons you need them are not those that you are thinking of.

So let's not get confused. A Cloud Management Platform is a great tool, a needed tool, any automation is good, but be careful and know where your limits are.

And test, test, test, do proof of concepts, refactor, and test, test, test. You will need it.

Sprint 1

Sprint 1 is all about infrastructure (it will finish in two days time).

We have the laptops to program.
We are using a virtualization platform (Red Hat Enterprise Virtualization), ready to go.

Our environment is quite simple: a couple of hypervisors, connected to a synology NAS that provides NFS and iSCSI, with a self-hosted-engine manager.
We have created a group on github to hold the code and for ManageIQ.
https://github.com/rhus

What is needed to do the rest:

• A continous integration server. We have evaluated travis-ci, but the version for open source software works in a IaaS/PaaS mode. We need to be able to deploy inside our environment. It is a pity, because ManageIQ code already is prepared to use Travis and have the configuration working. We will need to go with another wide used tool like Jenkins
• We are using BDD/TDD, so we need a test bench. We are using Rspec, as it is integrated with Ruby, and it is the one chosen by ManageIQ.
• Did I mention github?

And we were so lucky to get some time of greggt, that allowed us to go through the code... amazing how a quick 60 minutes can reduce your self-study time several days.

Any ideas on how to solve the travis-ci issue? We don't want to reinvent the wheel.

This Friday, first demo day, and preparation for the sprint 2... where we actually will code. Exciting :D

Gathering requirements

So, in a closed development process,  the main (and only?) source of information would be the customer. Perhaps with a little bit of work on our side to see how that can be reused, and even with more than one customer been queried about their opinion.

In this case, the result has to be good and useful for the community, so I am opening an open wiki:

https://sites.google.com/site/chargethecloud/

You are invited to participate, comment, and enhance it... I will be reviewing comments and requests for enhancements to the content.

Let's make this a real solution, with a clear definition of what is needed and the priorities of it.

A thought on internships and Open Source

I often wonder what would be the best outcome of an internship - for every one, the company, the intern, even society. I as an intern myself for most of the years of my studies - something not quite normal in Spain - and as a result I want to make sure that everybody gets something out of this.

Being an intern is good - for me was quite an experience - because you really start looking at the world from a different perspective. In my time that didn't mean too much, I was just cheap workforce, but nevertheless I came to see things from a different perspective completely: my colleagues were real workers, and the kind of things they talked about was not what I was used to see and hear with my fellow students.

In this case, this group is different. They are still located in a environment that is still mostly university related, and thus they will have a limitation on the kind of soft skills they will have. That is a pity, but makes things easier for them, as they don't have to be physically in a place, and can adjust their schedules to their studies.

On the other hand, they are involved in an amazing project, I would have loved to have an internship like this. And, more than that, we are making a great effort to have them trained. After this internship, our team should be the first ones to get a job - and I hope that Red Hat and our partners Logicalis and Produban are their first options.

Nonetheless, participating in an open source community makes thing easier to get a job afterwards. Red Hat in fact employees a lot of engineers that starting working in a community in their free time, and finished working in the community under a contract.

So, it is training the solution? I want to make sure that any body that goes through this internship ends up being a better worker in the future, and quite biased to Open Source (and Red Hat). So we are getting them trough the best training we can get.

I believe that is good, it is something that adds value to the internship, students really need to learn a lot after they finish, and we are getting them access to a lot of great people inside the organization to make things even better. Of course they can't have a coffee with them - most of them being in the US (United States), only with me. It is hard to have a coffee with somebody in the US (Universidad de Sevilla) - but they have meetings, video calls, and the contacts to work with them.

Finally, and in any case, pressure is there, this is not coffee brewing, but doing work in a real environment, with quality, a real customer behind, something so good that can  become open source.  Sometimes I hear comments about how this work should not be done with interns... and I wonder if that is something we should say to all those people that are already contributing in their free time to open source: "hey, you don't know enough to do this, go and have some drinks with your fried".

The answer is no, I am convinced that in this group we have some of the future stars of the community. We need to train people to work with Open Source, not just hope that they will for whatever reason that got us here. Having them doing something meaningful, a challenge, having trust in them, is part of the open source way. Of course this will be more difficult for them that going around for three months doing easy tasks that nobody wants to do... but that would be a case of money only.  I often find people that dedicate their careers to do manual things that should be automated, and the reason is the same: it' easier to make it start, it's easier to keep going without finding a new challenge every day... but it's a waste of talent.

Let God help us live up to the expectations. And let's put some value in people's life, just a little bit.

Training

How do you get two interns trained... When you have another one joining on Monday?

Of course, doing the courses you would do in any other case. We really need to make them work full speed as soon as possible if we are going to make them do a good job... And we want them /us to success.

This week Tamara has -almost- also joined the team along with Rafa and Amaury. Coming from a Telco Engineering, she will have a different point of view to our team She will be joining in August, just next day, and happened to pass by and say hello. Produban (The IT of Banco Santander) has decided to collaborate with us giving her an internship that she will be doing with us. So now we have Produban, Logicalis,  and Red Hat working together in this project.

Meanwhile, we have moved into CloudForms training - the downstream project for ManageIQ. The tool is great, but complex. There is so many nuances in it that is convenient to them to do formal training, and that is what we are doing the following next two weeks before holidays, making sure that they do the training a consultant would receive to work on the tool.

Tamara, Rafa, Sergio and Amaury. A winning team

It is also exciting that we have got them to choose a Master and Grade project aligned with this project. Amaury will be devoted to analyze what is the best way to do configuration in a highly complex environment like this, what parts and elements are needed, what options should be available. Tamara will be investigating what are the best options to create Charging Data Records to be exported.

This is being really fun!

Imponderables

So now we are working full speed.

Our two first interns have started learning what is needed, they are finishing a couple of books on Ruby and Rspec, reading some code.
We've talked on Cloud, Virtualization, Open Source, charging...
We've got the hardware prepared, and I am installing it.
We are preparing some courses on ManageIQ/CloudForms

And then the first management problem. We have the opportunity to work with another two interns, but the University has a quite complicated way of doing things... so complex that my time suddenly has been absorbed by petty management things.... like making sure every contract, every detail is worked out.

One never things that the most complex thing in the beginning is going to be administration...

Smile

A great two days talking about the project and discussing next steps in the University of Seville. Still working to get another programmer...
Rafa on the left, Amaury on the right, myself in the middle.

And.... we started

Last week we started.

July 1st, and I couldn't be with the interns... July 2nd, and I was there for a day.
Not enough, tomorrow I will go back again.

So I have a lot of things to think about:
- How do I empower the interns? We are a team, and they need to make decisions, more decisions than myself.
- I need to teach them while I am remote, and I need to learn from them. I can't forget they are the professional in this... I am just learning. The problem is the big list of things to learn

•  What is charging and billing
• Ruby
• Ruby on Rails
• ManageIQ, what is it for
• Open Source...

So we had an interesting meeting where I talked with them about Open Source, charging, a little about the product.... and that is all. This week I expect to show them a demo, see how they are doing, start listening...

At least, we know:
- We are going to be using TravisCI for CI... Trello, Rspec... More things to learn in the real world

Plain English

Just imagine what would happen if you had to select three interns, worked with them, liked a few of them - a lot - and then discovered that you can't choose them....

That was what happened to me last week. I went to the University to select three interns for our development, and I found a group of very good candidates.... that didn't speach English.

I was not asking too much, a good candidate for me what either a very good student, or a student that has been working throughout the university, whatever job, whenever place. There is a lot to learn in McDonalds about how to work in a team that can be very good in the working place. Little knowledge of Ruby / Python, passion for Open Source, willingness to learn, maturity to take decisions were part of the deal, but I could cope with it, and then English, just enough to work in a community where you have to interact.

I am still wondering how that is possible. XXI century, people that are finishing their studies in a technical field, that are used to read books in English, as there is no much literature on computer science that is properly translated... but not enough.

The direction was telling me they try to convince students to study English, and they try, but they don't listen. What kind of jobs are they trying to get? Any high level profile one will start with an English exam, even for jobs where English is seldom used...  My 6 years old has a better accent than most of them.. (yes, I am angry because I could see the potential)

So at the end I have a great candidate that will work with us, and two pending positions for what I will need to start searching again good candidates... This time in the selected group of people that can speak more than one language.

This is being fun, and quite tiresome.

Interns

So... If you have to find interns, what would you be looking for...

In my case:

• Passion for open source
• English, at a good level
• Technical knowledge at a certain level
• Something more that I just can't define

I was told as a student - eons ago - that you can have to types of university studies that are worth looking for:
- The first one has high marks, they are good students, you know they can learn and do academic things.
- The second one is the collaborative one. They work, they do more things than just studying, they are open to the world, work in teams.

I like both. I understand that every team need to have a mix of different people, and profiles, be balanced. I wonder what is the best for Red Hat. So I want to see their grades so far, I want to know how they work with people, and I want to see if they are good programming. I've asked them to bring a small program in a language that they don't normally use but they will need in the project, Ruby and Ruby on Rails.

And, what do you ask them? I haven't realized how complex is to do the right questions in a short interview to know them.

We are so close to start now...

The starting point

What is our starting point?

Imagine you want to collaborate in an open source project. It should be easy, you think, just go there and program, but in fact, there are some soft rules to follow:

- First: you need resources. So we have started to work with the University of Seville to find people interested in Open Source, in cloud, and in working with us for an internship. Amazingly, we won't be alone, at least 3 companies will be collaborating.
- Second: a good value proposition,  I was going to recommend one of the books I am reading, but I have discovered that although the first book on business models was using licenses from creative commons for charts and slides, and thus was perfect to share, this one is not, so I don't recommend it any more...
- Third: we need some infrastructure, a small one, just enough to run RHEV so it can be managed. Working on that
- Fourth: whatever is needed to make the development, TDD (test framework), CI (Maven and/or Jenkins), a compiler,.
- Fifth: let's not reinvent the wheel... what open source components can be used which license is compatible with ManageIQ? The license should be compatible and no problem created beacuse of it. And architectural should be compatible too...
- Sixth: Documentation: where should we store documentation? Wiki style?

So I've been investigating and:
- Trello is a good tool to keep requirements and make them go through a Kanban process. Building a project there.
- Github is needed, upstream uses it already, so aligning is the sensible thing to do. We could think of some tool like Gerrit for code approval, but due to the small team we are, it wouldn't make sense to have an installation for gerrit onsite. We need to evaluate gerrithub as an interface to github, any way
- We are not mandating anything to code, as long as it works :D

Starting point

Back to the future

A few years ago I was having the best time of my life at work. I was doing standardization at OMA, working in R&D on charging and billing at Vodafone, travelling around the world, singing every morning.

For different reasons, I decided to move on. Let's say that I was comfortable with my work but the impact was somehow tiny, just pats in the back, and pay "could be better", so I was impelled to change work. A pity, as I believed - and still believe - that I was making a difference.

I've been missing that since I left, enjoying my new work, learning new things, and thinking how could I have done the work better, and created new opportunities for me and for the company.

A few weeks ago I had the opportunity to do it again. I am currently working with a couple of universities to see if we can develop, in the best of the R&D world, a new functionality into the ManageIQ Open Source product. 

Coming back to charging and billing, with universities, researching new ways of gather and present information in the cloud, developing something real and that can make a change.

 I will use this blog to show what we are doing and, in the best of the open source world, gather inputs and ideas to implement

This is so good :D

Sergio