I was fortunate in being able to attend Citcon this weekend and met some wonderful, talented people. I proposed a couple of openspaces, one loosely based on automating value and the other around test reuse.

I’ve been thinking for a while about how I see people (including myself) rewrite the same business logic in our behavioural specifications, for our integration tests, for our performance tests, for our security tests, and for our behaviour-driven monitoring. This seems counter-intuitive to someone as lazy as I am.

I think we should reconsider the deployment pipeline as a testing pipeline. The purpose of the pipeline is to increase confidence in the release candidate as the stages are passed. These stages can be considered as hoops for the release candidates to jump through. As new functionality is added, so the hoops will need to be tailored to ensure the new functionality is fit for purpose.

Here’s an idea of a delivery pipeline I’ve seen used many times before. It’s in no way ideal, nor suitable for every use case, but it provides an example for discussion:

Behaviours(product owner) –> unit(dev) –> code(dev) –> VCS –> unit tests(CI) –> behavioural tests(UAT) –> load tests(staging) –> security tests(staging) –> monitoring(production)

The issue I’m attempting to highlight is that as desired functionality is added by the product owner (typically as a feature in the backlog) and committed to a release candidate by the developer, so the hoops need to ensure that the new functionality is delivered as specified, and no regression bugs have been introduced. This typically involves either the new functionality going untested through load/security/monitoring, or a developer/operations person having to rewrite the new business logic repeatedly in the DSLs of whichever tools are used to create the hoops.

I’m currently using cucumber-nagios to reuse our functional behaviours for monitoring and I’ve also had some success using PushToTest with Selenium from Cucumber. I’ve yet to look at how to tackle security requirements. Please get in contact with me if you’ve had any success reusing your business logic assertions throughout your pipeline; I’d be very interested to hear about your experiences.

My (current) vision would be that, either as a part of or along with a release candidate, an artefact is created that outlines the business value assertions made about the candidate. As the candidate then moves through the pipeline, so the hoops are updated with the relevant assertions they need to test about the candidate. These could be a certain number of virtual users through the user journeys in the expected ratios, lack of injection holes in the application, the application’s ability to degrade gracefully through various failure modes, and anything else that’s required to establish confidence in the release candidate.

You should be able to make an API call to cloud-based testing providers(such as Soasta) so they can use your artefact to automatically test the behaviour of your application, such as external load and security tests. The feedback from these providers can be returned to your CI server for tracking over time, and form a basis for diagnosis of any assertion failures.

Without a fully-automated testing pipeline, I see one of two outcomes regularly at the moment:

• New functionality is not tested before release. This implies risk.
• Releases are delayed while scripts are updated, and manual load/security tests booked with vendors and executed. This inhibits business agility and delays feedback cycles.

I’m attempting to write an abstraction layer for Cucumber scenarios to be reused in load tests, if you’re interested in helping please get in touch. I’ll post it on the Githubs once I’ve got something small working.

I’ve always(well, for longer than my attention span) been a massive fan of outside-in development via BDD, in particular Cucumber. Despite this affection I’ve thought it a little unusual that BDD frameworks ignore/discard the business value proposition stated at the beginning of a feature. It’s always felt like we’re saying “that’s too difficult to do anything useful with, we’ll leave that to the business idiots”. With that in mind, and being a business idiot, I’m going to attempt to construct a model for value and automate the testing of business-based assertions.

What I’m really trying to achieve is to extend the automated feedback cycle back into the business. The sooner the decision makers gain meaningful feedback about the impact of their decisions, the sooner they can start making useful decisions (and fewer useless ones).

I think this is where the devops camp needs to join forces with the lean startup people (*dev*ops*). I’m very interested in exploring if there’s any useful mileage in attempting to automate the Deming Cycle, and if there isn’t, to examine the reasons why.

Humans are great for the PLAN (hypothesise) phase, although I’ve heard and read some interesting thoughts around AI and entropy being used to automate this (Skynet++).

Devops and the associated tooling are great for the DO phase, and the Continuous Delivery bible has had a massive impact here.

CHECK is often currently left either to chance or abstracted via some BI monstrosity. This is where I feel we need we may benefit from exploring automating assertions on business value. Most companies I work with have some capabilities around business metrics, but they’re infrequently linked back to the planning/strategic capabilities.

In theory an automated system can use some form of immunity metrics to roll back if the business assertions are not met, thus providing an element of automation around the ACT phase. I’ve seen a variety of immune systems employed around business metrics, however they often seem to be used as a tactical weapon rather than strategically to inform business decisions.

There are clear issues with attempting any kind of valid implementation, the obvious ones being around gaining useful feedback on a value proposition. From what I’ve looked at so far, automating value raises far more questions than it answers, but if we don’t push the boundaries on automating the feedback cycle then I feel we’re just locally optimising. I’ve had some really useful input from the folks at OpenCredo, who are thinking along similar (but better formulated) lines, now I’m trying to actually make this concrete.

At the moment I’m working on extending cucumber, specifically on “Lines 2-4 are unparsed text, which is expected to describe the business value of this feature”. I’d like to change this, making the business value executable (and testable) so we can assert the business value is being delivered as promised.

I’d like to change this

  Feature: Addition
    In order to avoid silly mistakes
    As a math idiot
    I want to be told the sum of two numbers

To this

  Feature: Addition
    In order to make 0 silly mistakes
    As a math idiot
    I want to be told the sum of two numbers

Or this

  Feature: Addition
    In order to make fewer silly mistakes
    As a math idiot
    I want to be told the sum of two numbers

And I want to assert that delivering this feature ensures the relative/absolute quantity of silly mistakes.

Attempting to write the code for this has made me realise how much time I’ve wasted in meetings of late rather than making myself useful, so if you’d like to lend a hand on the implementation of these ideas, please get in contact. I’ll post on this blog when I get the proof of concept up on the Githubs.

UPDATE Cucumber-value has now been dragged kicking and screaming on to Rubygems and been inserted forcibly up those Githubs.

I had the good fortune to attend Devopsdays Goteborg a few weeks back, and met a whole bunch of wonderful people. Many thanks to the legendary, words-cannot-do-this-man-justice Patrick Debois and the other organisers.

Whilst there seemed to be a lots of fantastic, enlightening conversation around the devops space, there also seemed to be a lot of complaints about lack of sponsorship for devops from those people in “the business”. I took my usual subtle, sensitive approach to a perceived problem and proposed an exploratory openspace entitled “F**k devops, noone cares, where’s my money?”. Whilst I undermined myself somewhat by turning up late for my own openspace, my ideas didn’t seem to get a huge amount of traction.

“The business case for devops” (very faint on the board)

This was probably because there was a tools-orientated openspace happening in another room.

Devops’s achilles heel is the potential it has for being introspective(ignore the ITIL/process arguments; absolute nonsense) and purely tooling focused. Changing the THEM and US attitude from dev vs ops to devops vs THE MANAGEMENT just shifts the issue. Patrick Debois brilliantly summarises that we should, instead, be thinking about *dev*ops*; unfortunately some of the devops people I’ve met seem to be more interested in teaming up against others than including them.

I’m taking a look at how I think we can address this problem, by perhaps changing the focus from “delivering” to “delivering value”. Coming from a devops background, I’m obviously going to create a tool to help me. That should teach those management types a lesson.

Tumblr seemed like an inappropriate place for my epic rants, so I’m giving Octopress a go. Like the cool kids do.