Voices in DevOps – Episode 3: A Conversation with Daniel Vincent of Pyson

Guest

Daniel joined Psyon as their first head of software development to boost growth and further develop its technology platform, Psyon Elysium. Before that Daniel Vincent joined the EMCOR Group (UK), a facilities management provider, where he was head of data quality and solutions, overseeing the creation of an in-house development capability designed to bring unique FM insights to clients. Prior to this he worked for media auditing company Ebiquity plc, initially as a developer, then as development manager, IT operations manager and latterly, UK IT Director.

Transcript

Jon Collins: Hello and welcome to this edition of Voices in DevOps, where I'm going to be speaking to Daniel Vincent, can I call you Dan, is that all right?

Daniel Vincent: Absolutely, of course you can.

I always have, so now I've known Dan for a long time largely through musical connections, but it transpires Dan that you are a head of software development for a company called Psyon and I know very little about Psyon, so maybe you can tell us a bit more about about Psyon and your role there.

Psyon is a part of Howden Employee Benefits which is part of the Hyperion Insurance Group. We've recently joined that group from the Punter Southall Group. We work with employers to help them best administer their employee benefits. This includes things like medical insurance and life and income protection and all these kind of things. So...when you join a company, these are kind of the benefits that come on the other side of your contract sheet that you get in the post.

So obviously employers spend a lot of money on this kind of stuff and using a platform such as Psyon's Elysium really allows those employers to get deep into the MI or whether they're getting the best deal on the insurances they are purchasing, but also to allow them to supply the insurers with more accurate information to keep their premiums at a manageable level.

So who sits in front of your stuff? Is it the HR department, [or] is it the employees themselves?

For our flexible benefits platform, ‘youflex’ it’s the employees themselves, so they can go in and purchase holiday or childcare vouchers or dining cards and little bits and pieces like that. They can add family members onto their insurance schemes as well, for private medical, things like that. So that's a very broad church, I think that kind of flexible benefits platform.

For our flagship product Elysium, it's really for employers. So it's the HR department especially, for them to go and be able to access their data that is shared obviously with their insurer and also to provide the insurer with the latest information so that they can keep their premiums at a manageable level and optimize the level of insurance where that's supported by the insurer.

And how many people are you, in terms of Psyon?

Psyon is a relatively small part of the business, it's around 30 people. So we have a very small development team and we work alongside corporate IT to go away and deliver our platforms out either on to ‘on prem’ data centers or on to cloud or virtualized environments.

And the question I said I would start with I didn't actually start with: How did you, yourself get into this then? Have you always been in development?

So yeah I've been involved in software development for just over 20 years, more by attrition I think rather than anything else. But it seems like a good job, so I started off as a developer doing, like most people doing little bits for building up my skills and working with more experienced hands building platforms out. I would say the first sort of 10 years of my software development experience was doing what people refer to as ‘rad rapid application development’ or... ‘we’ll build a thing; it will draw on the back of the fag [cigarette] packet and put it together. For a number of businesses that's incredibly profitable and it works really well.

And you know in the days of agile now, there are many people who kind of repackage that rad approach as ‘we're doing our job—we now just do it every two weeks.’ And as time went on, as the company I worked for at the time grew, we obviously became more mature in our approach, and then we hit basically that kind of PRINCE2 level of project organization.

So we would have a defined software project and we would kind of work out a bunch of milestones and go away and deliver that. And then in latter years you know, obviously like everybody else, I've started to adapt to using Agile methodologies to get that continuous improvement cycle into that delivery. So that has the advantages and disadvantages, and to be honest, different businesses that I've worked for at different times necessitate different approaches. ”Your mileage may vary,” -- that's probably why we're talking about it.

Yes well indeed, and the one term that we haven't used yet but don't have to, is ‘DevOps’ and we'll come back to that. But it's interesting that you link it back into the rad approaches because we've seen extreme programming. We've seen DSDM. We've seen all kinds of different manifestations of what essentially comes down to ‘anything but waterfall.’

But then what fascinates me is that you then link that back into PRINCE2. Do you see sort of structured project management approaches and Agile development methodologies as friends or enemies? Do they get on?

I think it depends on the environment you work in. We're incredibly collaborative here at Psyon and I think that's what attracted me to this role in the first place. PRINCE2 has its place because sometimes you need to plan a route from A to B and if you've got very distinct drop dead dates that you need to deliver to or whether you've got external dependencies on a plan, sometimes that doesn't quite work out with that. So... the best way to describe it is as someone said to me once: you can turn your Agile plan into a lot of very small waterfalls, I think is the way that somebody described to me once. But I think that the two can be complementary.

So in terms of in a corporate environment where we may be delivering change requests for users, those change requests have (and this is common to a number of different businesses that I've worked at) a very distinct lifecycle that's not entirely related to the development itself. It's about the costing up of whether or not it's worth doing in terms of measuring those business benefits. It's about making sure that when we deliver that new piece of functionality or that new piece of software, that there is adequate training and that the business makes the necessary changes to get that business value back. And I think sometimes with Agile, those things can be lost in the excitement of delivering a product.

There is always the danger that you're delivering product out into space and no one's really prepared to receive it; and there's not a plan to make sure that those benefits are realized once the delivery has taken place. So I think a more formal project management methodology does have a certain amount of overlap, but then other people do it in different ways, and obviously Agile is derived from Lean and there are very good methods of measuring that business benefit in other ways that you don't see when you go back to the, as I say, ‘lean manufacturing principles’ and everything else. That obviously has its place as well.

That's like that common misunderstanding about Agile stuff is that it's less formal in some way and as you hint at its roots in lean manufacturing, that the opposite has to be true, if you want to be efficient.

Yeah. And I think it's very easy and I think that's where the DevOps... to a certain extent you could say, “We're going to go and build some software, we're gonna use Agile, we'll use Jira or some other tool to or we'll do some sticky notes up on the board. We'll do it for two weeks, see where we get; here's a bunch of user stories and we can go away and do it. We can do that forever. We can go and deliver incrementally more and more functionality, but somebody external to the team... will talk maybe about, say we'll have a product manager and a product owner.

But then outside of that software development lifecycle environment, somebody has to make sure that those benefits are realized and actually that everyone gets trained and things like that. Quite often teams aren't really geared up to do that whole thing and there's that level of business engagement and so on and so forth. And quite often in software development in a very commercialized environment or a corporate environment, there are business reasons that you're solving a business problem in doing the development in the first place.

It's not all about product development in that purest sense. As we we may see when we go to say a DevOps or an Agile meetup and somebody will say “yeah we built, you know and then we added this feature and this feature.” And it's often quite mundane. It's “well we're going to expand this office, and therefore we need to increase caching of this thing” or “you know, we're going to launch a new product elsewhere in the market and we need to add this new feature” or “you know we've restructured a team and we need a new dropdown [that] says which team is going to handle this particular piece of work.”

So many things don't have the same level of excitement or interest that you might see in some more traditional "agile environments." So I think DevOps, to a certain extent (as we'll talk about I guess in a minute), is often the glue that kind of binds a lot of the different disparate elements together, because in DevOps (in my mind) -- that's really around process automation and actually building a dependable and scalable workflow that your whole Agile software development lifecycle sits in.

Hold that thought, and I think you've set things up very well for the second half of this podcast. I've just been thinking... and I've written some notes here [about] a conversation I had a few months ago with an old colleague, Michael Göthe who is now pivotal. He said to me, ”The kids today—they don't even know they're doing DevOps.” I paraphrase but it's words to that effect. And I think we do have these environments and I've just written down five bullets so I'll list them out here, bear with me. It is software that's more consumer oriented, it's definitely going to be targeted in the cloud platform, tends to be more startup type organizations, tends to be more collaborative environments possibly with people who haven't... I've written down ‘younger’ but they don't have to be younger, just not necessarily familiar with old fashioned ways of doing things and they're just doing the DevOps thing as a kind of a natural approach because that's all they've ever known, and one of the things I've seen and you talk about meets ups.

But I've also seen it with DevOps vendors and so on that tended to preach to those converted when they go into environments where all of those things aren't true. That's where things slow down considerably and so hence the backdrop to this podcast around enterprise and scaling DevOps into the enterprise or DevOps in the enterprise and just removing the friction.

That was a long preamble, but do you see your organization as one where it's almost like you're bridging the old and the new, because you're doing the newer stuff but you've still got the background, it's an older domain, you're working with on premise IT systems and so on?

I would actually say that I don't think I've ever worked in an environment to be honest where all of those things are true or many of those things are true. My roles have always been within larger corporate or enterprise environments with mixed or hybrid environments, so part cloud, part on prem you know and scaling out from... over the years, from a server under somebody's desk to large data centers and then to the cloud.

I think DevOps as a mindset is really around that kind of constant measurement and that drive to constantly improve, and to make sure that at each step you can measure the success and you can carry on into the next step and so on. So to take away all of the exciting things that people associate DevOps with: it is basically the process of delivering something out in a semi automated or automated consistent fashion that you can measure and that you can control. And I think with you know, doing it with Agile—what you have is you have a really good framework for developing software and measuring your progress as you build being able to perhaps change course when you need to, but it doesn't really cover the concept of change in release of having some kind of change advisory board.

You have a product owner who in the sort of archetypal Agile setup is the kind of the ‘be all and end all’ and that's great within a smaller business, but when you have multiple stakeholders, you do need to have an auditable process that you can then show to people and go, ‘and that's why this dropdown is in this form, on this page’ because in certain enterprise environments, that is very, very important because some of the things that you're developing will be for compliance reasons, or in order to do a change to a live environment.

There are compliance considerations: does it need to go through security or penetration testing or other elements like that. So having a DevOps framework in place really does...it accelerates I think a lot of that. It certainly makes people more empowered to be able to make all of the decisions that need to be made within their domain. It also then joins up all of those different roles and responsibilities in all those different domains to allow people to safely deliver a change to a piece of software. Or if we're talking about infrastructure as code, then obviously you're making a change to an environment, but you're safely delivering that change from the person who's making it to a person who's checking it, to a person who's authorizing it, and all of that stuff can be audited at the end because you have a scalable and repeatable process.

I was thinking as you were speaking: the kind of nub of it is the whole CICD thing, the continuous development, continuous integration element. But actually what you're talking about is development integration and deployment and the ‘continuous’ [part]... is the proof point that you're getting it right. It's the proof point that you've got the process automated and you are able to do something consistently and repeatedly.

Absolutely. And part of DevOps obviously, is being able to measure your environment and measure performance to make sure that once your thing is embedded in... let's assume that we all coded our bugs for a moment. But once your change is live, then you're also monitoring the performance of your software platform or your infrastructure platform to ensure that there are no long term adverse effects to performance or no unforeseen [glitches, like] “oh my goodness, you know we didn't open up this port and this thing that's meant to be reading us back telemetry doesn't do it correctly in live.” You know those kind of things, so having a DevOps process, DevOps method, and following a DevOps methodology, using the tooling and getting everybody on board with that process and getting everybody's ‘buy in’ on that process and on the tooling and making sure everybody understands how they fit into that process I think is highly important.

And I think sometimes when people talk about Agile, [or] they talk about DevOps, it's really simple to do it on paper, or it's as complicated as you want to do it on paper, but when it comes to the practicalities of doing it, sometimes people forget that the world exists outside of the software development team, or the DevOps cloud team or whatever.

There are stakeholders out there in the business who need to know when this particular change is going to be made. And they also want to know when the auditor comes in and they say “Well can you talk me through the last six releases that you did?” You know sort of looking at your fingers and going: well it happens every Friday and it just happens on its own, is probably not going to cut it, so being able to... [use] your software tooling or whatever wherever it is, where these and whether you use Octopus or Azure as your DevOps or whatever, then you've got to be able to understand that platform enough that you can say, Well here you go: here's the log, these are the changes that took place, these are the database migrations that we did and so on and so forth. And also it's for the odd reason.

Having a good DevOps pipeline means that you can just put out another... you can rebuild your environment and hopefully integrate your backup and restore technology in there as well. So I think without going super ‘into the weeds’ about the technology part, it's about having that collaboration across multidisciplinary teams inside and outside of IT. It's having the communication out to people understanding what it is you're doing, and then being able to do that and be able to make it scalable and repeatable is what really makes people have confidence in what you're doing, and they're more likely to give you the right push, I think, to make your changes that you want to make.

I'm really picking up on the word ‘repeatable’ and something you're saying reminds me of the old military term, when we had “Harry over there, he talks a good project.” We can all talk a good DevOps, but actually make making it happen in practice is the trick.

So a question for you: you said that you're obviously responsible for the development side of things, and then you work with the internal IT teams who are clearly also going to be doing the Ops side of things. How does that relationship work and how does that tie into these notions of repeatability and scaling and so on?

So from a pure developer point of view, we had obviously set up our CICD pipelines we feed in, automated testing and everything else, I think within the sandbox that any IT department would provide (in the last few years I've worked in software development both inside and outside of IT which has been somewhat different), but if for the brief moment I could refer to IT as like I guess ‘the infrastructure people’ and the software developers as ‘the software people.’ But that obviously varies depending on - that's largely where we are.

The infrastructure people are very process led if it's a regulated environment or a client has specific security consideration, then that process absolutely needs to be 100% auditable, and they will have their own change process, whether it's swapping out a hard drive, whether it's ferrying data in and out of a secure area, taking in new hardware or any of those kind of things. Because remember we're not all in the cloud and hybrid and tin environments still exist. Those guys are not going to want to put out any pieces of software. They also don't really like getting phoned up in the middle of the night when it doesn't work. So they want some assurances that not only does it work because it passes all your unit tests, and because your QA guy’s looked at it, they want a way of putting it in and backing it back out again.

So what DevOps, I think and having those really strong and clearly defined pipelines should mean for those infrastructure guys is that your pipeline should take you up to the edge of your IT environment, so you'll have a dev environment, you'll do some new IT testing in your new IT environment, all automated with probably internal gateways in terms of approvals, and then you'll go out to your infrastructure team or your (as you say) ‘onsite IT team’ and at that point there, you stop.

Now with having a good DevOps process and having the right automation tools in place, that probably means that somebody somewhere, [like] the change in release manager is gonna get an email to say “Hey you know there's this new release. It's all packaged up ready to go; this is what's in it; these are the user stories or features that have been completed as part of this release. These are the bugs that have been fixed. This is why effectively, this is why we're releasing this piece of code or this piece of configuration” or whatever. So at that point there, the value is the already... rather than kind of going up to the guy's desk and going, “Can you not do some code tonight, I've got some bugs to fix,” they're getting something that they can then pretty much cut and paste into their meeting notes for the next change audit. That's the process, they can present that to whoever the decision maker is or the security team, to have some kind of decision point as to whether or not they need to do any kind of further testing, software testing or whatever, will come back and ask any questions.

Once they're happy with it, they can approve it and the whole process should just then roll on and (a) they don't have to do a whole bunch of work, you know they have to copy all the files on and install this and do this do that, do the other. And also if it doesn't work, God forbid there should be... as part of your pipeline, there should be a clear way to revert back either by a backup automation or migration that will basically roll all your previous changes back.

I think as someone who has worked on both sides in terms of pushing for development, but also managing an IT operations team, I know what it's like to be cautious, and I also know what it's like when you're desperate to get something done. I think DevOps, going back to my point I made earlier on is the glue that takes the disparate parts of Agile and... I'm not going to say ‘rapid application development,’ but that kind of need to push forward constantly. It's the glue that kind of sticks that to the more sensible, structured and more cautious approach. And basically DevOps, if it's done well, I think it is the pragmatic approach and I think it blends the two, if you're doing it right.

So I've got a question for you, but before I ask it, I'm going to play the analyst. I haven't always been an analyst you know. he said, it just reminds me back in the day when I was actually doing DSTM consultancy, I was working for an insurance company, it was Royal London in Colchester, and I was in that very meeting where the developers and the ops people were there and it's exactly as you said, no person on the operational side wants to say “You can't have it, you can't have it, you can't have it.” But at the same time they've got all their risks to manage and they don't want flaky software coming in and breaking everything. So it's just that feeling of disappointment when you realize you're being asked to... get something thrown over the wall... and it's not going to be properly tested and it's probably not going to work as well as it should. And just so that you know, I've been in that meeting, so I hear you. I understand what you're saying.

On the question fronts, you're working very much in a more regulated environment. Well I guess all environments are regulated, anything that touches customers, but you are in insurance. How much of these things do you think apply even more because of the environment you're in, and how many of the things you're saying are just general rules [that] always apply?

To be honest there are obviously regulatory considerations. I think when you work in a financial or insurance environment. In my previous role I was working in facilities management where you're working with a lot of sensitive location data or because you're working for some huge clients and then you have a lot of their information to handle as well.

It would be easy to say, “well yeah if it's insurance or if it's banking, you should definitely take more care.” But the reality is: no, you should take that level of care no matter what you do. And I think it's very easy to cobble it together and not knock it out, f--- off early as they say. But in reality, the same rules apply across all industries. The difference is that in more regulated environments, it's expected by the management and so on. However, if it's not a regulated environment, it should be expected by you as the person who’s delivering it. I think that's the way, if you're going to go out and kind of be an evangelist, for want of a better word, of DevOps, then you've got to have that mindset no matter what software you're developing.

I think it’s really interestingly [that] what you're saying as well is (I'll flip it back against a lot of the conversations I have are about developers are about Ops people, engineers, they're about the people working at the cold face of technology)... And a lot of the things you're saying are at the decision making level, the risk registers, and the project management training questions and deployment cutover times and all those sorts of things, which are at the level above. I think a lot of the conversation we're having at the moment around DevOps are importantly at the lower level, but they're not necessarily taking into account all of these things at the governance level.

Absolutely, and well governed projects deliver better than ones that aren't, and anyone who will ‘tut’ and say “but that’s not Agile or that’s not…” Well maybe it’s not, but maybe if you're not being aware of the environment you're in, you're not aware of the risks that having a website with people's data on... We've all just been through the process I'm sure in all walks of life where we're putting cookie notices on because of those regulations and things like that, but people who are doing it are like, ‘ahh, it's annoying isn't it?’ But when there's a data breach or when there's some exploit or whatever, we're all quite quick to point the finger, but you know good hygiene starts at home as they say.

I think the more that we're mindful of those things, from top to bottom... it's my job in my role in software development, is to give people confidence we're doing the right thing. At the end of the day, we've all been in a situation over the years, where you pull a release because you're just not happy with it. I don't care if it's going to be late, we're not putting that out because sometimes the risk of doing something is greater than the risk of not doing it. It's important I think and DevOps really gives us the opportunity to deliver that scalable, unrepeatable, understandable and auditable process, and the automation elements of it take away human error. They also add in the opportunity to take in those workflow elements in terms of authorization and review, which is the only thing we haven't really covered: the element of (you mentioned right at the top of the show), extreme programming and things like that.

But having a sense of peer review, having a sense of sort of collective responsibility for the piece of software or code that you're delivering, or the functionality you're delivering, I think it's really important, and it is good to see the younger people who are sort of still cutting their teeth really in terms of all the technology, all the new frameworks, and Microsoft’s ever changing world of dot net. To see them really early on, really grasp those principles that ‘I understand what I'm doing, and I understand that if I check in this piece of code and I tag it with the work item I'm working on, it means that when we release it in two weeks time, you can attribute every single line of code in that delivery back to where we started doing it, and why we were doing it,’ that's great, that's the tagging the cattle I think really.

That's fantastic. Well we've got less than a minute to run, so I was going to ask you for your last thoughts, but I think we're even way out of time for that, so I'm just going to say thank you very much Dan Vincent from Psyon, for all your thoughts and I will leave the last word to you if you want to just say thank you and anything you'd like to leave people with, you've got twenty seconds to do it.

Process, process, process, no. DevOps is good, do it.

Fantastic, okay cheers Dan, great to speak to you.

Thank you.