Whois Asher Bond?

Question: Whoami?

Answer: My name is Asher Bond and I am a Software Designer focused on solving problems of unpredictable scale. I entered the Technology field in 1995 as a business Internet access solution provider. These days I research distributed and autonomous computing systems and I combine today's practical knowledge from the field in order to continuously discover optimal, direct, hands-on approaches to software and systems engineering disciplines which I call Test-Driven DevOps Design.

Question: How do you define DevOps Engineering?

Answer: A DevOps engineer is someone with practical knowledge, discipline, and dedication around the areas of expertise which facilitate development, testing, and ongoing operations of quality software, systems, and reliable services. DevOps engineering requires comprehensive knowledge of the culture, process and tools used for producing quality software and then taking responsibilities of gracefully delivering the software at minimum or managing end-to-end software delivery such as user-facing operational service endpoints or software products which users depend on.

Question: Isn't a DevOps Engineer the new name for IT or systems administrator?

Answer: No. Not that it makes any difference who you complain to if there is a problem. If you're hiring someone to solve a problem, you might want to be a little more precise in managing their functions, even though at a high level they are taking on many of or potentially all of these responsibilities. If you are managing an engineer you should expect different output (machines and code, for example); not really the same as what you would expect as output if you are managing an administrator (who produces little output such as reports and access credentials and who more importantly keeps things like systems and services running smoothly). Administration, in a pure sense, does not involve re-engineering software nor systems. An administrator simply uses existing software to set up and manage systems. A DevOps engineer can't get away with simply using things off the shelf.

If you confuse a DevOps engineer with a systems administrator or IT manager, then you might also confuse an ATM machine maker with a customer service representative for a bank. If the teller fails and it's automated, you complain to the ATM machine maker. If the teller fails and they are a customer service representative for a bank, you go to their manager. So from a high level perspective DevOps engineers are wearing a systems engineering hat in order to take on some of the functions of infrastructure management and systems administration in order to automate them.

It's worth mentioning that a systems engineer does much more than basic systems administration. A DevOps engineer is a software engineer who is making a machine out of what was historically done by hand, but the DevOps engineer should know (and document for the engineering team) how these systems work inside and out as though it theoretically could be done by hand if you had enough hands to do it. If you think that a systems engineer is a systems administrator, then you should further examine the differences between administration and engineering.

Instead of thinking of a DevOps engineer as a glorified infrastructure manager or an IT person for a software engineering team, think of a DevOps engineer as a software engineer's software engineer who does backend engineering plus some systems engineering and who should be familiar with site-reliability engineering if reliability is a concern. Reliability should be a concern if you are working with something other than a prototype. Depending on the prototype, some software engineers might even argue a case for paying some attention to reliability concerns just to build a prototype. Part of reliability is security of applications, systems, networks, and services. Some DevOps engineers also take on release engineering responsibilities. A DevOps engineer should at least work very closely with release engineers. If the software relies on networks then the DevOps Engineer should have a basic understanding of network engineering, such as how to make applications, systems, and services fault tolerant around the expected failures of networks, especially if the network is connected to the internet. If there is a separate testing team, then the DevOps engineer should work closely with that team. Testing is the only actual real way to achieve quality control in DevOps Engineering.

Question: How is a Test-Driven DevOps Design approach different from ordinary DevOps Engineering?

Answer: Taking a more direct-path to the software Development lifecycle according to the user-experience requirements, then testing the results after trying many different things quickly is what fundamentally differentiates the Test-Driven DevOps Design approach from other DevOps approaches which are often compatible. This is not to be confused with "hacking it until it works" which is an approach that is entirely agnostic of the design process as it pertains to developer or user experience concerns. Service-classes and priorities based on both functional and non-functional requirements are combined to form a Design pattern. The pattern can be delivered or polished into a Blueprint and the blueprint can be delivered.

Patterns and blueprints are Templates which can be rinsed and repeated to produce output at a highly efficient rate. Time is managed in a Kanban style, where many tasks are prioritized according to user experience and business requirements, for example. Technical capabilities which improve the user-experience or business/engineering efficiency are measured as milestones in every iteration to measure productivity based on technological progress primarily. Efficiency is also measured. In a minimal DevOps approach only velocity is measured, but in Test-driven DevOps Design velocity is only considered as an expected result of ongoing efficiency without losing focus on quality.

Quality is measured by the passing of tests which consider the user-experience. In DevOps engineering as a discipline, failure is expected, but an approach making use of Test-driven DevOps Design strongly encourages a more proactive, early discovery and remediation of failures such as bugs and operational mistakes in environments which end-users should never experience if software delivery gates of testing are properly managed. Being able to repeat results is essential for these proper tests to be conducted in a proper regression from development, to staging, and operations environments, for example.

Thus, Test-Driven DevOps design is far stricter than other DevOps engineering disciplines when focusing on the topic of enforcing parity between Development, Staging, and User-facing production operational environments. Due to the increased focus on testing there is often the use of user simulation or client-side test routines which would functionally test a system or service or other running software implementation. Automation of client side tests is common in Test-Driven DevOps Design because it facilitates horizontally scalable tests which may be used to expose performance bottlenecks by producing stress against the running software being tested. The same principle of implementation testing is applied not only as it is required for performance testing but also for security testing for the purposes of DevOpSec and security-oriented penetration testing. Identity, access management, user management, and attack vector minimization concerns are all considered to be extremely important implementation details which could never be ignored in operating any user-facing system of decent quality.

Question: What are the similarities and differences between Test-Driven DevOps Design and Test-Driven Development?

Answer: Test-Driven Development is focused on testing the quality and improving the structure of a software developer's code units. Behaviour-Driven Development emerged from Test-Driven Development by putting a new focus on the user-experience. Test-Driven DevOps Design evolved from Behaviour-Driven Development by putting a new focus on unifying design, implementation, and long-term operational concerns from cradle to grave without necessarily imposing that the entire implementation of one or more systems must be shoe-horned into one programming language. Instead of focusing on test simpllicity or a programming language, Test-Driven DevOps Design focuses on what is required to create, test, and operate high quality software without interrupting a high quality user experience.

The most important difference between Test-Driven DevOps Design and Test-Driven Development is that Test-Driven Development facilitates simplicity of design; where conversely, Test-Driven DevOps Design facilitates the evolution of scalable design patterns which improve continuously as the design process continues to convey user experience requirements and the implementation of their design constraints being tested. Additionally, testers participate in a lengthier design discussion by providing feedback which validates the constraints of the initial design plus any modifications which may be introduced at any time. In summary, you can always go back to the drawing board (without shame).

Any development which is derived from visiting or revisiting the drawing board is validated by testing. Test-Driven Development encourages simplicity and readability of tests. Readability is required and test simplicity can also be helpful in a Test-Driven DevOps Design approach which expects tests to change as software behaviours change, but simplicity of one test does not guarantee simplicity of an entire system or set of systems behind one or more service endpoints which must be considered in order to achieve thoroughly tested software implementation.

There is no shame in constructing a system which is only as complex as needed to produce capabilities for the user, if the system's design, architecture and implementation details can be known by others through straightforward, efficient means of knowledge transfer. The motivations for greater investment in a well tested, more comprehensive design pattern include quality, scalability, and long-term efficiency when considering the work involved with both design and implementation.

Test-Driven Development solves the problem of changing requirements by cutting the design process short, but Test-Driven DevOps design instead reduces the number of attempts to implement or reimplement a feature which would be likely inadequate for a user experience which is more demanding than the use of a simple prototype for the purposes of proving a concept. When going back to the drawing board, the process of Test-Driven DevOps design is fundamentally different from traditional Test-Driven development, which explicitly avoids testing implementation details which are essential to operational responsibilities DevOps engineers should be concerned with if they are to produce and deliver quality software implementations. Test Driven Development ignores or avoids precise behavioural testing of software, which is essential to validation of scalable, robust software deployment and software lifecycle management. Traditional Test-Driven Development sacrifices cleanliness in running, user-facing software implementation for cleanliness in the structures used to produce the code. Therefore; in order to unify (instead of separate) concerns of User Experience Design, Development, and robust, scalable user-friendly operations; we must encourage and facilitate going back to the drawing board in an efficient manner.

Questions: Is there a Lean version? How do I simply prove the concept initially?

Answers: Test Driven-Devops Design is not necessarily lean, but if delivery of a lean prototype is the goal, then straighforwardly design a minimal prototype and go back to the drawing board when the prototype is accepted or rejected. Unification of design and development concerns, as opposed to separation of these concerns, facilitates a more productive and efficient prototyping process. Reducing the friction between design and development is more productive and efficient, without bothering to pay the cost of micromanaging narrow measurements into only coding velocity, for example.

Question: Wouldn't Continuous Integration be slower due to the slower velocity of implementation testing as a requirement?

Answer: Perhaps velocity would be slower in the first iteration or two as comprehensive test coverage is developed. If you focus on the first word in the term "Continuous Integration" and develop repeatable tests which are inexpensive to maintain, then focus on the second word in the term "Continuous Integration" by building features and tests which integrate well with existing features and tests, then you have embarked upon a much more effiencient and long-term performant continuous integration path.

In Test-Driven DevOps Design, any part of a system in a continuous integration pipeline is tested; therefore, moving parts only move as intended. Change control must be systematically manageable by a version control system. Version Control, Configuration Management, Automation, Provisioning, and Change control are all critical responsibilities of the DevOps Engineering team. Changes happen in a controlled manner as the user experience is contrived and polished by software engineers and their team of user experience experts. This reduces time spent debugging changes which may come as a surprise. Complexity is further reduced by constraints which keep other systems which serve interests such as continuous deployment or continuous delivery well integrated with the continuous integration, build, and release pipeline. Development velocity and more importantly general engineering momentum is built as a prioritized cooperative effort by DevOps Engineers and other software engineers because feature code, testing code, and automation scripts are well-tested, well-integrated and made repeatable and therefore more manageable via a version control system. When a test fails, the question of "what changed" can be trivially answered even by someone without expertise in this more efficient approach.

Some teams have even constructed their own continous integration, build, release, and deployment pipeline entirely in-house using a Test-Driven DevOps Design approach. Other teams use customizable frameworks with heavily customized configurations, but these teams try to minimize the complexity of configurations and these teams test and control software updates in order to keep feature integration in step with testing and deployment automation software which may not be developed in house even though changes are controlled by a DevOps engineering team.

Question: If DevOps is about the culture, process, and tools, then how can you describe culture surrounding participants in Test Driven DevOps Design?

Answer: Culture and its quality is best described by the capability for an engineering team to work together to solve problems. Take the problem of efficiently producing quality code which behaves well for the user, for example. In Test-Driven DevOps Design, undesireably clever shortcuts are exposed prior to staging as code visibility is prioritized when engineering teams properly invest time in code reviews. The expensive ongoing debate of "what code looks good" vs "what works" is made moot and can be eliminated when "what works" is properly tested as an implemented behavior prior to code review. Due to the emphasis on comprehensive testing and requirement of comprehensive code reviews, there is no such thing as "hacking it until it works" in a Test-Driven DevOps Design approach. In teams where engineers have a track-record of strong collaboration, developers who move more quickly than others can perhaps be rewarded assuming that their quickly produced work is actually more efficient engineering. Efficiency in this approach is easily identifyable and difficult to confuse with excessively hasty code output. Strong collaboration when made essential is measureable, efficient, and empowering for both developer engineers, operations engineers, and engineering managers who are key stakeholders in the teams which have embarked upon a Test-Driven DevOps Design approach. The engineering manager, on behalf of the greater pool of end-users, is a power-user who provides expertise by painting a clear picture of the greater user-experience and how it pertains to internal functions while features are being developed, tested, staged into production operations, and maintained as running services. The DevOps engineering team is always informed of the engineering manager's agenda and reports system functionality which is usually summarized as services or service components. The purpose of these reports is to provide insights and visibility to the engineering manager with a good signal to noise ratio.

The second-best way to describe culture and its quality is by the capability for an engineering team to communicate. Technically, good communcation is also a key part of collaborative problem-solving. Although it's obviously of general importance for engineers to not loathe each other's company, good communication is measured in more ways than counting the number of lunches the team has together or the number of people who have lunch with each other. Good communication happens with a high signal to noise ratio. Which suits the interests of engineering managers who are familiar with the limitations of engineering time as a resource. Similar to some blogs and social media posts, the best communication can often be in the code's comments. With the right DevOps engineering culture, documentation is no more of a chore than maintaining comments when the code and its structure is not self-explanatory.

Documentation should never be out of sync with code which runs in a user-facing environment. This is enforced by a code reviewer who refrains from guessing too much. Consulting the author of software is often the most efficient knowledge transfer for good engineers, but consultation should never be the only option available to a good engineer. Therefore, documentation should be done as close to the code as possible and should be updated whenever the code changes. Source code should be available to software engineers unless there is a very good reason for it not to be. The ratio of comments to code should be directly proportional to the degree to which the code is esoteric. This balanced consideration of how and where to document code helps tremendously to facilitate a process which avoids the expensive inefficiencies of producing documentation as a daunting separate task which runs the risk of becoming a task of forgotten priority. Documentation being required to live nearest the code being executed helps even more tremendously to facilitate a process which avoids the expensive error of maintainability becoming an afterthought.

As a bonus side-effect, comments which appear next to the code change at a time more close to the time in which changes to the executable code is pushed or commited; some engineers may choose to commit or at least push code and its comments at the same time, in one fell swoop. If comments appear more verbosely in the source code files and less verbosely in the version control commit messages, then there is less dependence upon the lifespan of one specific implementation of a version control system. Good documentation (whether it be comments, readme, or richer texts) should generally be committed to the same repository as the source code in order to keep text meant only for humans in sync with the code meant for execution by machines. Where possible, the presence of comments should not be required in order for engineers to make sense of executable code. Instructions to machines should generally also be informative to human engineers. Executability should not preclude readability when a writing code.

The truth should always originate centrally from the version control system. For better or worse, the version control system and its use may be a source of truth regarding engineering culture. A culture which revolves mostly around advanced knowledge or implementation of one specific version control system is probably not diverse enough to properly respect and include feedback from operations experts, systems experts, and other non-developers from various engineering fields. Programmers who code to the strengths of a specific language's idioms should be encouraged as long as they are able to document the correctness of a particular programmatic idiom which may not appear obvious to other engineers on the team. Idiomatically correct programming should never be esoteric engineering. Some would argue that if a program is truly idiomatic, then it must be succinct; therefore, it must be both brief and communicate clearly. This is true, which means that not communicating clearly is not idiomatically correct. But it's also true that in fact it is possible for idiomatic, briefly written code to be left uncommented; and if it's not clearly expressed to a human, for example, then it's both idiomatically correct from a programming perspective and esoteric from an engineering perspective. What's worse is when an idiomatic program performs an idiodic behaviour at runtime (in regards to the user experience or use of system resources, for example). In order to promote good communication with engineers in the team, the quality of communication and code behaviour at runtime should go far beyond the minimum standards such as a programmer's idiomatic correctness. Good comments help prove that communication is intentional from an advanced programmer to other engineers who may be working to improve a newer grasp on a specific language's idioms. These are the engineers who should participate in a collaboration which facilitates and welcomes improvements to the quality of software structure and more importantly a sustainable evolution of good runtime behaviours.

Question: What are the tools / frameworks used in Test-Driven DevOps Design?

Response: In Test-Driven DevOps Design it is acknowledged and enforced that the Engineer makes the tools. The tools can't make the engineer. The tool, for better or worse, is an extension of its user. Therefore, it is essential that any tools or frameworks which are procured from outside sources be understood by engineers using them. If the tool is not simple, then its use should be justified economically. In justifying the use of a pre-existing tool some very important considerations include how much it costs to maintain an understanding of the tool. Who controls how the tool or framework changes over time? How can these changes be tested efficiently? No tool made nor bought should be mysterious, majickal nor arcane like a black box.. even for a short time. Those who are capable of using excessively sophisticated tools have perhaps proven their intelligence, but perhaps they may also prove their ignorance of economics as well as their negligent refusal to embrace the discipline of general engineering principles. Software engineering, when done right, never conflicts with the greater, more general discipline of engineering. The same goes for DevOps engineering as a type of software engineering which is concerned with both software and its systems. All engineers must have a clear understanding of using the right tool for the job. This is an understanding which goes much deeper than mere following of a tool or its popularity. The entire software engineering team should participate in procuring or deciding to make the right tools for the job when the tools are part of development. If there are many engineering teams then they should work together to arrive at a properly considered decision about which set of tools is right for the jobs.DevOps engineers should control the changes of any tools introduced into the build and release pipeline. Most businesses, technical or not have the capability of packaging their own product. So if you choose to use a package manager that someone outside your team made or maintains, A DevOps engineer should think about how these outside packages are going to be packaged along with your software. All tools must be tested.

Question: In Test-Driven DevOps Design, How do you expect a team of software engineers to make tools?

Answer: The tools are created in the interim as features are delivered in order to produce the features. The tools should never be overly complex to learn to use and should never be overly complex to maintain.

Generally if you can take this approach, tools and their maintenance will actually be a bonus output with each feature delivery iteration, but other tools which already exist may be tested against interoperability constraints in the Design process.

In the process of producing a feature, a DevOps engineer who is making use of a Test-Driven DevOps design approach will involve stakeholders such as product managers or other representatives of the user experience such as alpha/beta testers; plus, of course, software programmers, perhaps some systems engineers.

This group will produce a list of requirements for a user experience, or minimally changes to an existing user experience. The DevOps engineer using a Test Driven DevOps Design approach will keep the feedback loop open at all times and may produce interim deliverables prior to the staging of the main deliverable which is the feature required to produce the desired user experience. These interim deliverables include, for example, automated tests, provisioning scripts, re-usable common code, etc. The interim deliverables may or may not be required by a project or product manager, but they are produced in a manner which is always justified by the interests of the user-experience.

The DevOps engineers involved in involving stakeholders take responsibility for any resources used to produce deliverables along the way to successfully staging and operating quality software; therefore, as part of the Test-Driven DevOps Design discipline, the engineer is continuously attentive to feedback loops of both systems, users, and developers... considering the probability of requirements changing as user demands or business demands may change, for example. Some examples of such tools which are produced as interim deliverables could be monitoring tools used in the ongoing operation of the software and its systems as a service, or tests such as functional tests or stress tests executed before the software reaches a production operational environment where a more professional degree of software maturity and quality is expected. Other examples may include profiling tools which provide insights and introspection into a software runtime. Other examples of tools delivered in the interim may include automated provisioning tools which should never obfuscate software management tasks and should minimize the opportunity for human error or malice while also providing some efficiency in the delivery process. Similarly to Agile methodologies, the DevOps engineer who embraces Test-Driven DevOps design approaches may produce interim deliverables such as reports to management or other stakeholders who represent the interests of the user or the interests of those sponsoring the work. Additionally a DevOps engineer should assist developers in discovering any errors in their programmatic ways which may be viewed as bugs such as segmentation faults, for example, where programming logic has become wild and untamed in such a way that it begins allocating improper areas of memory. Introspection and profiling reports and tests are interim deliverables which are produced on behalf of developers in order to support them in their development efforts. This also supports operations engineers who may be responsible for infrastructure or site reliability.

Question: How involved should a DevOps Engineer be in infrastructure operations and site reliability?

Answer: A DevOps engineer who has a comprehensive understanding of DevOps Engineering as a discipline should certainly be as involved as possible in not only collecting data, information, and insights from the production operations environment, but also very involved in the automation of repetitive work in the production environment. This is for the purposes of scalability and efficiency as well as to improve the quality of the production operations environment by greatly reducing the opportunity for a human error or malicious use by a system vulnerability exploiter to become a service failure or service interruption which may be witnessed by a user. In Test-Driven DevOps Design, operational concerns should be a key part of the design process in order to ensure scalable quality software operations. Site reliability concerns which are voiced, for example, can be critical information used to influence the design and architecture of a robust systems and services which make use of self-healing techniques, high-availability architectures, or disaster recovery mechanisms.

Conversely, to give an example of what could happen without input from operations: Ongoing tasks regarding administration of a data persistence system may be troubled by poorly designed data structures which lacked consideration that could have been provided by a database operator had that person been involved in the ongoing design process. In this scenario the data persistence would become very expensive and eventually reach a physical limit long before the application reaches a limit. Efficiency could grind to a halt and an unmanageable backlog of data transactions could cause the entire application to fail. In a Test Driven DevOps Design approach, none of these problems could happen because it is forbidden to ignore feedback from systems in operation and the people responsible for operating these systems.

Wherever there are site-reliability concerns, at minimum, a DevOps engineer should be involved in sharing these ongoing operational responsibilites (and probably the pager duties). If the team decides to use infrastructure-as-a-service, such as an Elastic Compute Cloud like Amazon Web Services EC2, then the DevOps Engineer should participate as much as possible in producing solutions to the likely failure scenarios where these infrastructure services are unavailable. High availablility architectures should be produced by the collaborative efforts of a site-reliability engineer and the DevOps engineers who are also charged with providing site-reliability. To ignore the probability of a failure scenario when service providers recommend that you architect your application for failure is to live in a fantasy world which is not ready for prime time. If you are hosting your own infrastructure or relying on managed services, then the DevOps engineer should coordinate testing which validates that the application can continue to be available and usable when some of the underlying systems fail. If your team is using physical machines then a robust application and its services would be available and usable during a hard drive failure, network failure, power supply failure, etc. Furthermore, there would be alerts of the system failure when the service remains usable. This is the right way to do it and anything less is asking for trouble eventually. The DevOps engineer is responsible for site reliability and service availability when underlying systems infrastructure inevitably fails. If you are subscribing to Infrastructure-as-a-Service, the DevOps Engineer should read the service-level agreements and understand that failure is likely. The DevOps engineer should drive the process of architecting your application for reacting to system failure scenarios. Get real. Get redundancy. Get coordination. Get monitoring and alerts. Get the DevOps engineer in charge to take charge of pushing these initiatives starting with the design and architecture of the systems and service components.

Question: What are the 4 fundamental concerns of Test Driven DevOps Design?

Answer: USIR...

  1. Usability
  2. Scalability
  3. Interoperability
  4. Repeatability