The 6 Principles

Scalable ServiceNow instance management encompasses these core principles:

Principle 1 - Go Native

ServiceNow is the most robust platform for managing IT, developing enterprise applications, and running your business. The Now platform provides powerful capabilities beyond simply being a system of record. The Now platform provides the infrastructure, automation, workflows, and codebase upon which you build your apps, features, and customizations.

Since you are comfortable relying on the platform to provide business solutions quickly, you should also become comfortable leveraging the platform to support delivering those exact solutions.

Many companies look to “standard” CI/CD tool integrations to help accelerate their delivery; however, CI/CD tools were built to manage, merge and migrate large collections of text files (aka source code). Your development efforts involve the Now platform, not source code files.

You need to redirect your strategic thinking from tool integrations to instance management. Another reason integrating external CI/CD tools into ServiceNow fails is that they require your ServiceNow team to learn and master a set of tools entirely foreign to their day-to-day work experience, which increases cognitive load and mental fatigue. Instance management focuses on providing real-time visibility and mobility of native components across all instances, immutable release plans instead of spreadsheets, and enforcing a level of governance that reigns in privilege sprawl.

The Now platform provides powerful capabilities beyond simply being a system of record.

Principle 2 - Visibility

Being able to track the end-to-end pipeline and user story data from ideation to production release is just part of the visibility needed to see the big picture successfully. Despite being able to track user stories, today's enterprises need help to ensure that a planned release set of artefacts (also known as a Bill or Materials or BOM) is integrated into and synchronized with the release tickets and plan.

Manually using spreadsheets and documents is error-prone, laborious, and does not provide cross-instance visibility nor a traceable and auditable record for compliance concerns. Your teams need a type of visibility that gives them the situational awareness and context to unblock bottlenecks and make appropriate adaptations to the release plan. Having real-time 360° multi-instance visibility gives you the confidence that what is planned is what is delivered.

Am I Lacking Transformative Visibility?

The principle of Visibility is defined as the extent to which an organization can accurately predict future release payloads and confidently know the status and location of all release BOM artifacts. When ServiceNow teams face problems, a lack of pervasive visibility is often the culprit. Here are three signs you might need better visibility.

Manual Methods

ServiceNow releases include many assets, such as update sets, scripts, data, XML files, store apps, plugins, plugin activations, and more. Manually tracking a ServiceNow release BOM is error-prone and laborious. Manual effort not only inserts risk into your production deployments but also inserts lag into your Mean Time to Resolution (MTTR) when resolving production errors.

Plan Vs Actual

A false positive can result when your plan states XYZ was supposed to be deployed, but you actually deployed XZ. The disconnect between planned BOM and actual release payload is an outcome of the previous three signs. This blindspot represents a debt that will manifest as errors, conflicts, and instance drift. Instance drift drives unnecessarily elongated releases, frequent cloning exercises and perpetuates error-prone deployments.

Cowboy Coding

When bugs are found developer may just log into a QA instance and fix issues directly on the target creating an almost certain collision later in the process and contributing to instance drift

Last Update Set In Wins

Developers' work often overlaps each other and is done in non-production-like instances, which results in conflicts on target instances. It is a common resolution practice to promote the last update set. This "resolution" unintentionally orphans some assets and disconnects your planned BOM from the actual release payload.

Principle 2 - Visibility: Best Practices

The following are the visibility best practices for scalable instance management. These practices will help you understand the value your ServiceNow team produces, identify bottlenecks, implement improvement, measure success, and drive a better ServiceNow ROI.

Release Packaging

This principle will be discussed later in this ebook, but having visibility to named and versioned release payloads will dramatically impact your productivity throughput. Release packaging must represent the actual ServiceNow release payload.

User Story Integration

User stories articulate how your planned features and enhancements will deliver value. However, this plan must be integrated with the actual release payload. Planned vs. Actual must be as close to identical as possible.

360° View

You must have a real-time, multi-instance view of every update set, app, plugin, etc., across your entire ecosystem. Without this capability, all your plans are merely academic or theoretical and don't represent reality.

Leverage Platform Engineering Metrics

Measure your performance based on lead time to delivery, lead time to production, and deployment frequency.

Lead time to delivery measures the time it takes from a feature/enhancement request to its being released to production.
Lead time to production measures the time it takes for a feature/enhancement to be released to production after it has been migrated to your lowest QA/Test instance.
Deployment frequency includes the deployment frequency between all instances (e.g., Dev to QA, QA to Prod, etc.).

Principle 3 - Quality

Test automation is nothing new to ServiceNow delivery processes and procedures. ServiceNow provides an Automated Test Framework (ATF) that can be used as well as Code Scan. Test automation must be utilized, but utilized along an automated pipeline. It isn’t good enough to execute tests after a developer pushes updates to a QA instance. The mean time to give feedback (MTTF) to developers must be as short as possible.

Test execution should be triggered as soon as a developer promotes change to a QA instance. The longer the MTTF is the greater the chance is that a developer has already moved on to another work item. Late feedback causes a developer to context switch from the work they are doing to remember the work they did. This context switching is a huge productivity lag.

Principle 4 - Speed

Bill McDermott said, "the problems that businesses used to solve in years we're solving in weeks and months. So the whole thing is speed."

Bill McDermott, interview w/ Maria Bartiromo, June, 2022

Bottlenecks

A process bottleneck is a release phase receiving more work requests than it can respond to while executing at maximum capacity. This capacity debt interrupts the flow of work and grows your undeployed backlog

Late Stage Conflicts

Conflicts arising as update sets are migrated beyond development instances are time-consuming to resolve. You're left with taking the shortcut already discussed or detangling the work from multiple developers.

Repetetive Tasks

Human beings are terrible at repetitive tasks because as they develop muscle memory, they will make mistakes without realizing they have done so. We will reason why we did something differently once when it was done correctly the other dozen times.

Manual Work

Manual effort is always slower than automated tasks. However, there is no avoiding the fact that software development involves manual effort. The goal is to automate everything as much as possible.

Pervasive Automation

Automation not only speeds things up, but the beauty of automation is that it is a record of what actually, physically, and factually happened. Not what was planned to happen. Not what was supposed to happen. Not what someone thought happened. Automation is a viewable plan of what will happen and an auditable record of what did happen.

Anything that can be automated must be automated. Moving as many repetitive tasks from humans as possible is essential to maximizing your delivery pipeline throughput.

Principle 4 - Speed

Continuous Synchronization

In principle, continuous synchronization (CS) is the automation practice of automating forward and synchronizing backward all deployed changes across all sub-prod and prod instances. Automating deployments forward from development instances is intuitive and obvious. However, the practice of CS demands that changes to higher-level instances synchronize backward to all lower-level instances. For example, suppose an update set is completed in a Dev instance and migrated to your QA instance. In that case, that change MUST now be automatically migrated backwards from QA to all of your other Dev instances. This back-syncing must be continuous, which means it must be automated.

Here’s another example. It is common to make hotfix changes in your PROD instance(s) or other high-level sub-prod instances.

These changes must also be synchronized backward to all other instances below them. Continuous Synchronization seeks to keep all instances as close to production-like as possible. Automatically synchronizing all your instances in this manner does two things.

Reduce the need for cloning by ensuring all sub-prod instances are as production-like as possible. By reducing instance drift, you don't need to resort to the lengthy cloning process to resolve the errors associated with instance drift. Developers are always working against the latest-of in their instance(s) and don't have to put their efforts on hold waiting for a new clone. Instance drift is a key contributor to error rates, rework, and productivity lags and delays.
Shift-Left conflict detection and resolution. Backward synchronization will either prevent conflicts from happening in the first place or allow developers to correct them before they begin new work. CS is a huge preventative measure and reduces MTTR because your team deals with conflicts when they happen, not days or weeks later.

Principle 5 - On Demand Deployments

Continuous Deployment

Continuous Deployment (never abbreviated) is a practice that builds upon CD. CD makes release payloads available for production. Continuous Deployment goes one step further and automatically deploys into production. This isn’t as risky or scary as it seems. Consider that the same automation mechanics that will be used for production deployment will have been tested and vetted across all your sub-prod instances. Automated production deployments are built on previous successes in sub-prod instances.

On demand deployment builds upon the automation recommended in principle of Speed. There are two phases (1) Continuous Delivery and (2) Continuous Deployment.

Remediation Propagation

Despite everything covered so far, conflicts will arise and they need to be addressed. On demand deployments stipulate that these remediation steps be recorded and played back as part of your deployment automation and continuous syncing. In other words, the remediation steps associated with a specific update set conflict need to be propagated throughout your ServiceNow ecosystem. Continually remediating the same error is a form of manual task that must be automated. This keeps your ecosystem as production-like as possible and ensures you can practice phase two, Continuous Deployment.

Change Requests

On-demand deployments not only use automation native to ServiceNow but also integrate that automation into ServiceNow Change Requests. On-demand deployments emphasize speed and agility with traceability. ServiceNow was built to be a system of record for change (among other datasets), and customizations, features, and apps to ServiceNow should be included in that record.

Anytime, Anywhere

On-demand deployments should be able to happen by click, schedule, or triggerable action (e.g., a change request update, successful test completion, or a specific time has come). Your scheduling system should also include blackout dates to help safeguard you from making deployments during unsafe windows. This safeguard implies that triggered, automated, and click deployment are tied to your blackout calendar.

Can you override blackout windows? Absolutely, but remember all deployment actions, whether manual or automatic, create forensic records. On-demand deployments mean you always have complete visibility before and after deployments. Remember, On-demand deployments mean you have speed and agility with safety.

Continuous Delivery

Continuous Delivery (CD) is a practice that is applicable to ServiceNow production releases. The goal is to make release payloads available for deployment to your production instance(s). CD accomplishes this goal by implementing the pervasive automation described in principle 2, Visibility.

Principle 6 - Command & Control

Command and Control (CC) applies and codifies all previous principles into a governed system to enhance and enrich what you are already doing. It aligns ServiceNow admins, developers, and stakeholders toward the common goal of delivering features and enhancements to ServiceNow as quickly, safely, and compliant as possible.

Package Agility

On-demand deployments also require curated release packages defined by selecting and ordering the list of update sets, apps, plugins, records, scripts, etc., actually installed on the source instance. Using spreadsheets is notoriously error-prone, takes hours to days to build manually, and does not represent an immutable record of each release's BOM.

Orchestration is a pivotal aspect of release packaging, where the true power of a curated BOM is harnessed. It's not enough to have a curated BOM if people then manually migrate or import each line item. The real impact is felt when this curated BOM is orchestrated using ServiceNow's native automation mechanics to deploy the payload, enhancing flexibility, speed, accuracy, and governance for instance-to-instance migrations.

Principle of Least Privilege

The principle of least privilege is an instance management best practice in which a user should only have access to the specific instances and levels within instances required to complete their work. It is common for ServiceNow developers to be granted elevated privileges across the ecosystem (e.g., Dev, QA, UAT, SIT, PROD, etc.) so they can deploy their update sets and resolve conflicts.

If you have correctly implemented the automation recommendations in principle 2 (Visibility), the need for elevated privileges should be significantly reduced because:

The automation mechanics are migrating all the features/enhancements instead of people manually performing this work.
Continuous syncing minimizes the likelihood that conflict errors will occur in higher instances because they were addressed in lower instances where it is more appropriate for developers to have elevated privileges.

Focus on Systems, Not Goals

CC focuses on system function, robustness, and improvement, not just the goal of shrinking your backlog faster. Obviously, shrinking your backlog as fast as possible is a goal, but that is an outcome of an efficient system that provides command and control built on top of the other four principles discussed in this ebook.

In his book Atomic Habits, author James Clear said it best. "Achieving a goal only changes your life for the moment. That's the counterintuitive thing about improvement. We think we need to change our results, but the results are not the problem. What we really need to change are the systems that cause those results." (Clear, James. Atomic Habits [p. 25]. Penguin Publishing Group)

Platform engineering focuses on the mechanics and practices within the system, and the previous four principles provide the foundation of having command and control with speed, scalability, stability, and compliance.

What follows next are additional practices that embody platform engineering command and control.

Friction-Free Communication

Friction-free communication and fast-feedback loops are the lifeblood of platform engineering success. Continuous syncing and deployment depend on automation, platform native integration, and cross-functional collaboration. Everyone across the release pipeline shares the same data - no one across the value stream can operate in a silo.

No More War-Room Exercises

Most ServiceNow release practices include a multi-day pre-prod release planning meeting. This exercise is conducted to compensate for the lack of visibility, quality, and BOM-to-payload integration. With proper platform engineering practices in place war room exercises can be eliminated.

Leverage Collective Wisdom

Providing everyone in your ServiceNow value stream with shared, real-time information increases your team's decision-making accuracy and effectiveness. In his book, The Wisdom of Crowds, author James Surowiecki found that "[diverse] groups consistently make better decisions and come up with better answers than most of their members, and surprisingly often the group outperforms even its best member."

Andon Cord

The beauty of command and control is that anyone at any time can pull the "andon cord." Andon is a Japanese word denoting lighting equipment such as a lamp. Andon was later adopted as part of the Toyota Production System in which it signalled (i.e., a blinking light) that a defect had been found at a specific workstation. The Andon also represented that the entire production system had been halted and awaiting remediation.

It may seem like a speed-killer to stop a system when a defect or problem is found. However, TPS found that by stopping the manufacturing system at the time of a problem, you had an immediate opportunity for improvement as opposed to letting the defect move further down the line. The output metrics of TPS are legendary and factual and have been written about in numerous books and academic papers. It is a concept recommended by xtype.

Being Present

Resolving issues in the present rather than the future allows you to remediate issues with the highest chance of being solved because they are top of mind. To not do so is to pass forward a form of technical debt that comes with compounding interest in the form of instance drift, longer planning meetings, privilege sprawl, more troubleshooting and more.

Additionally, the learnings from immediate troubleshooting will improve your process and platform engineering setup with every release. In other words, your practice and process get better and more efficient with every release. Without the practice and recommendation of the "Andon Cord," the opposite will happen.