Troubleshooting: art, science or both?

Troubleshooting IT issues is often seen as an art due to the creativity required to navigate unfamiliar challenges. However, it’s equally a science. By balancing art and science, organizations can significantly improve their approach to IT troubleshooting.

The art of troubleshooting involves intuition, creativity, and experience. IT professionals rely on these qualities to adapt to unforeseen problems and explore innovative solutions. On the other hand, the science of troubleshooting demands a disciplined, repeatable process. Preconfigured tools, dashboards, and workflows ensure consistency and efficiency.

Why troubleshooting takes so long

Two key factors often delay IT troubleshooting:

1. Complexity: Modern IT ecosystems involve diverse users, locations, platforms, and protocols, such as Zero Trust architectures. These elements add layers of intricacy, create gaps in visibility, and complicate root cause analysis.
2. Lack of preparation: Organizations frequently lack updated documentation, sufficient telemetry, or preplanned workflows. New applications may be deployed without comprehensive visibility or performance management strategies.

A structured approach to troubleshooting

To address these challenges, I recommend adopting a scientific, repeatable process built on four key pillars:

1. Preparation and onboarding:
   • Monitor assets and onboard applications to ensure visibility from deployment.
   • Maintain updated architectural documentation for quick reference during incidents.
2. Instrumentation and telemetry:
   • Define key performance indicators (KPIs) and collect granular telemetry data.
   • Use custom dashboards and daily performance reports to establish baselines for normal operations.
3. Workflow and process:
   • Map out troubleshooting workflows for each application or service, identifying where to look and what data to analyze.
   • Integrate change management protocols, defining rollback criteria and measuring performance impacts.
4. Continuous improvement:
   • Review every incident to refine processes and address visibility gaps.
   • Foster collaboration among teams, such as security, cloud, and virtualization, to ensure alignment and shared understanding.

Getting on the same page: a performance troubleshooting taxonomy

For maximum effectiveness, organizations must adopt a structured framework to standardize the process of diagnosing and resolving issues. This framework aligns teams, simplifies workflows, and sets the stage for automation. Key considerations include:

• Symptoms and conditions: What are you observing?
• Possible causes: What might be contributing to the issue?
• Investigations: What steps will you take to identify the root cause?
• Devices and domains: Where might you need to look?
• Tools and data: What tools and data will you use?

In simple cases, this approach can lead to a single streamlined workflow. In more complex scenarios, however, multiple issues, potential root causes, and investigations may require collaboration across teams from different domains, resulting in more intricate workflows.

Real-world example: resolving a network performance issue

A real-world case study illustrates the value of a structured troubleshooting approach. A global organization was experiencing poor file transfer performance, causing weeks of frustration, missed deadlines, and decreased productivity. The root cause was initially unclear, and the investigation involved several pivots. Here’s how the issue was resolved:

1. Initial investigation: Symptoms included slow file transfers impacting productivity. The team started with user feedback and monitoring tool data.
2. Data analysis: Telemetry revealed no retransmissions at one site but inconsistent traffic visibility at another, suggesting a network configuration issue.
3. Root cause: The investigation uncovered misconfigured SD-WAN tunnels caused by a simple typo, leading to routing asymmetry and degraded performance.
4. Outcome: Correcting the misconfiguration resolved the issue, restoring performance and enabling effective use of available bandwidth.

During the investigation, these Riverbed solutions played a crucial role:

• NetProfiler: Enterprise-wide flow collection, analytics, and reporting.
• AppResponse 11: Real-time packet-based performance monitoring, analytics, and packet capture.
• Packet Analyzer Plus: Fast, focused performance analysis of large capture files.
• Transaction Analyzer: Detailed decodes, advanced performance analytics, and transaction simulations.

This case underscores the importance of visibility, collaboration, and disciplined processes in troubleshooting complex issues.

Recommendations for IT teams

To enhance troubleshooting efficiency and minimize downtime, I recommend the following steps:

• Invest in visibility tools: Ensure comprehensive observability across the IT stack.
• Adopt proactive practices: Preconfigure dashboards, alerts, and workflows before problems arise.
• Foster collaboration: Align cross-functional teams with shared processes and communication strategies.
• Document and iterate: Maintain living logs of investigations and continuously refine processes based on lessons learned.

Troubleshooting is both an art and a science, requiring creativity and discipline to solve problems effectively. By adopting a structured, repeatable approach, IT teams can improve service quality, reduce resolution times, and better support mission success. As I summarized during the session, reducing time to troubleshoot starts with preparation, collaboration, and a commitment to continuous improvement.

For further information, contact your Riverbed account manager or check out the Riverbed Performance Foundations Course.

At MilCIS 2024, we explored the need for a strategic shift towards unified observability, cultural transformation and continuous improvement to elevate enterprise IT and organizational success.

The risks of a siloed approach to IT performance management

The traditional siloed approach to IT management—where individual teams operate in isolation with their own tools and priorities—has proven inadequate. Key risks include:

• Loss of visibility: As IT systems grow in complexity, isolated monitoring solutions create blind spots, making it harder to detect issues.
• Reactive operations: A firefighting culture develops, focused on resolving immediate issues rather than proactively enhancing the digital experience.
• Inefficiencies in troubleshooting: Data and tool silos hinder collaboration, delaying root cause analysis and problem resolution.
• Cultural fragmentation: Teams prioritize “mean time to innocence” (MTTI)—proving their system is not at fault—over achieving overall mission success.
• Slowed velocity: AI-driven operations (AIOps) initiatives are stalled, and performance data fails to align with mission outcomes and SLAs.
• Increased costs: Redundant tools and inefficiencies drive higher operational expenses without delivering unified insights.

Even when domain tiers meet SLAs and report few incidents, users often experience frustration, leading to lost productivity, time, and revenue. This misalignment arises because end users feel every bump along the “IT road.”

According to EMA Research, the percentage of fully successful network operations (NetOps) teams dropped from 49% in 2016 to 27% in 2022, underscoring the urgent need for change.

Moving from visibility to Unified Observability

To address these challenges, organizations must shift from simple visibility to Unified Observability. Unified Observability integrates performance metrics across domains, including user experience, network performance, and application monitoring, to provide a holistic view of IT services. This fosters better decision-making and proactive issue resolution. By delivering actionable insights across the digital infrastructure—from networks to applications—Unified Observability empowers IT teams to anticipate issues, improve collaboration, and enhance end-user experiences.

This transformation involves:

• Depth and breadth of telemetry: Collecting high-quality, granular data across all technology domains.
• Cross-team collaboration: Breaking down silos to share data and insights.
• Actionable analytics: Leveraging enriched data and advanced analytics for meaningful insights and automated responses.

A strategic approach to Performance Management

Strategic Performance Management (SPM) is a holistic methodology that integrates policies, processes, skills, staff and tools to enhance IT service delivery. SPM aims to improve stability, predictability and confidence in IT systems while aligning with mission outcomes.

An effective SPM strategy delivers:

• Customer satisfaction: Improves user experience and reduces frustrations.
• Operational effectiveness: Streamlines processes and eliminates redundancies.
• Mission relevance: Aligns IT with organizational goals and SLAs.

SPM requires commitment, persistence, and expert guidance. Organizations should assess their IT operations maturity, identify growth areas, and set achievable targets. Different areas or services within the same organization may vary in maturity, requiring tailored approaches.

Cultural changes for a “service” mindset

Adopting Unified Observability and continuous improvement necessitates a cultural shift towards a “service” mindset. This shift involves:

• Collaboration over competition: Encouraging teams to work together rather than operate in silos.
• Accountability for outcomes: Aligning teams around shared goals such as mission success rather than individual performance metrics.
• Focus on continuous improvement: Cultivating a culture of learning and adaptation, where feedback loops drive innovation and service excellence.

Challenges likely to be encountered include resistance to change, skills and tooling shortages and changes in command commitment (due to ongoing rotations in the case of the military).

Raising the bar: elevating enterprise IT

True transformation requires organizations to aim higher by embracing change and innovation. A key enabler is the establishment of Performance Command Centers—centralized hubs for managing telemetry, governance, and KPIs. These centers foster alignment, transparency, and unified decision-making across IT domains.

Key steps include:

• Defining purposeful metrics: Measuring what matters—metrics that reflect mission outcomes and user satisfaction.
• Engineering Unified Observability: Deploying solutions that provide end-to-end visibility and insights across hybrid IT environments.
• Aligning with mission goals: Ensuring IT strategies support resilience, adaptability, and cost-effectiveness.
• Fostering continuous improvement: Establishing feedback mechanisms and iterative processes to refine IT service delivery.

Case study: Unified Observability in action

The U.S. Special Operations Command (USSOCOM) offers a compelling example of Unified Observability’s impact. By transitioning from device-centric monitoring to end-user experience management, USSOCOM achieved:

• Improved troubleshooting: Enhanced staff skills and reduced resolution times.
• Streamlined tools: Reduced tool redundancy while maintaining comprehensive visibility.
• Mission success: Fewer IT-related disruptions and improved productivity.

These outcomes increased confidence in IT systems and strengthened alignment with mission objectives.

Learn more about how Riverbed has partnered with Defence customers over the years to optimize distributed network environments, speed applications, unify data, and gain actionable insights from across IT ecosystems.

Key takeaways

A siloed approach to IT performance management is no longer viable in today’s complex IT environments. Organizations must adopt a strategic approach emphasizing Unified Observability, cultural transformation, and continuous improvement. By aligning IT with mission outcomes, fostering collaboration, and embracing change, organizations can elevate enterprise IT and deliver lasting value.

Transforming IT performance management is a journey, not a destination. Success requires commitment, collaboration, and a willingness to adopt new methodologies. Organizations prioritizing Unified Observability and continuous improvement will lead the way as the IT landscape continues to evolve.