Solving Hospital IT Issues Quickly

Northeast Georgia Health System (NGHS) is headquartered in Gainesville, Georgia, less than a 90-minute drive from downtown Atlanta. The health system serves the region with a network of four hospitals and other care facilities across multiple cities and campuses and has become one of Georgia’s most awarded hospital systems.

The IT staff at NGHS was experiencing an extended series of issues with the Workstations On Wheels used throughout their hospitals. Workstations On Wheels (WOW) are a common sight at hospitals; these powerful, mobile equipment stations enable doctors, nurses, and other staff to provide timely and high quality care to patients. The WOWs were losing connection to the network and becoming useless.

Medical and administrative staff were growing increasingly frustrated, because WOWs are used for most patient interactions and a wide range of tasks. Clinician productivity was suffering significantly, and patient service was noticeably affected, particularly in the emergency department (ED) where responsiveness is crucial. ED staff are the heaviest WOW users and are highly mobile. Hospital management also wanted to know why
these expensive assets couldn’t be used reliably.

IT staff were having a hard time identifying the root cause of the problem, which was occurring more frequently over time. Eventually, when a WOW stopped responding, it became standard procedure to remove the battery and reinstall it to restart the system. This brute force method typically worked, and many people were convinced it must be a wireless problem.

Finding the needle in the (network) haystack

Dustin Deadwyler, Network Engineer for NGHS, was the engineer leading the investigation and the WOW issue was consuming a significant portion of his time. Initial troubleshooting began in the NGHS data center, looking for network and wireless access issues across the hospital sites. Because the problem was intermittent and unpredictable, it could not be replicated and there was no obvious root cause.

Existing tools were not up to the task and didn’t offer Deadwyler the ability to drill down into the data the way he required. “The emergency department is stressful enough, we can’t have staff pulling out batteries to reboot a system while patients are waiting; we needed a real answer,” said Deadwyler. Packet capture analysis and controller logs were also unable to shed light on the problem. The investigation moved on-site to the hospitals, adding travel time and hours spent trailing staff with a spectrum analyzer– with no new insights.

>Visibility with VMware Edge Network Intelligence

>When NGHS deployed VMware Edge Network Intelligence™ (formerly Nyansa Voyance) in production, it provided a platform ideally suited to the ongoing WOW issue. Deadwyler now had the capability to drill down to the level of individual access points (APs) and WOWs with real-time and historical analysis. Edge Network Intelligence collects data that wasn’t visible with previous tools, and correlates data across the network from multiple sources.

When a WOW failed, Deadwyler could see the exact point where the system went from roaming normally to dropping off the network. Examining the access point showed it was still transmitting normally, because the client count only dropped by one. Simultaneously, the number of Layer 2 packet retries spiked to 100% because the AP still had packets bound for the client that was no longer responding.

After sixteen minutes, the WOW client became disassociated – effectively ‘timed-out’ of the system, starting when traffic is no longer seen from the client. This behavior is exactly what the controller is supposed to do in this situation. Eventually, a frustrated staff member removes the battery and reboots the WOW, which reconnects to the network and resumes communicating. Before, during, and after the outage the wired and wireless networks were performing normally, pinpointing the issue with the WOW client. In this case, it was not a network problem and the proof was in hand.

>Identifying the precise time frame of the failure, and correlating it with the other network data, was the key to solving the problem. Armed with this new insight, Deadwyler was able to extract the necessary data from the client logs and packet capture to identify the culprit as a faulty Network Interface Card (NIC). Working with the manufacturer, he was then able to trace the problem to a specific part number. This now explained why the problem in the Gainesville site was much worse than in Brownsville – all of the WOWs in Gainesville had the same NIC model.

Results: Workstations on wheels rolling smoothly

After weeks of frustration, the faulty NIC issue was quickly addressed with no recurrence of the problem. Patients were no longer kept waiting, and clinicians and administrators returned to full productivity. Management regained confidence that technology investments were paying off with validation that staff were more efficient, and most importantly delivering superior patient care. This incident even benefited the NIC manufacturer, who now had additional information to assist in their own remediation process.

Next Steps

Deadwyler is taking full advantage of Edge Network Intelligence capabilities across the NGHS network, applying the insights available in multiple use cases. WOWs use Citrix Virtual Desktop and will incorporate Edge Network Intelligence-Citrix integration for more detailed client analysis. Edge Network Intelligence has been certified by Citrix to work with both XenApp and XenDesktop as part of the Citrix Ready certification program for healthcare. The advanced integration and certification provides detailed visibility into proprietary Citrix ICA session metrics such as ICS latency and session logon times for faster and more complete analysis.

Edge Network Intelligence is invaluable as mobile client use expands throughout NGHS. Use cases include mobile devices for scanning medication and hundreds of Wi-Fi-based phones for voice communication. Also on the horizon are identifying, managing, and securing a range of biomedical devices such as IV pumps and EKG machines used throughout the hospitals. Monitoring and troubleshooting are simplified and Edge Network Intelligence provides a single source of truth for the entire network – without having to make site visits. “I’m using Edge Network Intelligence daily, it’s the first place I go when troubleshooting clients. Other tools don’t give me what I need,” said Deadwyler.

Step-by-step diagnosis with Edge Network Intelligence

All of the data that NGHS used for resolving the WOW issue is readily accessible presented graphically with Edge Network Intelligence. To see the steps, download the case study PDF.

For more information about VMware SD-WAN, visit sdwan.vmware.com.