Data is the most valuable asset in every sector in our century, but having the right data is crucial when making decisions. Transforming Systems’ vision is to enable the National Health Service and social services to utilize real-time information to make data-driven decisions about health and social care at a price that is affordable for all commissioners in these fields.
Transforming Systems Supports Health Institutions
Transforming Systems helps the healthcare ecosystem through its product portfolio, which enables NHS teams to determine actionable insights from a wide range of healthcare data.
Antreas Pogiatzis is the architect and lead developer for Transforming Systems and works as a KTP Associate at the University of Greenwich. Currently, he mostly conducts his research on the AWS cloud, although this wasn’t always the case. Legacy data pipelines had been used in the organization until Pogiatzis encountered modern computing environments, such as containers and serverless, during his research.
At Transforming Systems, we focus on data analytics and we have to tackle real-time data flow.
Legacy data pipelines were very difficult to maintain in a dynamic environment where real-time data flowed 24/7, and a change was needed. Pogiatzis began evaluating serverless pipelines and investigated how different architectures behave under different circumstances in the context of big data analytics, which led him to test each architecture’s latencies, throughputs, cost scaling, and more.
As a result of his research, Pogiatzis and his team decided to move away from legacy data pipelines and into serverless technologies instead, as they believed that creating serverless pipelines would better serve their customers' needs."Serverless programming is a new paradigm that must be adopted with extreme care. It is easy to make costly mistakes in the serverless world, so migrating without a well-defined plan is extremely risky. Ensuring high-level observability must be a priority and Thundra provides this with almost no overhead."
The team also made this choice due to the fact that the legacy software ran on a huge, clunky server that was receiving data and performing the compute process. There were some Java services running on it, but when the team looked at the big picture, there was always a latency in data processing. Additionally, when the server was idle, Transforming Systems still had to pay for it and undertake the difficulty of maintenance. This made the serverless pay-as-you-go model an ideal, cost-effective option for them.
Although there was a learning curve when they transitioned to serverless, Pogiatzis’s team respectively relaxed in regards to system maintenance and operations after they became more familiar with AWS serverless services. Transforming Systems receives operational data from hospitals, and the volume and size of data can vary day-to-day. Since the frequency of the amount of data Transforming Systems receives is unpredictable, scaling up and also scaling down is very important for maintaining their workflow, which meant having elasticity in the cloud environment was crucial for them.
Dealing with Streaming Healthcare Data Brings Observability Challenges
When Pogiatzis’s team initially shifted to serverless, they thought that Lambda functions, S3 buckets, and SNS and SQSs combined together would meet their needs, and proceeded with those tools. They thought that it wouldn’t be difficult to manage the health of the applications built by serverless tools.
The team did a proof of concept, and in the beginning things ran smoothly aside from debugging. But when production was implemented, they discovered that, while the process was generally going well, observability was lacking. This led them to realize the problems in both stages of development: pre-production needed solutions for debugging and testing, while production needed solutions for observability and troubleshooting.
Like many others, Pogiatzis and his team tried out various solutions for the problems raised by modern microservices, such as exploring several different third-party solutions for managing logs, metrics, and traces. But nothing really provided a solution for both pre-production and production stages.
Starting Debugging in Pre-Production
The team was already struggling to become acquainted with serverless tools like AWS SAM, Terraform, AWS SDK, etc. But then – voilà! Pogiatzis discovered Thundra, which provided true end-to-end debugging for his applications on the cloud at the pre-production stage. While the Transforming Systems team learned about the serverless toolset and changed their mindset about what serverless had to offer, Thundra was there each step of the way, supporting Transforming Systems by creating bug-free microservices that stand alone interact with each other in distributed modern architectures.
Thundra’s platform is the missing piece of development tools, providing a fully immersive serverless development.
Transforming Systems built a resilient system using S3 buckets, SNS topics, SQS queues, Lambda workers, and DynamoDB databases. Pogiatzis’s team built event-driven architecture and began observing it with CloudWatch logs.
“Digging into the logs is a poor practice and it is already an experience in the past,” said Pogiatzis while talking about the challenges he faced in both production and pre-production regarding observability.
Thundra: A Highly Reliable Solution for Pre- and Post-Production
One of the biggest pain points of serverless is that developers need to go to the log group and find the stream related to the particular error they are looking for; but the data stream changes frequently. This makes it necessary to go back and forth multiple times to understand the root cause of an error – unless you use troubleshooting tools like Thundra.
According to Pogiatzis, “If you have a lot of moving components in your application architecture, Thundra is a lifesaver when you need to understand the behavior of distributed architectures. Without Thundra, you always need to go back and search for the needle in the haystack again and again.”
Debug and Troubleshoot in Every Phase of Development
While building the serverless architectures in Transforming Systems, Pogiatzis’s team built around 100 Lambda functions. After getting started with Thundra, they immediately discovered that 50% of their Lambda functions had errors that they were not aware of. Troubleshooting became much simpler for the developers of Transforming Systems after they plugged Thundra into their systems.
Optimizing Performance with Thundra
Before Transforming Systems began utilizing Thundra’s services, the execution of traces were taking so long that they timed out. Worse, Pogiatzis’s team wasn’t aware of the issue because they continually got lost in their log piles. After integrating Thundra, they were able to get a clearer picture of which part of the trace was taking a longer time to execute. But that was just one benefit Thundra provided them. “Error rates decreased by 50%, and time to resolve issues decreased by 60%,” said Pogiatzis from Transforming Systems.
The latency of the functions also decreased by at least by 50%, anda delay in the response time of components of the application improved by 60%. Additionally, Thundra’s cost prediction works very efficiently for Transforming Systems, allowing Pogiatzis to optimize applications from a business perspective.
Thundra provides deep performance insights into the modern cloud applications, helping organizations ease the debugging process with real-time and post-execution debugging of applications. Thundra makes it easier to monitor microservice applications, detect issues, debug, troubleshoot, optimize costs, remove security vulnerabilities, and prove continual compliance.