Dynatrace ventures into AI observability with new solution, covers entire LLM stack

Dynatrace, a Massachusetts-based company that provides technology to monitor and optimize application performance, announced today that it will expand into the artificial intelligence space.

At its annual Perform conference, Dynatrace revealed plans to augment its core analytics platform with new capabilities to track both large AI models and applications powered by them. The new offering, called Dynatrace AI Observability, aims to give enterprises tools to closely monitor generative AI systems as they are increasingly adopted.

The move comes at a time when enterprises across all sectors are bullish on the potential of generative AI and are racing to embrace the technology across internal and external applications — while keeping a close eye on the risks they can bring along, including hallucinations, biases and security gaps. 

“This technology (Gen AI) enables organizations to create innovative solutions that boost productivity, profitability, and competitiveness. While transformational, it also poses new challenges for security, transparency, reliability, experience, and cost management. Organizations need AI observability that covers every aspect of their generative AI solutions to overcome these challenges. Dynatrace is extending its observability and AI leadership to meet this need, helping customers to embrace AI confidently and securely with unparalleled insights into their generative AI-driven applications,” Bernd Greifeneder, CTO at Dynatrace, said in a statement.

What will Dynatrace AI Observability do?

Today, businesses see generative AI as the key to remain competitive. They are tapping the novel technology to improve efficiency and productivity, drive automation and foster innovation. However, with all these benefits, gen AI also brings the risk of high costs and biased or inaccurate answers, leading to bad experiences and poor retention. This can easily affect the whole project and the return expected from it.

The only way to address these problems is to stay vigilant and proactively identify and fix the underlying issues, be it model drift, unforeseen data scenarios or underlying system failure. This is where Dynatrace AI Observability comes in.

Leveraging Dynatrace’s ability to bring together metrics, logs, traces, problem analytics and root-cause information, the observability solution monitors the entire AI stack behind modern applications end-to-end, right from the infrastructure layer involving Google TPUs and Nvidia GPUs and foundational models such as GPT-4 to semantic caches, vector databases and orchestration frameworks covering modern RAG architectures (like LangChain).

This gives teams an operational view of the entire lifecycle of AI applications – allowing them to identify performance bottlenecks and root causes. 

For instance, the solution can provide insights into infrastructure utilization (including temperature, memory utilization, and process usage), saturation and errors or the accuracy of the models in use. When the models are running at scale, it can also highlight resource consumption and operation costs, enabling better optimization.

“Integrations with cloud services and custom models such as OpenAI, Amazon Translate, Amazon Textract, Azure Computer Vision, and Azure Custom Vision provide a robust framework for model monitoring. For production models, this provides observability of service-level agreement (SLA) performance metrics, such as token consumption, latency, availability, response time, and error count,” Dynatrace’s Florian Lettner, senior director of product management, and Alois Reitbauer, chief technology strategist, wrote in a joint blog post.

Notably, the observability solution also comes integrated with the company’s proprietary Davis AI engine. This enables it to trace the output of AI apps with precision, paving the way for better compliance with privacy and security regulations and governance standards.

Available starting today

While Dynatrace is making the observability solution available for all its customers starting today, some companies, including OneStream, appear to have received early access for testing purposes.

“Generative and predictive AI will unlock new possibilities for our business with our ML and LLM services, but to implement them successfully, we need to ensure that our services supporting these critical workloads are reliable and perform well. That’s why we rely on Dynatrace, a leader in AI and observability. Our teams use Dynatrace to build and optimize generative AI apps that perform well and are cost-effective to manage and deploy at scale,” Ryan Berry, SVP of engineering & architecture at OneStream, said in a statement. 

In the coming years, as generative AI investments grow, it will be interesting to see how companies adopt the observability solution from Dynatrace. Monitoring, after all, is the most critical phase in implementing gen AI – which is set to become a $1.3 trillion market by 2032. However, it is also worth noting that Dynatrace is not the only player vying for the elusive AI observability space. The market already includes players like Monte Carlo, Arize, Context AI, Weights & Biases, and Datadog.