Data & AI Technology Radar

Introducing the Unit8 Technology Radar - a comprehensive guide that empowers businesses to navigate the ever-evolving landscape of technology. The radar serves as a strategic compass, providing insights into emerging technology trends, platforms, tools, languages, and frameworks. Join us on this journey as we explore and develop the Unit8 Technology Radar and discover the latest technological innovations that can propel your organization to new heights.

Download PDF Key Trends 2023

What’s the radar about?

Technology Radar is a comprehensive tool, inspired by the pioneering efforts of our colleagues at Thoughtworks, that showcases the latest trends and developments in the Data, Advanced Analytics, and AI space. This tool is a culmination of the collective experience of our engineering team, drawing from hundreds of projects and collaborations with our customers each year.

There are no blips on this quadrant, please check your Google sheet/CSV/JSON file once.

Methodology

The radar categorizes key technological trends and tools into four main quadrants: Infrastructure & xOps, Data & Analytics Platforms, ML & Data Science, and Apps & Visualisation. Additionally, we have ranked these trends and tool by how confidently we would recommend them to our customers.

Adopt: We believe the industry should embrace these technology trends and tools. We incorporate them into our projects and see them as suitable for most of the Enterprises.

Trial: Worth pursuing. Most mature Enterprises should be poised to adopt those trends, even though many of the best practices, whether around architecture or the target operating model, have not been firmly established yet.

Assess: Consider testing the technology to evaluate its maturity and experiment with its potential effects on your Enterprise in the future.

Hold: Proceed with caution. Evaluate carefully if your organisation is internally prepared (talent, skills, infrastructure & data readiness) to embrace the tech trend.

Key trends 2023

1LLMs: The Good, the Exciting, and the Risky

In 2023, Large Language Models (LLMs) like GPT stand on the brink of transforming businesses by enhancing workforce efficiency, customer experience, and streamlining operations.

Integration of LLMs with organizational data is often achieved using vector databases like Pinecone, Chroma DB or Vector search in Azure Cognitive Search. Retrieval augmented generation (RAG) approaches can then be used for efficient document retrieval to feed the LLM and generate responses.

To optimize LLM performance, prompt engineering techniques have emerged, using methods like few-shots and ensembling multiple outputs. Tools like LangChain further enhance LLM capabilities by allowing them to interface with external tools, such as Python interpreters, for complex tasks.

New foundation models like Llama 2, Anthropic, and Dolly from industry players, and Falcon from research keep being released. Some with a commercial license.

However, the industry remains wary. Azure OpenAI is often chosen to increase data privacy and security. Hallucination and lack of reproducibilty and output structure remain a challenge partially addressed by guardrails on outputs. OpenAI improves this with recent features like JSON mode and reproducible outputs

The evolving best practices in LLM engineering have birthed the term "LLMOps". Like MLOps it structures experiment tracking, versioning, and deployment. Cost management and reducing inference time become also critical, leading to methods like shortening prompts or caching results.

For a deeper dive, refer to our technical guide on LLMs.

2Keeping cloud & IT infrastructure cost in check

The escalating cloud costs have become a pressing concern for many CIOs and CFOs early this year, with expenses growing by a staggering 20-30% year-over-year , significantly straining their budgeting and financial planning.

To mitigate these challenges, FinOps tools and practices have emerged as a viable solution. FinOps-type tools, including commercial solutions like Harness and homegrown BI reports, can offer increased visibility into cloud expenditures. Homegrown BI reports, in particular, can be tailored to an organization’s specific needs, providing custom insights into cloud resources consumption. They can identify potential waste, such as unused files or datasets that occupy valuable storage space, or scheduled compute tasks without clear end consumers.

FinOps practices also fosters accountability across business units by providing granular insights into dollar consumption per project or department. This level of details enables the identification and elimination of unused or wasted resources, both in storage and compute. The ultimate goal is to optimize performance to cost ratio and efficiency of IT expenditure, a topic discussed in depth in Unit8 Talks #25, available on YouTube.

3Simplifying Data Access and Empowering Users of All Skill Levels

In 2023, data apps and visualisation trends revolve around simplicity, self-service capabilities and automation. Personalised dashboards have become commonplace, where data insights are created and consumed, empowering users to do more with less. The goal is to make data accessible, comprehensible, and actionable for all users, regardless of their tech proficiency.

Three types of applications are leading this trend (i) classic BI dashboards , (tools like Tableau, PowerBI, Qlik), (ii) low-code tools (Foundry Workshop), and (iii) webapp prototyping tools (i.e. Streamlit acquired by Snowflake).

More recently, the rise of GenAI trends is further accelerating this transformation. For instance, the integration of Copilot with PowerBI now enables users to simply describe the visuals and insights they seek using free text. Similarly, Databricks has introduced English SDK for PySpark, which allows to express data queries in plain English instead of code.

Although these advancements are still in their infancy and there are many open challenges, such as the consistency/reproducability of the results, we believe these new features are great news for everyone who wants to work with data. They not only promise to increase speed and ease of use but also democratize data-driven decision-making, making it an integral part of every business function.

4Bringing data closer to business

In 2023, the surge in all-in-one Data Platforms/Data Lakes is evident. Previously exclusive to major conglomerates in sectors like pharma, finance, and tech, these platforms are now accessible to smaller and mid-cap firms.

Recognizing data's value, smaller entities, such as healthcare firms and private banks, are investing in infrastructure to enhance operational efficiency and remain competitive. Lacking vast IT resources, these companies opt for comprehensive "one-platform" solutions, anticipating their needs over the next 5-10 years. These platforms cater to needs like data processing, lineage, cataloging, health checks, and data quality metrics. Additionally, they offer AI and ML capabilities, emphasizing auditability, bias tracking, and experimentation. A user-friendly interface for those with minimal technical know-how is essential.

The data platform market is fiercely competitive, with giants like Databricks, Snowflake, Dataiku, and Palantir Foundry at the forefront. Support, quality and cost are pivotal for customers. There's a paradigm shift towards treating data not just as a resource but as a product, necessitating clear ownership and roadmaps. Data now drives analytics and decision-making. Effective data management, especially in regulated sectors, requires robust governance and investment. However, many organizations, even large ones, grapple with fragmented and inefficient solutions due to a lack of a unified approach.