Artificial intelligence and data science are big news in 2023. The rise of generative artificial intelligence has, of course, led to this dramatic increase in visibility. So what could happen to this area in 2024 to keep it on the front page? And how will these trends impact business?
Over the past few months, we’ve conducted three data and technology leaders surveys. Two of them attended the MIT Director of Data and Information Quality Symposium—one sponsored by Amazon Web Services (AWS) and the other by Thoughtworks. The third survey was conducted by Wavestone, formerly NewVantage Partners, whose annual surveys we have previously written about. More than 500 senior executives took part in the new surveys, with perhaps some overlap in participants.
Surveys don’t predict the future but suggest what people closest to data science and companies’ artificial intelligence strategies and projects are thinking and doing. Based on this data, here are five major emerging issues that deserve your close attention:
As mentioned, generative artificial intelligence has attracted enormous attention from businesses and consumers. But does it bring economic value to the organizations that use it? The survey results show that while interest in this technology is high, its value has not yet been realized. Many respondents believe that generative AI has transformational potential; 80% of AWS survey respondents said it will transform their organizations, and 64% of Wavestone survey respondents said it is next-generation technology. The vast majority of respondents are also increasing investment in technology. However, most companies are simply experimenting at the individual or departmental level. Only 6% of companies in the AWS survey have any production generative AI applications, and only 5% in the Wavestone survey have large-scale production deployments.
Production adoption of generative AI will, of course, require more investment and organizational change, not just experimentation. Business processes must be redesigned, and employees retrained (or perhaps, in some cases only, replaced by generative AI systems). New AI capabilities need to be integrated into existing technology infrastructure.
Perhaps the most significant changes are related to data – managing unstructured content, improving data quality, and integrating different sources. In an AWS survey, 93% of respondents agreed that data strategy is critical to benefiting from generative AI, but 57% have yet to make any changes to their data.
Companies are feeling the need to accelerate the creation of data science models. What were previously craft activities are becoming increasingly industrialized. Companies are investing in platforms, processes and methods, feature stores, machine learning operations systems (MLOps), and other tools to improve productivity and speed of deployment. MLOps systems monitor the health of machine learning models and determine whether they are still making accurate predictions. Otherwise, models may need to be retrained using new data.
Most of these capabilities are provided by external providers, but some organizations are now building their platforms. While automation (including automated machine learning tools, which we’ll discuss below) helps improve productivity and enable greater participation in data science, data science’s most tremendous boon to productivity is arguably the reuse of existing data sets, functions, variables, and even the entire model.
According to a ThoughtWorks study, 80% of data and technology leaders said their organizations use or consider using and managing data products. By product data, we mean packaging data, analytics, and artificial intelligence in a software product offering for internal or external customers. From concept to implementation (and continuous improvement), it is managed by data product managers. Examples of information products include recommendation systems that tell customers what products to buy next and price optimization systems for sales teams.
However, organizations view information products differently. Less than half (48%) respondents said they include analytics and artificial intelligence capabilities in their data product vision. About 30% view analytics and artificial intelligence separately from data products and use the term only for reusable information assets. Only 16% say they don’t think about analytics and artificial intelligence in a product context.
We favor a definition of data products that includes analytics and artificial intelligence because that is where data can be helpful. However, what is essential is that the organization consistently defines and discusses information products. If an organization prefers a combination of “data products” and “analytics and artificial intelligence products,” that can work, too, and this definition retains many of the positive aspects of product management. However, without definitions, organizations can become confused about what product developers should be building. READ MORE. computeritblog