Developing an AI Strategy: Effective Research
At Veratai, we follow a simple but well-refined recipe for developing AI Strategies called Discover which is designed for SMEs and business units within larger organisations. Discover consists of three tasks: Stakeholder Interviews, Research & Ideation and Analysis & Synthesis. The first two of these can (and should) be run in parallel.
This post is the second in our series on developing such an AI strategy for your organisation. The first covered Stakeholder Interviews. This one covers the Research & Ideation. (The third - and final - blog in the series covers Analysis & Synthesis.)
Where are we in the process?
At its heart, a good AI strategy needs to answer four questions:
What does the journey look like? What should it do and what should it not do?
How should the organisation change to mitigate future risks and seize future opportunities?
What is the gap between the organisation’s capabilities today and the capabilities it will require to do this?
What impact will AI likely have upon the organisation, its stakeholders and its market ecosystem in the short, medium and long term?
N.B. Business strategy purists will point out that the last point - the specific initiatives that describe how the organisation will transform itself - properly belongs in an implementation or business plan, not a strategy. This is true. That said, in our experience, the stakeholders sponsoring the AI strategy will really need these specifics in order to approve the strategy and to get moving.
The stakeholder interviews should be giving you a good sense of the business. In particular, you should be amassing details on all of the following:
Data: the organisation’s data estate - the good, the bad and the plain missing.
Key business processes: how they work, the role of human expertise, the use of data and systems.
Significant needs and pain-points experienced during day-to-day operations by staff, customers and other stakeholders.
AI maturity: including skillsets, existing capabilities and infrastructure - as well as a sense of “AI literacy” within the organisation.
Furthermore, some of your interviews will start to sketch out elements of the “bigger picture”, for instance like:
What are competitors doing with AI?
What market and customer-related trends are unfolding?
What does the business strategy say and how is the organisation changing in response?
At this point, the bigger picture will very much be a sketch; interviews alone will not give you everything you need!
Research & Ideation: the goals
The purpose of the Research and Ideation task is to help us complete the bigger picture - and in so doing to enable you to develop a “longlist” of big ideas, ideas that you will evaluate and refine in the next phase of the project.
In addition to the three questions listed above, you should use the desk-based research to answer the following:
What developments in AI are most relevant to this organisation, and why?
What lessons are being learned by teams building real solutions with these relevant AI technologies?
Of these developments, what is the state-of-the-art right now? What will be possible in the medium - and longer - term?
Armed with your research goals, you’re ready to start fact-finding. The principal challenge here is the sheer quantity of AI-related information being published, huge swathes of which will be vacuous, over-hyped or just plain wrong. You need to sift the nuggets of valuable information from the mountains of rubbish.
The bad news is, if you’re embarking on this task with little prior knowledge of the field, it’s going to take a long time. Therefore, before you start on the strategy-specific research for your strategy, spend the time developing a solid foundation of background knowledge to help you evaluate and contextualise what you find.
Building background knowledge
As an AI advisor, keeping abreast of the field is part of my job. I set aside time, each and every day, to do this. Given the extraordinary pace of the field, properly understanding the most recent developments would be beyond a full-time job; I need to be focused and systematic.
I have, over time, curated around a dozen general AI news sources, sources chosen for their reliability, quality of insight and breadth of content. Most are delivered by email and I read them every day.
Fundamentally, I am interested in four things:
Relevant R&D. What is happening in the world of AI R&D? This includes:
Releases of frontier and open source models: what new innovations they include and how they perform on the most important benchmarks;
Trending research and repositories - together with any related commentary from experts and developers I can find.
Literature reviews: if someone else has done the hard work of reading all the papers in detail, that saves me a lot of time!
d. Understanding the capabilities - and limitations - of the best models, architectures and techniques in the field. Where are the frontiers and how fast is progress on specific sub-fields or use cases?
Case Studies. What capabilities on well-described task domains have been demonstrated in the literature or by reputable labs? Specifically, I am looking for reports where the methodology, datasets and results have been quantified and well-described, giving me a data-point for what is achievable and guidance on how to replicate the results. An honest assessment of the lessons learned is a bonus.
Platforms. What capabilities and integrations do the major cloud providers and frontier labs offer their customers?
Frameworks and Services. What new SaaS products, libraries and frameworks are generating excitement? What do customers and developers think of them? (Forums and Discord channels are invaluable sources here.)
Your knowledge base
It’s important to maintain a knowledge base of some sort where you can compile everything that strikes you as important, interesting or insightful. I use Obsidian notebooks. I have processes to ingest, connect and retrieve curated information from a selection of email newsletters, research papers and podcasts. These notebooks contain well over 1,000 pages, at present. In the last week I have used the knowledge base twice:
I needed to quickly research the state-of-the-art for time series forecasting - and in particular whether Language Models were being successfully used to advance the field. I was able to find the answers - together with expert commentary - immediately.
I was looking at methods for constructing fine-tuning datasets for Vision-Language Models and was able to quickly find a dozen descriptions from the relevant literature. I dropped these into a Perplexity Space and generated a comparative analysis of different approaches.
So, that’s my process for continually developing and updating my background knowledge. You should develop your own, in a way that works for you. It does take time - that’s unavoidable - but if you make it a habit, you’ll be far more effective at the strategy-specific research tasks.
Doing the research
As the Stakeholder Interviews progress, as you work your way through key business documents and observe key business processes in action, you’re regularly reviewing your notes and asking yourself questions to help stimulate your thinking. Drawing upon your background knowledge, you’ll highlight correspondences between what you’re learning from the interviews and the problems, solutions or processes you’ve learned about previously.
The role of this research is to generate proof-points, fleshing out your ideas and helping to support or discredit any hypotheses you generate. When doing specific research, focus on the technical viability of your ideas and hypotheses; don’t worry about whether particular data, cultural or cost issues could prevent your ideas from bearing fruit.
To focus your research and avoid the “doom-loop” of aimless Googling, I recommend you come up with a list of specific questions first. I tend to divide these into two buckets:
The big questions involve taking a helicopter view of the business and its industry and challenging yourself to think about possible futures and their implications.
The counterfactual questions are about asking how the business could do things differently.
The big questions
Which specific activities, processes and services that this business undertakes will likely be radically changed by AI automation technologies? You are looking for areas where change is coming and consequently you expect the business to experience cost pressure or productivity gains.
Could AI technologies change the need that customers have for this businesses’ products and services? Will expectations change? How about delivery channels? Will AI enable substitute goods and services? Try to build a list of areas where competitive pressure or market transformation is going to affect the business.
Which elements of the business’s value proposition will be very difficult for an AI technology to substitute for? It’s important to recognise these where they exist. You should be generating ideas to support and enhance them.
What data assets does this business hold that is of critical value to the customer? Are future technologies likely to bypass the need for collecting and curating this data? A deep consideration of data assets is crucial when developing an AI Strategy. Sometimes, the data are a considerable part of the value proposition or they impose switching costs or generate a network effect. Depending on the specific context, AI technologies can promise two very different effects here:
The data could be used by a machine learning technology to deliver new or improved outcomes which will be difficult for others to replicate; or
AI technologies could erode the value of the data - either by collecting and processing it from open sources, or by having sufficient general capabilities so as not to require it in the first place.
Of the key business processes we have explored, have others tried to automate or replace them using AI? How successful were they? What lessons did they learn? What dependencies needed to be resolved first?
When thinking about products and services, who is attempting to “disrupt” current markets? How are they doing it? Are they credible - with proof-points and customer feedback available online?
What are the most important and relevant AI research fields? Who has attempted to apply them in contexts similar to yours? Are there research reports, papers or talks that cover the learnings?
What are competitors and industry experts publishing on the subject of AI research and adoption?
N.B. One assumption I’m making throughout this document is that the AI Strategy exists to serve the wider business strategy. Hence, notably absent from the questions above are “classic” strategic considerations around market positioning, value chain analysis or pricing strategy. These (if they exist) are inputs to our Discover process, not outputs.
The counterfactual questions
These tend to be more detailed, focused on the particular processes, activities and pain-points that you’ve uncovered during the Stakeholder Interviews.
“What if” questions ask what consequences and benefits would arise if a new capability existed or if an existing problem was resolved. (”What if the sales team had a concise briefing about their top five prospects ready in their inbox first thing on a Monday morning?”)
“What would need to be true” questions expand on the above and encourage you to draw up a list of challenges or other dependencies which would need to be met. (”What would need to be true for a customer never to contact us with this issue again?”)
“How else” questions force you to think about how existing processes or services could be done very differently. (”How else could the business get feedback on the relevancy of the content in its last publication?”)
Finding good sources
So, you have a good list of questions. Where can you get the answers?
Your first point of call should be the Knowledge Base you’ve been building up. Since this has been curated from trusted sources, it’s a great place to search to get a good picture of research trends and AI products and services. Often, simply searching my knowledge base and following the links gives me everything I need. Other times, I need to augment or corroborate this information. I recommend you prioritise your reading from the following kinds of source:
Industry reports on AI adoption or literature reviews.
Long-form podcasts and articles from industry experts.
Meetup slides, technical blogs and Github repositories.
Business strategy documents and other internally-produced research.
Podcasts with AI developers and blogs of “lessons learned” are invaluable. They get into the nitty gritty of the problems and limitations.
Speaking to anyone in your network who has worked on something relevant. Find out the lessons learned. How effective was their solution? What are they doing next?
The websites of consultancies and AI service providers working in the relevant sector - including (and perhaps especially) those of early-stage start-ups.
Forums, LinkedIn groups, Reddit and Discord servers. These are great places to ask specific questions of experts with more specific knowledge than you might have yourself.
For business units within a larger body corporate: policy, governance and procurement documentation that will help to constrain or shape the kind of recommendations you can make?
Tips for online research into AI
News services like Meltwater, Signal or Newswhip can be useful - if you have access.
When you come up with ideas, be sure to fire off new searches to find the experience of people who have used any of the relevant services, models or libraries. It’s a great shortcut to discovering the current limitations without having to get stuck into them yourself.
Use the new Perplexity Spaces feature (available in Pro). Upload your small collections of relevant documents to form mini knowledge repositories. You can then use the search tool to start generating specific summaries, asking detailed questions and extracting references for follow-on enquiries.
Evaluating the sources
What data would be needed to make a particular model or approach work well?
What would need to be true for a methodology or service to work well in your context? Would you expect to see similar results?
For a case study: what was challenging for the writer in their work? Is this likely to be easier to achieve in 1 year, in 3 years…”?
Does the information contradict your knowledge base and background knowledge? If so, is the evidence or argument offered strong enough to make you update your beliefs?
How much compute would something require? Can you do a rough-order-of-magnitude estimate for training and inference costs? How might you expect those costs to change in 1 years’ time?
When it comes to AI, there’s a great deal of hyperbole, out-of-date information and ill-informed opinions out there. Here are some tips for critically evaluating the claims you come across:
Have any claims or results been verified by others or incorporated into a production system somewhere (a positive indication) - or does it appear to be experimentation only (less reliable).
Fleshing out the long-list
As the strategy-specific research progresses, you will naturally be expanding and refining your “long list” of big ideas. When you have an idea, consider fleshing out the following aspects where you can:
What could the pay-off be and how would it accrue?
What might the timings for an implementation look like?
Can you list key dependencies, risks, unknowns and assumptions?
Are there any interactions or conflicts with other ideas in your list?
Does the idea align with the company’s strategy, AI position & vision, culture?
What commercial imperative, competitive ratchet or existing need is solved by it?
What should the business outsource, buy in or simply bide it’s time to wait & see?
What new capabilities would need to be in place to make effective use of your idea?
Broadly, what is the pathway to implementing your idea? What would the key milestones be?
Remember that the aim here is ultimately to generate and iterate new ideas. The key is to give them room to develop; don’t shut them down prematurely the moment you foresee difficulties. Just make a note of any complicating factors: risks, costs or blockers; try to avoid ruling things out at this stage.
Wrapping up
The Research & Ideation phase ends with a long list of ideas. It’s a series of rough sketches, probably far too long and with many details missing. As a portfolio, it may be inconsistent, overly ambitious, impossible for the business to parallel process.
That’s ok.
In the next phase you’re going to critically examine these ideas. You’ll flesh them out, merge them, drop them and reinvent them. Perhaps you’ll even generate entirely new ones.
In the final blog in this series, we’ll cover this process: how you can take a long-list of big ideas use it to build a coherent and compelling strategy.