Scoping Machine Learning Projects

I recently signed on for the MLOps course by DeepLearning.AI and have completed the first part “Introduction to Machine Learning in Production”. Ironically the section that triggered my interest was made optional on the course. It stood out for me because I had been in a situation where I had to scope a ML project, but was completely lost on how to start. I ended up wasting a lot of effort that ultimately weren’t as promising as I’d hoped. Completing the first part in MLOps, I’ve learned valuable lessons one should apply before diving in headfirst, which includes — assessing feasibility, determining value, and planning milestones.

One of the most important — and difficult — parts of any new project is choosing which idea to pursue. With ML use cases everywhere, what can really make or break a project is whether you pick the right one to sink your time and resources into.

Art created with Dall.E 3

Understanding Scoping in ML Projects

Scoping involves defining the problem that the project will address, identifying potential solutions, and assessing the feasibility of those solutions.
There are several reasons why scoping is important.

1. Scoping helps to ensure that the project is feasible. It can help to identify any potential challenges or roadblocks that could prevent the project from being successful.
2. Scoping helps to ensure that the project will address a real need. Scoping can help to ensure that the project is addressing a need that is important to the business or organization.
3. Scoping helps to set realistic expectations for the project by ensuring that everyone involved in the project has a clear understanding of what the project will entail. This can help to avoid misunderstandings and disappointments down the road.

In addition to these benefits, scoping can also help to:

1. Improve communication and collaboration between team members
2. Reduce the risk of project failure
3. Save time and money
4. Increase the likelihood of project success

Imagine deploying an NLP system for customer support without a clear understanding of its objectives. The system might be great at understanding language but may falter in categorising or prioritising customer queries. Scoping ensures that the NLP solution knows whether it’s categorizing queries, routing them, or auto-responding. Without this, the system can become a costly white elephant, impressive in capability but lacking in business value.

Art created with Dall.E 3

Scoping Projects for Technical Feasibility

This is a critical step; how likely is it we can achieve the intended outcome given the available data and model capabilities?

For problem types involving unstructured data like images or text, a takeaway from the course is one can gauge feasibility by asking a human to perform the task using only the same data the model would see. If a person can’t perceive or infer the desired outcome, it implies our “features” lack predictive power.

For structured data like financial records, analyse whether there is data suggestive of the target variable. E.g. Do trends in past customer purchases predict future sales?

Also, the history of related past ML projects should be taking into consideration. Does the rate of prior improvements hint if goals are potentially achievable? Feasibility measures whether the training data supports the problem difficulty — before entrusting lots of effort.

Art created with Dall.E 3

Validate Value and Return

Does this concept address business needs profitably? Here, discovering root causes behind problems and relevant metrics clarifies value potential. Root causes reveal strategic opportunities and constraints, while metrics relate problems to monetisation.

Tangible returns like time or cost savings make value quantitative. Yet intangibles like quality-of-life also have worth.

Art created with Dall.E 3

Milestones

Setting milestones is crucial as they act as checkpoints, ensuring the project remains on track and meets its objectives. For an example; an NLP-driven customer support system, the following milestones could be established:

1. Requirement Gathering: Understand support process and identify pain points.
2. Data Collection: Accumulate and preprocess past customer queries.
3. Model Selection: Choose and train the appropriate NLP model.
4. Integration: Embed NLP into existing customer support platform.
5. Pilot Testing: Process subset of queries and gather feedback.
6. Optimisation: Fine-tune based on pilot feedback.
7. Deployment: Launch NLP system for all queries and monitor performance.
8. Feedback Loop: Set mechanisms for regular feedback and iterative improvements.

Art generated with Dall.E 3

Impact of Proper Scoping on Different Business Areas

Business Strategy: Properly scoped ML projects ensure that technological endeavors are in sync with overarching business goals. Instead of isolated tech experiments, projects become strategic tools driving business growth.

Financial Management & Planning: A scoped project has clear objectives and metrics of success, allowing for better budget allocation and ensuring a higher probability of return on investment.

Commercials: In sectors where pricing is dynamic, ML can be a game-changer. Proper scoping ensures that algorithms are optimised to factor in variables like market demand, competitor pricing, and inventory levels, ensuring optimal pricing strategies.

Product Management: For product teams, ML can offer insights into user behavior, preferences, and pain points. With clear scoping, these insights can be channeled into tangible product enhancements, elevating user experiences and fostering loyalty.

Conclusion

The key to a successful ML project lies in meticulous project selection and scoping. It’s not just about technological prowess but aligning that prowess with business vision.

I hope you picked up some valuable nuggets as I have be written this piece. I am looking forward to completing the MLOps Specialisation.