David O'Keeffe

Don't only think on this front about scaling in the sense of compute, although I do see that overlooked sometimes. Think about scaling in terms of developers and users. You need to design up front for how many people can effectively on-board data into your system. If you are opting for a centralized model of ingestion instead of decentralized model. Know why. Is the data mesh relevant for you and why not?

The key advantage of a data lake is it is simple to land all kinds of data in at scale. If you put too high a barrier for what can go into your lake, you're better off opting for a structured data warehouse. Always work from raw inputs if you desire data pipelines that are idempotent and concurrent. The ability to decouple compute from storage is powerful.

The best way to manage scope is to not only truly understand your requirements but then codify them as well. How do software engineering teams achieve that? You might've guessed it already, but the answer is Test Driven Development (TDD). Believe it or not, it is possible in the data space too. Tests are requirements! So write your requirements as tests.

This is actually a spot on definition in my opinion. A lot of what DataOps is about actually is not anything fancy. The simplest way to start DataOps is to look at your team and ask questions about the way you work. - How fast can you create actionable analytics? - What stage of your work takes the longest? Why? - What is "testing" to you? To what level of automation is necessary? How does it relate to requirements? - How often do you encounter errors in production? - How quickly do you fix errors in production? - How do you learn from your mistakes?

An interesting new development is thinking about what security threats exist in AI models and how they can be exploited. NIST even have an AI Risk Management Framework that's worth checking out @ https://www.nist.gov/itl/ai-risk-management-framework You can find a database of vulnerabilities here too: https://airisk.io/

Alongside all these, another important aspect of having research skills is the ability to form hypotheses, then evaluate them according to the evidence at hand. That is the key to making the scientific method work, and what set the likes of Johannes Kepler apart from the rest. It is pretty much the core principle of lean, and something I see lacking in so many places.