Your team is building a data engineering and data science development environment.

The environment must support the following requirements:

support Python and Scala

compose data storage, movement, and processing services into automated data pipelines

the same tool should be used for the orchestration of both data engineering and data science

support workload isolation and interactive workloads

enable scaling across a cluster of machines

You need to create the environment.

What should you do?