HPE AI Essentials integrates Apache Airflow to orchestrate data processing pipelines and other scheduled workflows using Directed Acyclic Graphs (DAGs).
The platform supports retrieving DAG definitions from a configured Git repository. Administrators define the repository address, branch, subdirectory, and authentication credentials through the Airflow framework configuration.
When several teams maintain their DAGs in separate GitHub repositories, HPE recommends using Git submodules to organize those repositories under a common root repository.
The platform administrator creates a central GitHub repository and adds each team's existing repository as a submodule. Airflow is then configured to reference the central repository.
This approach preserves the teams' existing development repositories while providing a centralized mechanism for retrieving DAG definitions.
When teams update their DAGs, the central repository's submodule references can be updated to the appropriate commits. This allows administrators to maintain controlled versions of the workflow definitions.
Option A is incorrect because the documented Airflow configuration references a repository rather than independently managing several repositories through separate credentials. Option B introduces unnecessary framework duplication. Option C creates additional Spark-based synchronization infrastructure without providing a native repository-management advantage.
HPE explicitly documents Git submodules as the recommended practice for managing DAGs maintained in multiple user repositories.
Official HPE Reference: HPE AI Essentials Software — Airflow DAGs Git Repository; HPE AI Essentials Software — Defining RBACs on DAGs, including the documented Git submodule best practice.
Submit