rds_pg_url_airflow_database: The connection URL of the backend database for Airflow.( In this tutorial, we use RDS PostgreSQL as backend database of Airflow and another RDS PostgreSQL as the demo database showing the data migration via Airflow task, so ECS and 2 RDS PostgreSQL instances are included in the Terraform script.) Please specify the necessary information and region to deploy:Īfter the Terraform script execution is finished, the ECS and RDS PostgreSQL instances information are listed below: Run the terraform script to initialize the resources. Use Terraform to provision ECS and database on Alibaba Cloud Deploy and run data migration task in Airflow Prepare the source and target database for Airflow data migration task demo Deploy and setup Airflow on ECS with RDS PostgreSQL Step 1: Use Terraform to provision ECS and databases on Alibaba Cloud.In this tutorial, we will show the case of using RDS PostgreSQL high availability edition to replace the single node PostgreSQL built-in Docker edition for more stable production purposes. You can use one of the following databases on Alibaba Cloud: We will show the steps of deployment working with Alibaba Cloud Database to enhance the database's high availability behind the Apache Airflow.Īirflow supports PostgreSQL and MySQL. If you are experimenting and learning Airflow, you can stick with the default SQLite option or single node PostgreSQL built-in Docker edition. OverviewĪpache Airflow is a platform created by the community to author, schedule, and monitor workflows programmatically.Īirflow requires a database. Please refer to this link for more tutorials related to Alibaba Cloud Database. You can access the tutorial artifact, including the deployment script (Terraform), related source code, sample data, and instruction guidance from the Github project. We also show a simple data migration task deployed and run in Airflow to migrate data between two databases.
#Airflow docker emr how to
This is a tutorial on how to run the open-source project Apache Airflow on Alibaba Cloud with ApsaraDB (Alibaba Cloud Database).
0 kommentar(er)
