AN ENGINEER’S GUIDE.

AWS Glue Crawlers with RDS.

Create one, step-by-step!

Crawlers eliminate the need for tedious processes of schema discovery and metadata management.

AWS Glue Crawlers are automated processes that traverse the data source, extract metadata, identify data formats, and create table definitions in the AWS Glue Data Catalogue.
Some of the crawler’s main functionalities include:

• Automated Schema Discovery.
• Schema Evolution ( detecting data changes ).
• Central Metadata Repository usable by other services like Athena, ETL jobs, etc.)
• Integration ( Data Lake and Data Warehouses ).

This is part of a series of articles about different components in a data pipeline system using Glue, Athena, and RDS, among others, so please make sure to follow me to be notified when the next articles become available!

Prerequisites:

• You need your Glue Connection to RDS PostgreSQL ready; you can follow the following guide to set that up if needed:

Glue to RDS PostgreSQL connection.

Step1. -IAM Role-