October 20, 2020 trying out prefect cloud features to determine deployment strategy for productionalizing workflows logged into prefect cloud created a new project in the Team section: then Projects :

mocking lambda invoke from moto import mock_iam, mock_lambda def get_role_name(): with mock_iam(): iam = boto3.client("iam", region_name=_lambda_region) try: return iam.get_role(RoleName="my-role")["Role"]["Arn"] except ClientError: return iam.create_role( RoleName="my-role", AssumeRolePolicyDocument="some policy", Path="/my-path/", )["Role"]["Arn"]

noting here for reference — moto for mocking aws secrets manager from moto import mock_secretsmanager @pytest.fixture @mock_secretsmanager def mock_response(): mock_secretsmanager().start() sm_client = boto3.client('secretsmanager') response = sm_client.put_secret_value( SecretId='<aws-secrets-name>', SecretString='<secrets-string>' ) 2. import pendulum 3. mocking aws athena

October 6, 2020 thinking about creating a separate task for each repo we have. each repo usually stores the code for its lambda execution code. …

October 5, 2020 upon reading documentation, adding a schedule for your flow is as easy as: import datetime from prefect.schedules import IntervalSchedule schedule = IntervalSchedule(interval=datetime.timedelta(minutes=2)) flow = Flow('workflow-name', schedule) similarly adding a cron schedule would be: from prefect.schedules import CronSchedule daily_schedule = CronSchedule("0 8 * * *") flow = Flow('workflow-name', daily_schedule)…

reading multiple s3 files into pyspark file_paths = '<bucket-name>/<filepath>/<object-name>1.json,<bucket-name>/<filepath>/<object-name>2.json' file_paths_list = [f"s3a://{file}" for file in file_paths.split(',')] df = spark.read.format("json").load(queued_files_list) # or df_merged_schema = spark.read.option("mergeSchema", "true").format('json').load(queued_files_list) 2. display spark sql display(spark.sql("""<sql statement>"""))

September 28, 2020 continuing with work i’m doing to complete a proof of concept (poc) using prefect as our dag scheduler install prefect extras: pip install "prefect[aws]" adding secrets: export PREFECT__TASKS__DEFAULTS__MAX_RETRIES=4 read secrets from code: import prefect max_retries = prefect.config.tasks.defaults.max_retries adding aws credentials: export PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS='{"ACCESS_KEY": "<access_key>", "SECRET_ACCESS_KEY": "<secret_access_key>"}'

September 10, 2020 working on a dag scheduler proof-of-concept (POC) using prefect docker run -it prefecthq/prefect:latest i did not have image locally, so docker pulls the latest for me (note this image will bring you directly into python interpreter) docker images to see the prefect image: to run backend server: …