was updating a stack in aws cloudformation and got a CREATE_FAILED with error: Invalid parameter: TopicArn (Service: AmazonSNS; Status Code: 400; Error Code: InvalidParameter; Request ID: <request-id>; Proxy: null) learned that if you provide the wrong topic ARN, it can cause this error. we are using python’s troposphere library to…

trying to track down the reason for failed jobs run this morning. typically i look at logs through CloudWatch/Logs/Log groups, but we see that AWS wants us to start using CloudWatch Logs Insights: i’m documenting this process so that i can also ensure i have a clear understanding on how…

For whatever reason, my thumb drive got partitioned into several pieces making it hard to use. In windows you can do the following: open the windows command prompt type diskpart , it opens a separate command window type list disk identify your removable disk, you can confirm by going into…

October 20, 2020 trying out prefect cloud features to determine deployment strategy for productionalizing workflows logged into prefect cloud created a new project in the Team section: then Projects :

mocking lambda invoke from moto import mock_iam, mock_lambda def get_role_name(): with mock_iam(): iam = boto3.client("iam", region_name=_lambda_region) try: return iam.get_role(RoleName="my-role")["Role"]["Arn"] except ClientError: return iam.create_role( RoleName="my-role", AssumeRolePolicyDocument="some policy", Path="/my-path/", )["Role"]["Arn"]

noting here for reference — moto for mocking aws secrets manager from moto import mock_secretsmanager @pytest.fixture @mock_secretsmanager def mock_response(): mock_secretsmanager().start() sm_client = boto3.client('secretsmanager') response = sm_client.put_secret_value( SecretId='<aws-secrets-name>', SecretString='<secrets-string>' ) 2. import pendulum 3. mocking aws athena

October 6, 2020 thinking about creating a separate task for each repo we have. each repo usually stores the code for its lambda execution code. …

October 5, 2020 upon reading documentation, adding a schedule for your flow is as easy as: import datetime from prefect.schedules import IntervalSchedule schedule = IntervalSchedule(interval=datetime.timedelta(minutes=2)) flow = Flow('workflow-name', schedule) similarly adding a cron schedule would be: from prefect.schedules import CronSchedule daily_schedule = CronSchedule("0 8 * * *") flow = Flow('workflow-name'…

reading multiple s3 files into pyspark file_paths = '<bucket-name>/<filepath>/<object-name>1.json,<bucket-name>/<filepath>/<object-name>2.json' file_paths_list = [f"s3a://{file}" for file in file_paths.split(',')] df = spark.read.format("json").load(queued_files_list) # or df_merged_schema = spark.read.option("mergeSchema", "true").format('json').load(queued_files_list) 2. display spark sql display(spark.sql("""<sql statement>"""))