weekly misc items: August 31, 2020

1. Assigning pyspark ArrayType column with null values if there is empty array

df_orders = df_orders.withColumn("refunds", when((size(col("refunds")) == 0), lit(None)).otherwise(col("refunds")))

2. databricks notebook exit

was looking for a way to stop databricks from running additional commands in a notebook after an exception is raised or a condition is met, found:

dbutils.notebook.exit("exiting notebook")

so you can have a try/except like this:

try:
    print(4/l)
except:
    dbutils.notebook.exit("exiting notebook")

the result will be:

3. remove folder from git tracking

git rm -r --cached <path-to-folder>

if you need to add it back, just use git add