For the past weeks, I’ve found some interesting stories related to database migrations like Your legacy database is outgrowing itself and An unlikely database migration. The stacks are very different in each story but shows how challenging a migration is when you need to keep systems up and avoid data loss. These kind of challenges will only increase in number with the move of processes to the digital realm, which foresees an increase of data engineers as stated on We Don’t Need Data Scientists, We Need Data Engineers and in How To Become a Data Engineer.

Adobe has heavily invested in Apache Iceberg as written on Taking Query Optimizations to the Next Level with Iceberg. This table format looks promising, specially for those already using presto/trino and hive.

Some of the reasons to be excited about airflow 2.0 are explained on Airflow 2.0 and Why We Are Excited at Databand.

For the ones working with analysts, SQL is a must-have and even shows some strengths like those in Simple Anomaly Detection Using Plain SQL. However it presents some challenges like linting, which is still a work in progress but is seeing new developments like in sqlfluff.

Be well and stay safe :-)