Meetup potential for data stream | Readings
07 October 2023
2 minutes to read
Tipping a bit again into typescript I’m foreseeing a lot of code for the next 2 weeks. And specially with testing I’ve gotten a lot of fun yesterday between checking components and integration tests. Sincerely if I get to deploy the changes with the same quality of a PR I reviewed this week I’ll feel a lot more confident. At least to move the progression bar to 20%.
Again no challenges done. Luckly this is one of those where I can stick with it for a couple of weeks and get the lost challenges done in a single swoop. But my escuse for having done none this week was due to have focus a bit more on the Data Stream project.
With the first loader prototype done I’ve gotten a proposal to go present in a meetup. For this to work I’ve decided to focus a bit on the capabilities of duckdb. So my plan is to structure the presentation in the following way:
- Explain the problem with porto public transportation
- Show the API
- Show the loaders
- Iceberg? (if done in time). Alternative is to show a glue job to migrate data into Iceberg
- Show duckdb reading the data from the storages. Possibily as live demo
- Show some examples of running queries on all data. How fast it is. Show integrations with python notebooks
- Present some of the insights in the form of the dashboard
In terms of progress I truly would like to have the loaders set using either spark or flink but I’ll leave the finished data pipeline to after the presentation (middle of the November). From all the rest I’m thinking that I should have 80% of “Data Stream” done.
I’m presenting next week internally on some of the capabilities of Duckdb to my company. Some of this insights will be usefull for the presentation on November. The focus however will be on the meetup talk.
Gotten back on track with the soccer game but not much luck with the other two. I’m getting aa kind of blocker to return so I’m thinking of doing 2 things.
In the morning I’ll just go for a quick walk before starting to work. During lunch this week I’ll pick on my bike and go visit the swimming pool to check how busy it is. I currently don’t have the car available so my only solution is to go do the workout during lunch with my bike.
Readings of the week
I’ve gotten more articles in the list than actually read. Luckly I also think I need to do a bit of trimming again as I added them with almost no criteria.
- 184.108.40.206 lookup failures on October 4th, 2023 by Ólafur Guðmundsson
- Visual Studio Code September 2023 by Microsoft
- Honey, I shrunk the npm package · Jamie Magee
- ClickHouse Keeper: A ZooKeeper alternative written in C++ by ClickHouse
- Eight years of organizing tech meetups | notes.eatonphil.com
- An Overview of Testing Options for dbt (data build tool) - Datacoves
Archiving articles with newsletter label
I'm José Cabeda, a data engineer focused on improving data systems and educating on how to use them. I also do a lot of planning and read as much as I can.