Data stream has started and small steps taken | Readings
01 October 2023
2 minutes to read
I’ve coded little TS this week aside from the Data Stream project. It has been a lot of terraform and deployments. However I’ve been tackling daily issues that arrive and I’ve been successfull to some extenct. So, I’ll consider it a win!
For this week I didn’t get any challenge done. I’m hoping that next week is more slow so that I can do not one but 2 challenges. Oh, and finnally decide if the priority is rust or Java on this section.
In the last week I’ve written a quick prototype to retrieve the data from explore.porto.pt and write it to a sqlite. The API is great and after doing some initial tests on a single stop I’ve easily started extracting data for the entire city. I got an issue where the response must return in less than 10 seconds but I’ve been able to solve it by removing unnecessary data fields. In terms of data analysis I’ve already gotten some interesting info but it clearly needs to be presented on a clean dashboard.
My next steps are:
- Deploy the deno script
- Setup frequent backups or use litefs
- Do visual data analysis. It’s interesting to setup a realtime dashboard with sqlite and another for data analysis using duckdb
No progress on this front sadly. I’ve wanted to pick on a data presentation at Fidel but I’ve found myself doing a lot of tasks. I’ll be hoping to show a bit on wha we can do with duckdb for debugging use cases on next friday though.
Since I came back from vacations I’ve had a a bit of a cold. I’m hoping to restart my training from Monday on. So, the plan is
- 1 soccer game
- 2 swimming sessions (going to try a new pool)
- Daily warm-ups
Readings of the week
Somehow I’ve found myself adding more and more articles to omnivore. After all the reading I’ve gotten myself with a to-read list of 78 articles. With a quick check to the wordCount I feel I might need a whole nont-stop day to clean the list. So I might need to pruning 😅
However I’ve started writting more and more notes which I think is great! To complement this I think it might be interesting to get the notes for each article so that I can review my notes and write some of them on this status update.
- Introducing incremental export from Amazon DynamoDB to Amazon S3 | AWS Database Blog
- Announcing DuckDB 0.9.0 - DuckDB by Mark Raasveldt and Hannes Mühleisen
- High Performance Functions in Rust on RDS PostgreSQL by Nsikan Essien
- TypeScript npm Packages Done Right | liblab Blog
- I’m All-In on Server-Side SQLite · The Fly Blog
- Harlequin: The DuckDB IDE for Your Terminal.
- Announcing Apache Pinot 1.0™ | Apache Pinot™
- Java 21, the Next LTS Release, Delivers Virtual Threads, Record Patterns and Pattern Matching by Michael Redlich
- The easy way to concurrency and parallelism with Python stdlib by Bite Code!
- When deployments are easy, code becomes simpler by Zarar Siddiqi
I'm José Cabeda, a data engineer focused on improving data systems and educating on how to use them. I also do a lot of planning and read as much as I can.