Let’s face it, a lot of a data engineer’s time is spent waiting to see if things executed as expected or for data to be refreshed; We write pipelines, buy expensive replication software, or sometime manually move files (I hope we still aren’t in this day and age), and in the end all of this has a cost associated with it when working in a cloud environment. In the case of Databricks jobs, we often find ourselves creating clusters just to move data, where the cluster lays dormant for the most part during these extractions. In my eyes, that’s wasteful and could probably be improved upon.
Continue readingTag Archives: Python
Foosball Scoring with Raspberry Pi
If you know me, you know besides corgis I have another passion and that’s for foosball. I might not be tournament level good, but I can keep up with my friends and in the bar scene. I had an old sporting goods store level table when I was in high school and got rid of it when I moved out of my folk’s house. I swore I’d never buy another one because no one would ever come over and play and every once in a while, during my adult life, I’d see a nice one pop up on Craigslist for a good price. I always passed up the chance of getting a new one. Until one day when I saw a Tornado going for around 400 bucks. I couldn’t pass up the deal. A few weeks after getting it setup in our sunroom, I was going through some boxes and found an old Raspberry Pi that I used to use for playing emulated games on. I thought to myself “what would be a good project to use this on?” and then it dawned on me. I should automate a scoreboard for the foosball table.
Google Geocoding API with Spark
A couple of days ago while browsing Reddit, I came across someone asking if anyone had used the Google Geocoding API and how they went about doing it. Having recently done it, I offered by assistance, but also felt compelled to write up a blog post. Because of course like most things Spark or API related, there isn’t much out there in terms of actual examples. So here’s me effort in trying to share how we went about adding geocoding to our dataframes for addresses or lat/longs.
Continue reading