Annals of data quality, Volume II

I finally have my full GPS data pipeline operating automatically all the time. GPS points go from phone app to server back to map on phone.

This last mile has been a long time coming. The data systems I build for work, by contrast, all operate without manual intervention basically all of the time, and I make sure we configure them to alert us when something looks fishy. But I guess the cobbler’s children have no shoes; prior to this week I’ve had a manual step in my various GPS data ingestion pipelines since I started tracking locations almost twenty years ago. Manual steps in a data collection pipeline can be completely fine: they’re opportunities to get a feel for the data and what it means, and for the kinds of problems they develop that hurt the quality of the collected dataset. But manual steps require attention, and this week I finally bothered to free myself from them. (I did this the old fashioned way: by writing code without assistance. On the Northeast Regional in this case.)

The whole thing has been a two-decade saga that will get written up in more detail later. For now, enjoy what counts for me as a historic screenshot, the first time I got to enjoy the fruits of a fully automated data pipeline: lunchtime in the middle of a great ski day with the older kid.