Some Thoughts About dbt for Data Engineering

Over the last week I have experimented with dbt (data built tool), a cmdline tool created by Fishtown-analytics. I’m hardly the first to write or talk about it (see all the references at the bottom of this piece). But I just want to record my thoughts at this point in time. What is it Imagine the following situation: you have a data warehouse where all your data lives. You as a data engineer support tens to hundreds of analysts who build dashboards and reports on top of that source data. [Read More]

Setting up a version controlled shiny-server

Last week I set up a shiny server, it was relatively easy! But I wanted something more, a way to make changes on my local computer and push it to the server. Shiny server (I used the open source version) has multiple installers provided by RStudio. The installers for shiny-server create a user shiny and installs all the services needed. I used a guide specific for my version of linux to install shiny-server and combined it with two other guides to make it version controlled. [Read More]

Introduction to R projects

It often makes sense to separate your projects. And since space is cheap you are probably creating separate folders on your computer. In RStudio you can create different projects that live in their own folder. When you start a different project the files of that project work independently from other projects. And the standard locations of your workspace and other things are also separated from the rest. In my case, for example, I have several projects and the last 10 or so are displayed in the dropdown menu: [Read More]

Version control with Git

Keeping track of versions You work on a project and would like to keep track of what you did. That is why keep old versions of your files. That way you can go back if you messed up beyond recognition. Usually that looks like this: Or you use dropbox or something like it: Other people use email. Emailing to themselves or to collaborators when they finished something. [Read More]