These levels have been defined by the software carpentry people, and I have modified them to this:

  • beginner: You have just started out in this topic. You do not yet know how things are supposed to work. You do not have a mental model of this thing
  • intermediate: You are a regular user of this software/tool/concept, you have a mental model, but it is not very sophisticated
  • advanced: You have a sophisticated mental model how things work, and you even know when the model breaks, when it does not match reality.

Are you a Fearless Deployer?

Fast experimentation and confident deployments should be your goal

Are you a Fearless Deployer?

 Posted on November 21, 2022  |  2 minutes  |  422 words  | 

how do you feel when you press the ‘deploy to production’ button? Confident, slightly afraid? I bet many data scientists find it a bit scary. It’s worth it to dig a bit deeper into this fear. In my ideal world we are not scared at all. We have a devops mindset. We have no anxiety, no fears at all. You should be confident that the deployment pipeline takes care of everything. [Read More]
deployment  iteration  MLOps  training-pipeline 

Should I Move to a Database?

 Posted on November 8, 2021   (Last modified on November 9, 2022)  |  15 minutes  |  2986 words  | 

Long ago at a real-life meetup (remember those?), I received a t-shirt which said: “biggeR than R”. I think it was by microsoft, who develop a special version of R with automatic parallel work. Anyways, I was thinking about bigness (is that a word? it is now!) of your data. Is your data becoming to big? big data stupid gif Your dataset becomes so big and unwieldy that operations take a long time. [Read More]
dataframe  database  python  duckdb  sqlite  pandas  polars 

Where does the output of Rscript go?

stdin, stdout, stderr

 Posted on April 14, 2020   (Last modified on November 9, 2022)  |  4 minutes  |  713 words  | 

We often run R interactively, through Rstudio or in the terminal. But you can also run Rscripts without manual intervention. Using Rscript. But where does the output go? Warning: This post is very linux/unix (macos) centred, I don’t know how this works in Windows. Also I’m using the standard shell in linux ‘bash’ I believe there are some small nuances in the commands in other shells like zsh. Why do I want to know this? [Read More]
rscript  standard streams  errors  logging 

Adding logging to a shiny app with loggit

 Posted on February 28, 2018   (Last modified on November 9, 2022)  |  2 minutes  |  371 words  | 

This is a very short post with example code Over time when you move your shiny app from your computer to a server, you want to add some logging. Generally logging is defined in levels : INFO (everything you want to print), WARNING (it does not stop the application, but it could be a problem), and ERROR (fatal things). Shiny server does already log all it’s actions to a file on the server, but that file can be hard to access. [Read More]
shiny  shiny-server  logging  loggit 

Setting up a version controlled shiny-server

 Posted on February 6, 2018   (Last modified on November 9, 2022)  |  3 minutes  |  612 words  | 

Last week I set up a shiny server, it was relatively easy! But I wanted something more, a way to make changes on my local computer and push it to the server. Shiny server (I used the open source version) has multiple installers provided by RStudio. The installers for shiny-server create a user shiny and installs all the services needed. I used a guide specific for my version of linux to install shiny-server and combined it with two other guides to make it version controlled. [Read More]
shiny  shiny-server  git  githook 

Adding bananas from the commandline (extending the oomsifier)

 Posted on December 29, 2017   (Last modified on November 9, 2022)  |  3 minutes  |  581 words  | 

Sometimes you just want to add bananas from the commandline. Previously I created a small script that takes an image and adds a dancing banana to the bottom left of the image. I wanted to make an API too, but that will have to wait till next year. Today we will create a commandline script that will do the same thing. With the excellent explanation in Mark Sellors' guide I have now created a cmdline thingy in very few steps. [Read More]
magick  example_function  hack  banana  gif 

Non-standard-evaluation and standard evaluation in dplyr

 Posted on June 13, 2016   (Last modified on November 9, 2022)  |  6 minutes  |  1069 words  | 

THIS POST IS NO LONGER ENTIRELY RELEVANT. DPLYR 0.7 HAS A SLIGHTLY DIFFERENT (AND SLIGHTLY MORE INTUITIVE) WAY OF WORKING WITH NON-STANDARD EVALUATION. EDIT: IT IS COMPLETELY DIFFERENT NOW. AND I RECOMMEND https://dplyr.tidyverse.org/. I love the dplyr package with all of its functions, however if you use normal dplyr in functions in your package r-cmd-check will give you a warning: R CMD check NOTE: No visible binding for global variable NAME OF YOUR VARIABLE 1. [Read More]
dplyr  NSE  optimize-your-code  data:duo2015  data:mtcars  lazyeval 

Your most valuable collaborator, future-you

You don’t want future-you to curse past-you

 Posted on May 26, 2016   (Last modified on November 9, 2022)  |  9 minutes  |  1824 words  | 

I was recently at a R users meetup where Hadley Wickham talked about data wrangling. He showed some interesting stuff! Did you know that you can put a data frame into a data frame? You can make a list of data frames and add that list to your data frame. Very cool, and more useful then I thought, but that is not what I wanted to talk about. I would like to give you some tips about working with someone you will probably work with in the future. [Read More]
version control  TDD  readme  functional programming