Samuel Castillo wrote this Shiny application in January 2018, during his mathematics studies at UMass Amherst. It takes a window of daily Bitcoin closing prices and marks the days when the close jumps away from the path an ARIMA model expects. The hosted detector still lives on shinyapps.io. This page is the walk-through, and the knitted note next to it keeps the original plots.

The series that enters the model is the natural log of the closing price. On a log scale a move is a relative move, and the scatter does not grow just because the dollar level is higher. Over the dates you choose, forecast::auto.arima selects an ARIMA model for that log series: the differencing order, then the autoregressive and moving-average orders, ranked by the usual information criteria. A short window in this project often lands on a simple structure after differencing, such as an AR(1), an MA(1), or an ARMA(1,1). A longer window can keep a richer order, and that changes which days still look unusual.

A day is flagged when the absolute residual from that fit exceeds the sensitivity you set. The residual is the gap, on the log-price scale, between the observed close and the value fitted by the ARIMA model. The slider runs from 0.04 to 0.20 in steps of 0.01 and starts at 0.10, so the default rule marks a log-residual larger than one tenth. A lower cutoff marks more days. A higher cutoff keeps the sharper breaks. The app opens on 17 November 2017 through 15 January 2018.

Three tabs read those flags. Anomaly Dates plots the dollar close and draws a red point on each flagged day, with the point scaled by the residual. Detailed Charts takes one flagged date and draws a BTC-USD candlestick chart with Bollinger bands (a 5-day simple moving average, two standard deviations) on the twenty days before and after that date. Data Table lists the event date, the dollar change, and the percent change, and that table downloads as a CSV.

The two pictures of a “band” in the repository are built differently, and it helps to keep them straight. In the knitted note, the ribbon is the fitted log price plus or minus 1.96 residual standard deviations, the usual normal 95% band on the log scale, and the example cutoff is α = 0.01 on 17 June 2017 through 18 August 2017. In the Shiny plot, the shaded ribbon is a loess smooth of an envelope around the dollar price: the close plus or minus five times (α times the standard deviation of the dollar residual, plus the mean of that residual). In both views the red points are the days that failed the residual cutoff. The band is context. The cutoff is the decision.

Key points

Open the Shiny app Worked example Source

Using the app Procedure Worked example Repository

Using the app Shiny

The controls are the date window and one sensitivity number. The original instructions still hold.

  1. Choose an evaluation period in Date Range. The picker allows dates from 1 January 2009 through yesterday, and it opens on 17 November 2017 through 15 January 2018.
  2. Set a risk tolerance with Sensitivity. This is the residual cutoff α.
  3. Read the days when the Bitcoin close breaks that cutoff. Switch among Anomaly Dates, Detailed Charts, and Data Table.
  4. Download the event table as a CSV if you want the flagged dates outside the app.

Procedure ARIMA

Three steps, as the 2018 note lays them out. The implementation details below follow the Shiny server and the knitted example.

  1. Make the series workable. Take the natural log of the adjusted close so the variance is steadier, then let the model difference away the trend. The knitted note also plots the first difference of the log close so you can see that stationary-looking series. auto.arima chooses the integer differencing order itself.
  2. Fit the ARIMA model. On the logged window, auto.arima searches AR and MA orders and keeps a model by its information criterion. The original write-up is frank about the fit: these models are rough, and they only have to be good enough to show which days refuse to sit on the fitted path. Heteroscedasticity, one-time breaks, and extra predictors are left outside this student version.
  3. Classify the outliers. Compute the residuals of the log-price model. If the absolute residual is above α, the day is an anomaly. In the note, α = 0.01. In the app, α is the slider, default 0.10.

The original note also describes the stationary step as a fractional difference, an ARFIMA idea, before the ARMA fit. The code that ships here calls auto.arima on the log close, so the differencing order it selects is an integer. The flag is still the absolute residual of that fitted model.

Worked example 2017

Same rule, one fixed window, with the plots kept in the page.

The knitted page Bitcoin Anomaly Detection Example pulls BTC-USD from 1 January 2017 through 23 January 2018 and then studies 17 June 2017 through 18 August 2017 at α = 0.01. It shows four pictures in order: the dollar close, the log close, the differenced log close, and the fitted log series with the 1.96-sigma ribbon and the red anomaly points. Source for that note is example_for_article.Rmd.

Repository R

The Shiny app, the note, and both price files stay in the project root so GitHub Pages can publish this site from the default branch.