---
title: Arrow
author: David Kane
tutorial:
  id: arrow
output:
  learnr::tutorial:
    progressive: yes
    allow_skip: yes
runtime: shiny_prerendered
description: 'Tutorial for Chapter 22: Arrow'
---

```{r setup, include = FALSE}
library(learnr)
library(tutorial.helpers)
library(tidyverse)
library(arrow)
library(dbplyr, warn.conflicts = FALSE)
library(duckdb)
knitr::opts_chunk$set(echo = FALSE)
knitr::opts_chunk$set(out.width = '90%')
options(tutorial.exercise.timelimit = 2400, 
        tutorial.storage = "local") 

# Despite much experimentation, I could not get the major download to work when
# executed inside of an Exercise block. So, that means we need do at least that
# outside of the tutorial itself. 

# We have the student download the data (and then manipulate it) in the temp
# directory associated with her current R session. This is somewhat convenient
# in that, when she restarts R, all the files are cleaned up. That is good since
# we don't want a bunch of 9 gig files filling up her hard drive. But it is bad
# in that she can't restart her R session while doing this tutorial! Would it be
# better to store the data on her Desktop? Or in a "data" directory, which we
# create within the current working directory?

# Key question: Should the student do most/all of her work in the Console,
# CP/CR'ing along the way or should she do most/all of the work in the Exercise
# code blocks. The former allows for using the real data. The latter could only
# work with our test data. The cool approach would be to provide both! Ideally,
# students would do stuff on their own, but, if they had computer problems or
# were working on Posit Cloud, they could still use our test data. Leave that
# for the next version of the tutorial. For now, we assign everything in the
# Console. But all our test code chunks use our test data.

csv_path <- paste0(tempdir(), "/seattle-library-checkouts.csv")
pq_path <- paste0(tempdir(), "/seattle-library-checkouts")

# Key problem now, for the parquet format section, is that seattle_csv does not
# exist. So, we used this code to create a tiny version of it. Of course, this
# assumes that you have already downloaded the data and that you are currently
# located in the r4ds.tutorial project directory.

# For some reason, you need to collect() before you can sample. Of course, this
# code is only run by hand, when we need to fix the test data. It is annoying
# that we only keep 100 rows, but R CMD check forces us to keep this package as
# small as possible.

# open_dataset(sources = csv_path,
#              col_types = schema(ISBN = string()),
#              format = "csv") |>
#   collect() |>
#   slice_sample(n = 100) |>
#   write_csv(file = "inst/tutorials/22-arrow/data/test.csv")

# We can now create seattle_csv to use in our test code chunks. We can also
# write out, and then work with, parquet files.

seattle_csv <- open_dataset(
  sources = "data/test.csv", 
  col_types = schema(ISBN = string()),
  format = "csv"
)

seattle_csv |>
  group_by(CheckoutYear) |>
  write_dataset(path = pq_path,
                format = "parquet")

seattle_pq <- open_dataset(pq_path)

query <- seattle_pq |> 
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") |>
  group_by(CheckoutYear, CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(CheckoutYear, CheckoutMonth)

```

```{r copy-code-chunk, child = system.file("child_documents/copy_button.Rmd", package = "tutorial.helpers")}
```

```{r info-section, child = system.file("child_documents/info_section.Rmd", package = "tutorial.helpers")}
```

<!-- Maybe do some more exploration of the data at the start. Just diving into dplyr code makes things hard to follow. -->


## Introduction
### 

This tutorial covers [Chapter 22: Arrow](https://r4ds.hadley.nz/arrow.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund. We will learn about the [**arrow**](https://arrow.apache.org/docs/r/) package for working with [Apache Arrow](https://arrow.apache.org/), a cross-language development platform for in-memory and larger-than-memory data. Key functions include [`open_dataset()`](https://arrow.apache.org/docs/r/reference/open_dataset.html) and [`write_dataset()`](https://arrow.apache.org/docs/r/reference/write_dataset.html).

Note that the data used in this chapter is more than 8 gigabytes. This might be too much for your computer. Most on-line tools, like Posit Cloud, will not allow you to download so much data. If this causes you problems, just read the R4DS chapter: [Chapter 22: Arrow](https://r4ds.hadley.nz/arrow.html).

Instead of writing all your answers inside Exercise code blocks, you will do a fair amount of your work in your R session itself and then **c**opy/**p**aste the **c**ommand and **r**esult (CP/CR) as your answer to the question.

## Getting the data
### 

CSV files are designed to be easily read by humans. They’re a good interchange format because they’re very simple and they can be read by every tool under the sun. But CSV files aren’t very efficient: you have to do quite a lot of work to read the data into R. In this tutorial, you’ll learn about a powerful alternative: the [parquet format](https://parquet.apache.org/), an open standards-based format widely used by big data systems.

### Exercise 1

Load the **tidyverse** library.

```{r getting-the-data-1, exercise = TRUE}

```

```{r getting-the-data-1-hint-1, eval = FALSE}
library(...)
```

```{r getting-the-data-1-test, include = FALSE}
library(tidyverse)
```

### 

We’ll pair parquet files with [Apache Arrow](https://arrow.apache.org/), a multi-language toolbox designed for efficient analysis and transport of large datasets. We’ll use Apache Arrow via the the [**arrow**](https://arrow.apache.org/docs/r/) package, which provides a **dplyr** backend allowing you to analyze larger-than-memory datasets using familiar **dplyr** syntax. 

### Exercise 2

Load the **arrow** library.

```{r getting-the-data-2, exercise = TRUE}

```

```{r getting-the-data-2-hint-1, eval = FALSE}
library(...)
```

```{r getting-the-data-2-test, include = FALSE}
library(arrow)
```

### 

As an additional benefit, arrow is extremely fast: you’ll see some examples later in the chapter.

### Exercise 3

At the Console, run `arrow_info()`. CP/CR the first 10 lines or so of the response. 


```{r getting-the-data-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

### 

There is some chance that the answer will look like this:

````
> arrow_info()
Arrow package version: 15.0.1

Capabilities:
               
acero      TRUE
dataset   FALSE
substrait FALSE
parquet   FALSE
...
````

If so, we have a problem. We need the capabilities for at least "dataset" and "parquet" to be TRUE.

### 

If you have this problem, you should try to install the **arrow** package from source with:

````
install.packages("arrow", type = "source", INSTALL_opts = "--no-lock")
````

### 

You want `arrow_info()` to look like this:

````
> arrow_info()
Arrow package version: 15.0.1

Capabilities:
               
acero      TRUE
dataset    TRUE
substrait FALSE
parquet    TRUE
...
````

The package version does not matter, as long as it is later than 15.


### Exercise 4

Load the **duckdb** library

```{r getting-the-data-4, exercise = TRUE}

```

```{r getting-the-data-4-hint-1, eval = FALSE}
library(...)
```

```{r getting-the-data-4-test, include = FALSE}
library(duckdb)
```

### 

Both **arrow** and **dbplyr** provide **dplyr** backends, so you might wonder when to use each. In many cases, the choice is made for you, as in the data is already in a database or in parquet files, and you’ll want to work with it as is. But if you’re starting with your own data (perhaps CSV files), you can either load it into a database or convert it to parquet.

### Exercise 5

Load the **dbplyr** library.

```{r getting-the-data-5, exercise = TRUE}

```

```{r getting-the-data-5-hint-1, eval = FALSE}
library(...)
```

```{r getting-the-data-5-test, include = FALSE}
library(dbplyr)
```

### 

In general, it’s hard to know what will work best, so in the early stages of your analysis we’d encourage you to try both a database and parquet, and then pick the one that works the best for you.


### Exercise 6

Type `tempdir()` and hit "Run Code" to take a look.

```{r getting-the-data-6, exercise = TRUE}

```

```{r getting-the-data-6-hint-1, eval = FALSE}
tempdir()
```

```{r getting-the-data-6-test, include = FALSE}
tempdir()
```

### 

`tempdir()` returns the same directory each time you run it, at least until you restart your R session. To learn more about temporary files and directories, check out `?tempdir`.

### Exercise 7

Type `paste0(tempdir(), "/seattle-library-checkouts.csv")` and hit "Run Code" to examine it. This creates the full path to file called "seattle-library-checkouts.csv" and located in the temporary directory for this R session.

```{r getting-the-data-7, exercise = TRUE}

```

```{r getting-the-data-7-hint-1, eval = FALSE}
paste0(tempdir(), "/seattle-library-checkouts.csv")
```

```{r getting-the-data-7-test, include = FALSE}
paste0(tempdir(), "/seattle-library-checkouts.csv")
```

### 

We will download a dataset of item checkouts from Seattle public libraries, available [online](data.seattle.gov/Community/Checkouts-by-Title/tmmm-ytt6) and give it the name and location of this file. This dataset contains 41,389,465 rows that tell you how many times each book was checked out each month from April 2005 to October 2022.

### Exercise 8

In the R Console (not in an Exercise code block), run:

````
curl::multi_download(
  urls = "https://r4ds.s3.us-west-2.amazonaws.com/seattle-library-checkouts.csv",
  destfiles = paste0(tempdir(), "/seattle-library-checkouts.csv"),
  resume = TRUE)
````

This will download a 9 GB CSV file which will take about 5 minutes. So go check your email (or read more [*R for Data Science*](https://r4ds.hadley.nz/)!) and come back in 5 minutes.

CP/CR. (That is, **c**opy/**p**aste the **c**ommand and **r**esult (CP/CR) as your answer to this Exercise.) 

```{r getting-the-data-8}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

### 

We recommend using `curl::multi_download()` to get very large files as it’s built for exactly this purpose: it gives you a progress bar and it can resume the download if its interrupted. (Sadly, the tutorial blocks our view of the progress bar.)

### Exercise 9

For the rest of this tutorial, you will type many (but not all) the commands into your own Console, and then copy/paste the command/response (CP/CR) as your answer. **Do not quit the tutorial before you are finished.** Doing so will delete your data and you will need to download it again, although your answers will be saved, as usual.

In the Console, run `list.files(tempdir())`. CP/CR.

```{r getting-the-data-9}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

In addition to the "seattle-library-checkouts.csv" file, there should be some other weird files in the temporary directory. Don't worry about them. (We don't really understand them ourselves, but they probably involve R's attempts to manage this directory.)

### Exercise 10

In the Console, create a new variable, `csv_path`, and assign it to `paste0(tempdir(), "/seattle-library-checkouts.csv")`. CP/CR.

```{r getting-the-data-10}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

```{r getting-the-data-10-test, include = FALSE}
csv_path <- paste0(tempdir(), "/seattle-library-checkouts.csv")
```

### 

We just want a more convenient way to store the location/name of the CSV file.

## Opening a dataset
### 

Let’s start by taking a look at the data. At 9 GB, this file is large enough that we probably don’t want to load the whole thing into memory. A good rule of thumb is that you usually want at least twice as much memory as the size of the data, and many laptops top out at 16 GB. This means we want to avoid `read_csv()` and instead use the `arrow::open_dataset()`.


### Exercise 1

Run this code in the Console. CP/CR.

````
seattle_csv <- open_dataset(
  sources = csv_path, 
  col_types = schema(ISBN = string()),
  format = "csv"
)
````

```{r opening-a-dataset-1}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

What happens when this code is run? `open_dataset()` will scan a few thousand rows to figure out the structure of the dataset. The `ISBN` column contains blank values for the first 80,000 rows, so we have to specify the column type to help arrow work out the data structure. Once the data has been scanned by `open_dataset()`, it records what it’s found and stops; it will only read further rows as you specifically request them.

### Exercise 2

For this and later Exercises which will be completed in the Console, you may find it useful to open a temporary R script (`File -> New File -> R Script`). You can write your answer there and then run it in the Console with `Cmd/Ctrl + Enter`. This makes it easy to edit your code, often by adding more lines to the pipe, and then submit the answer to the next question.

<!-- DK: Explain this better? -->


Run `seattle_csv` in the Console. CP/CR.

```{r opening-a-dataset-2}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r opening-a-dataset-2-test, include = FALSE}
seattle_csv 
```

### 

The first line in the output tells you that `seattle_csv` is stored locally on-disk as a single CSV file; it will only be loaded into memory as needed. The remainder of the output tells you the column type that arrow has inputted for each column.

### Exercise 3

In the Console,  pipe `seattle_csv` into `glimpse()`. CP/CR.

This file is big, so computations such as `glipmse()` will take a while to run.

<!-- AC: It may be good to give a bit of a disclaimer that computations will generally take a while since the file's so big. Done -->

```{r opening-a-dataset-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r opening-a-dataset-3-test, include = FALSE}
seattle_csv |>
  glimpse()
```

### 

We can see what’s actually in it with `glimpse()`. This reveals that there are ~41 million rows and 12 columns, and shows us a few values. 


### Exercise 4

Let's determine the total number of checkouts. In the Console, pipe `seattle_csv` to `summarise()`. Within `summarise()`, add the argument `Checkouts = sum(Checkouts)`. 

```{r opening-a-dataset-4}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

```{r opening-a-dataset-4-test, include = FALSE}
seattle_csv |>
  summarise(Checkouts = sum(Checkouts))
```

### 

This is quick, because no computations have been performed. R has just confirmed that this code should work. To run the calculation, you need to use `collect()`.

### Exercise 5

We can start to use this dataset with **dplyr** verbs, using `collect()` to force arrow to perform the computation and return some data. In the Console, add `collect()` to the end of the previous pipe. CP/CR.

```{r opening-a-dataset-5}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r opening-a-dataset-5-test, include = FALSE}
seattle_csv |>
  summarise(Checkouts = sum(Checkouts)) |>
  collect()
```

### 

There were more than 144 millions checkouts.

### Exercise 6

To determine total checkouts by year, add `.by = CheckoutYear` to the call to `summarize()` in the Console. Continue the pipe to `ggplot()`, map `CheckoutYear` to the x-axis and `Checkouts` to the y-axis. CP/CR.

```{r opening-a-dataset-6}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r opening-a-dataset-6-test, include = FALSE}
seattle_csv |>
   summarise(Checkouts = sum(Checkouts), .by = CheckoutYear) |>
   collect() |> ggplot(aes(x = CheckoutYear, y = Checkouts))
```

### 

Thanks to **arrow**, this code will work regardless of how large the underlying dataset is. But it’s currently rather slow, taking several seconds to run. That’s not terrible given how much data we have, but we can make it much faster by switching to a better format.

### Exercise 7

Add the `geom_point()` and `geom_line()` layers. CP/CR

```{r opening-a-dataset-7-ex}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

```{r -test, include = FALSE}
seattle_csv |>
   summarise(Checkouts = sum(Checkouts), .by = CheckoutYear) |>
   collect() |> ggplot(aes(x = CheckoutYear, y = Checkouts)) + geom_point() + geom_line()
```


###

We should have a cool looking graph here displaying checkouts by year

### Exercise 8

Add labels to your graph.

```{r opening-a-dataset-8-ex}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

###

Next we will be dealing with the parquet format.

## The parquet format
### 

To make this data easier to work with, let's switch to the parquet file format and split it up into multiple files. The following sections will first introduce you to parquet and partitioning, and then apply what we learned to the Seattle library data.

### Exercise 1

In the Console, Set `pq_path` to `paste0(tempdir(), "/seattle-library-checkouts")`. CP/CR.

```{r the-parquet-format-1}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

```{r the-parquet-format-1-test, include = FALSE}
pq_path <- paste0(tempdir(), "/seattle-library-checkouts")
```

### 

Like CSV, parquet is used for rectangular data, but instead of being a text format that you can read with any file editor, it’s a custom binary format designed specifically for the needs of big data.

### Exercise 2

In the Console, start a pipe with `seattle_csv`. CP/CR.

Again, instead of just typing everything directly into the Console, you might open a temporary R script and work from there, submitting to the Console with `Cmd/Ctrl + Enter`.

```{r the-parquet-format-2}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r the-parquet-format-2-test, include = FALSE}
seattle_csv
```

### 

Parquet files are usually smaller than the equivalent CSV file. Parquet relies on [efficient encodings](https://parquet.apache.org/docs/file-format/data-pages/encodings/) to keep file size down, and supports file compression. This helps make parquet files fast because there’s less data to move from disk to memory.

### Exercise 3

In the Console, continue the pipe with `group_by()`. Within `group_by()`, add the argument `CheckoutYear`. 

```{r the-parquet-format-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r the-parquet-format-3-test, include = FALSE}
seattle_csv |>
  group_by(CheckoutYear)
```

### 

Parquet files have a rich type system. A CSV file does not provide any information about column types. For example, a CSV reader has to guess whether `"08-10-2022"` should be parsed as a string or a date. In contrast, parquet files store data in a way that records the type along with the data.

### Exercise 4

In the Console, continue the pipe with `write_dataset()`. Within `write_dataset()` add two arguments:  `path = pq_path` and `format = "parquet"`. This will take a minute but it pays off as it makes future operations much faster.

```{r the-parquet-format-4}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

```{r the-parquet-format-4-test, include = FALSE}
seattle_csv |>
  group_by(CheckoutYear) |>
  write_dataset(path = pq_path,
                format = "parquet")
```

### 

Parquet files are “column-oriented”. This means that they’re organized column-by-column, much like R’s data frame. This typically leads to better performance for data analysis tasks compared to CSV files, which are organized row-by-row.

Parquet files are “chunked”, which makes it possible to work on different parts of the file at the same time, and, if you’re lucky, to skip some chunks all together.


### Exercise 5

Let’s take a look at what we just produced. Run this code in the Console. CP/CR.

````
tibble(
  files = list.files(pq_path, recursive = TRUE),
  size_MB = file.size(file.path(pq_path, files)) / 1024^2
)
````

```{r the-parquet-format-5}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

### 

Our single 9 GB CSV file has been rewritten into 18 parquet files. The file names use a “self-describing” convention used by the Apache Hive project. Hive-style partitions name folders with a “key = value” convention, so as you might guess, the `CheckoutYear = 2005` directory contains all the data where `CheckoutYear` is `2005`. Each file is between 100 and 300 MB and the total size is now around 4 GB, a little over half the size of the original CSV file. This is as we expect since parquet is a much more efficient format.

## Using dplyr with arrow
### 

### Exercise 1

Now that we’ve created these parquet files, we’ll need to read them in again. We use `open_dataset()` again, but this time we give it a directory. 

In the Console, assign `seattle_pq` to `open_dataset()` with the argument `pq_path` inside. CP/CR.

```{r using-dplyr-with-arrow-1}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

```{r using-dplyr-with-arrow-1-test, include = FALSE}
seattle_pq <- open_dataset(pq_path)
```

### 

As datasets get larger and larger, storing all the data in a single file gets increasingly painful and it’s often useful to split large datasets across many files. When this structuring is done intelligently, this strategy can lead to significant improvements in performance because many analyses will only require a subset of the files.

### Exercise 2

Run `seattle_pq` at the Console. Again, since we are building a pipe, you probably want to write `seattle_pq` in a temporary R script and then send it to the Console with `Cmd/Ctrl + Enter`. However you do it, CP/CR.

```{r using-dplyr-with-arrow-2}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r using-dplyr-with-arrow-2-test, include = FALSE}
seattle_pq
```

### 

There are no hard and fast rules about how to partition your dataset: the results will depend on your data, access patterns, and the systems that read the data. You’re likely to need to do some experimentation before you find the ideal partitioning for your situation.

### Exercise 3

Continue the pipe with `filter()`. Within `filter()`, add two arguments: `CheckoutYear >= 2018` and `MaterialType == "BOOK"`. Make sure to separate them with a comma. Send to the Console. CP/CR.

```{r using-dplyr-with-arrow-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r using-dplyr-with-arrow-3-test, include = FALSE}
seattle_pq |> 
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") 
```

### 

As a rough guide, arrow suggests that you avoid files smaller than 20MB and larger than 2GB and avoid partitions that produce more than 10,000 files. You should also try to partition by variables that you filter by; as you’ll see shortly, that allows arrow to skip a lot of work by reading only the relevant files.


### Exercise 4

Continue the pipe with `group_by()`. Within `group_by()`, add the arguments `CheckoutYear, CheckoutMonth`.

```{r using-dplyr-with-arrow-4}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r using-dplyr-with-arrow-4-test, include = FALSE}
seattle_pq |> 
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") |>
  group_by(CheckoutYear, CheckoutMonth)
```

### 

Writing dplyr code for arrow data is conceptually similar to **dbplyr**: you write **dplyr** code, which is automatically transformed into a query that the Apache Arrow C++ library understands, which is then executed when you call `collect()`.


### Exercise 5

Continue the pipe with `summarize()`. Within `summarize()`, add `TotalCheckouts = sum(Checkouts)`. Send to the Console. CP/CR.

```{r using-dplyr-with-arrow-5}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r using-dplyr-with-arrow-5-test, include = FALSE}
seattle_pq |> 
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") |>
  group_by(CheckoutYear, CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts))
```

### 

This will create a new column called `TotalCheckouts` that is the sum of checkouts for that year/month. 

### Exercise 6

Continue the pipe with `arrange()`. Within `arrange()`, add the argument `CheckoutYear, CheckoutMonth`. Send to the Console. CP/CR.

```{r using-dplyr-with-arrow-6}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r using-dplyr-with-arrow-6-test, include = FALSE}
seattle_pq |> 
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") |>
  group_by(CheckoutYear, CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(CheckoutYear, CheckoutMonth)
```

### 

This will arrange the data so that the latest checkout year/month is at the top of the tibble.


### Exercise 7

Assign the whole pipe to `query`. That is, in your temporary R script, you should have:

````
query <- seattle_pq |> 
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") |>
  group_by(CheckoutYear, CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(CheckoutYear, CheckoutMonth)
````

Send this to the Console. CP/CR.

```{r using-dplyr-with-arrow-7}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

```{r using-dplyr-with-arrow-7-test, include = FALSE}
query <- seattle_pq |> 
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") |>
  group_by(CheckoutYear, CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(CheckoutYear, CheckoutMonth)
```

### 

Writing **dplyr** code for arrow data is conceptually similar to doing the same for **dbplyr**: you write **dplyr** code, which is automatically transformed into a query that the Apache Arrow C++ library understands, which is then executed when you call `collect()`. If we print out the query object we can see a little information about what we expect Arrow to return when the execution takes place.

### Exercise 8

In the Console, run `query`. CP/CR.

```{r using-dplyr-with-arrow-8}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

```{r using-dplyr-with-arrow-8-test, include = FALSE}
query
```

### Exercise 9

Pipe `query` to `collect()` in the Console. CP/CR.

```{r using-dplyr-with-arrow-9}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 10)
```

```{r using-dplyr-with-arrow-9-test, include = FALSE}
query|>
  collect()
```

### 

Like **dbplyr**, arrow only understands some R expressions, so you may not be able to write exactly the same code you usually would. However, the list of operations and functions supported is fairly extensive and continues to grow; find a complete list of currently supported functions in `?acero`.

### Exercise 10

Let’s take a quick look at the performance impact of switching from CSV to parquet. First, let’s time how long it takes to calculate the number of books checked out in each month of 2021, when the data is stored as a single large CSV. Run this code in your Console.

````
seattle_csv |> 
  filter(CheckoutYear == 2021, MaterialType == "BOOK") |>
  group_by(CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(desc(CheckoutMonth)) |>
  collect() |> 
  system.time()
````

CP/CR.

```{r using-dplyr-with-arrow-10}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 8)
```

```{r using-dplyr-with-arrow-10-test, include = FALSE}
seattle_csv |> 
  filter(CheckoutYear == 2021, MaterialType == "BOOK") |>
  group_by(CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(desc(CheckoutMonth)) |>
  collect() |> 
  system.time()
```

### 

This is slower than we would like. But switching to paquet files will help because partitioning improves performance because this query uses `CheckoutYear == 2021` to filter the data, and arrow is smart enough to recognize that it only needs to read 1 of the 18 parquet files.


### Exercise 11

Now let’s use our new version of the dataset in which the Seattle library checkout data has been partitioned into 18 smaller parquet files. Run this code in your Console.

````
seattle_pq |> 
  filter(CheckoutYear == 2021, MaterialType == "BOOK") |>
  group_by(CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(desc(CheckoutMonth)) |>
  collect() |> 
  system.time()
````

CP/CR.

```{r using-dplyr-with-arrow-11}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 8)
```

```{r using-dplyr-with-arrow-11-test, include = FALSE}
seattle_pq |> 
  filter(CheckoutYear == 2021, MaterialType == "BOOK") |>
  group_by(CheckoutMonth) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(desc(CheckoutMonth)) |>
  collect() |> 
  system.time()
```

### 

The parquet format improves performance by storing data in a binary format that can be read more directly into memory. The column-wise format and rich metadata means that arrow only needs to read the four columns actually used in the query (`CheckoutYear`, `MaterialType`, `CheckoutMonth`, and `Checkouts`).

This massive difference in performance is why it pays off to convert large CSVs to parquet!

### Exercise 12

There’s one last advantage of parquet and arrow — it’s very easy to turn an arrow dataset into a DuckDB database by calling `arrow::to_duckdb()`. Run this code in your Console.

````
seattle_pq |> 
  to_duckdb() |>
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") |>
  group_by(CheckoutYear) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(desc(CheckoutYear)) |>
  collect()
````

CP/CR.

```{r using-dplyr-with-arrow-12}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 8)
```

```{r using-dplyr-with-arrow-12-test, include = FALSE}
seattle_pq |> 
  to_duckdb() |>
  filter(CheckoutYear >= 2018, MaterialType == "BOOK") |>
  group_by(CheckoutYear) |>
  summarize(TotalCheckouts = sum(Checkouts)) |>
  arrange(desc(CheckoutYear)) |>
  collect()
```

### 

The neat thing about `to_duckdb()` is that the transfer doesn’t involve any memory copying, and speaks to the goals of the arrow ecosystem: enabling seamless transitions from one computing environment to another.


## Cleaning up
### 

This step is probably unnecessary since R removes the temporary directory it created for the current session every time it exits. But cleaning things up ourselves can't hurt.

### Exercise 1

Now, we need to remove that 9 GB dataset which we downloaded and all the other files which we created. Press "Run Code".

```{r cleaning-up-1, exercise = TRUE}
unlink(list.files(tempdir(), full.names = TRUE))
```

## Summary
### 

This tutorial covered [Chapter 22: Arrow](https://r4ds.hadley.nz/arrow.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund. We learned about the [**arrow**](https://arrow.apache.org/docs/r/) package for working with [Apache Arrow](https://arrow.apache.org/), a cross-language development platform for in-memory and larger-than-memory data. Key functions included [`open_dataset()`](https://arrow.apache.org/docs/r/reference/open_dataset.html) and [`write_dataset()`](https://arrow.apache.org/docs/r/reference/write_dataset.html).

Read this **arrow** package [vignette](https://arrow.apache.org/docs/r/articles/dataset.html) for an another example of how to work with a large dataset.
 

```{r download-answers, child = system.file("child_documents/download_answers.Rmd", package = "tutorial.helpers")}
```
