Mark's Dev Blog

Coding Career Advice: Using Git for Version Control Effectively

Wed, 27 Jan 2021 10:00:00 -0500

Introduction

Git has become the standard tool for software development version control. Other VCS tools exist, and some work better than Git for certain scenarios, but most of today's development world relies on using Git. So, becoming comfortable with Git and knowing how to use it effectively is a key skill for any software developer.

I'd like to pass along some of the most useful Git concepts and tips that I've learned over the last few years. In addition, I've covered background info on how Git works and common operations, and there's some specific usage patterns I've found to be especially valuable when working with a team and trying to understand a codebase.

As usual, none of the info or advice in this post is completely new or original, and there's many other sites that cover the same topics (and probably explain them better). I'm just trying to provide an overview of the relevant material and provide enough details that you can do further research and learning from there.

This post is largely based on my slideset Git Under the Hood: Internals, Techniques, and Rewriting History, and I talked about rewriting repo history in my post Rewriting Your Git History and JS Source for Fun and Profit

Git Fundamentals

Git is notoriously difficult to work with, especially using the command line. The CLI commands and options are confusing, mismatched, and hard to remember. There's phrases and warnings like "detached HEAD". Git, frankly, is not easy to learn and kinda scary.

The good news is that once you understand how Git works, it becomes an extremely powerful tool that offers a lot of flexibility.

Git Terms and Concepts Overview

While I'm not going to turn this into a complete "Git tutorial from scratch", it's worth reviewing some of the key concepts.

Git Basics

Git is a tool for tracking changes to file content over time. A Git repository is a folder that has a .git folder inside. The .git folder contains all the metadata and stored history of the project's changes.

The working copy is all other folders and files in the repository folder that Git is storing and tracking. Any newly created files start out untracked. Git knows that the files are there, but you haven't told Git to save them.

To tell Git to start tracking a file, you add the file (git add some-file). Git then saves a copy of the file in an internal section called the staging area. Staged files are not being saved permanently, yet. Instead, they represent the set of files and contents that will be saved when you actually tell Git to save them.

Once you've added one or more files to the staging area, you can save them by committing them. "Commit" is both a verb and a noun here: we "commit" files to save them, and every time we save them, we make a "commit".

Git commits contain a certain set of files and their contents, at a specific point in time. They also contain metadata, including the author's name and email address, and a commit message that you write to describe the changes that were saved.

After a file has been added at least once, making further changes to that file will cause Git to mark it as modified. That means that Git knows the contents are different, but you haven't told Git to save the new changes yet. Once you add that file to the staging area again, Git sees that its latest copy of the file is the same as what's on disk, so it describes the file as unchanged.

Each Git repository folder is standalone. However, Git repositories can be shared across folders, computers, and networks, allowing developers to collaborate on the same codebase. A Git repo can be configured with the URL of another repo, allowing the two repos to send commits back and forth. Each URL entry is called a remote. Downloading commit data from a remote repo is a fetch or a pull (with slight differences in behavior), and uploading commit data from local to remote is a push. Downloading a complete repo from scratch is making a clone of that repo.

Repositories normally have a default remote repo they point to, called the origin. Whenever you clone a repo, the new local repo points to the remote source as the origin, but that entry can be changed later. Repos can be configured to talk to many other repos at once, and can push and pull data from any remote.

Branches

Git commits are tracked using branches. A branch is like a pointer to the latest commit in a specific series of commits. Any time you make a new commit, Git bumps that branch pointer to point to the newest commit. You can make many branches within a repo, and most devs create a new branch for each task they work on. You can also make tags, which also point to a specific commit, but don't get moved or changed automatically. Tags are normally used to identify checkpoints and releases, so you can easily jump back and see how the code was at that point in time.

Changes from multiple branches can be brought together using a merge process. If some of the changes apply to the same lines of code, there is a merge conflict, and it's up to you as the developer to look at the mismatched changes and resolve the conflict by picking out what the right contents are.

Historically, most repos use a branch called master as the primary development branch. More recently, the community has started switching to use a primary branch named main instead. But, you can configure Git to use any branch name as the "default development branch" if you want.

Git uses the term checking out to refer to updating the working copy files on disk, based on previously committed values. Typically you check out a branch, which overwrites the files on disk to match the files as they exist in the latest commit of the branch. However, you can check out other versions of files as well

Uncommitted changes can be copied and saved for later by creating a stash. A stash is kind of like an unnamed commit - it again points to specific files at a certain point in time, but it doesn't exist in a branch. Stashed changes can later be applied on top of your working copy.

Overall, the Git data workflow looks like this:

Understanding Git Internals

I really feel that understanding Git's internal data structures is critical to understanding how Git works and how to use it correctly.

Git tracks all content using SHA1 hashes of byte data. Running any specific sequence of bytes through the hashing function calculates a specific hex string as a result:

from hashlib import sha1

sha1("abcd").hexdigest()
'81fe8bfe87576c3ecb22426f8e57847382917acf'

sha1("abce").hexdigest()
'0a431a7631cabf6b11b984a943127b5e0aa9d687'

readme = open("README.md", "rt").read()
sha1(readme).hexdigest()
'45257c0245c56a4d5990827b044f897c674c8512'

Git hashes files and data structures, then stores them inside the .git folder based on the hash:

/my-project
  /.git
    /objects
      /00
      /01
      ...
      /81
        81fe8bfe87576c3ecb22426f8e57847382917acf
      /82
      ...
      /fe
      /ff

Git has three primary internal data structures:

blobs are file contents, and identified by a hash of the file's bytes
file trees associate folder and file names with file blobs, and are identified by a hash of the file tree data structure
commits contain metadata (author, timestamp, message), point to a specific file tree, and are identified by a hash of the commit data structure

Type	Contains	Identified By
Blob	File contents	Hash of the file's bytes
File tree	Associates names and folder definitions with file blobs	Hash of the file tree data structure
Commit	Metadata for author, commit timestamps, and message	Hash of the commit data structure

A file tree may point to multiple other file trees for subfolders:

Commit objects themselves form a linked list, which points backwards to earlier commits based on their hashes: A <- B <- C <- D.

A Git "ref" is a name label that points to a specific commit. Branches are names associated with a given ref, where each time a new commit is made, the ref is updated to point to that latest commit. So, you can start from the branch ref pointer, then walk backwards through the chain of commits to see the history.

HEAD is a ref that points to "whatever the current active commit" is. Normally this is the same as the current branch pointer, but if you check out a specific earlier commit, you get the ever-popular warning about a "detached HEAD". This just means that HEAD is pointing to a specific commit instead of a branch, and if you make any new commits, they won't be part of any branch.

Because commits are a linked list based on hashes, and the hashes are based on byte contents of files and other structures, changing any one bit in an earlier commit would have a ripple effect - every hash of each commit after that would be different.

Git commit objects are immutable - once created, they cannot actually be changed. This means that you can't change history, exactly - you can only create an alternate history.

Git Tools

I've seen a lot of arguments about whether it's better to use a Git GUI tool, or use Git from the command line. To those people, I say: why not both? :)

I find that having a Git GUI tool is absolutely invaluable. It makes visualizing the state of the repository and its branches much easier, and many operations are way simpler via a GUI. For example, I can view the diffs for many pieces of a file at once, and selectively add specific changes to the staging area by clicking "Add Hunk" or CTRL-clicking a few lines to select them and clicking "Add Lines". This is much simpler and more intuitive than trying to use Git's "patch editing" text UI to manipulate pieces of changes. Interactive rebasing is also much easier to do via a GUI. I can't remember what the different options like "pick" mean, but it's straightforward to use a GUI listview with arrow buttons that lets you reorder commits and squash them together.

On the other hand, it's often faster to create or switch branches from the CLI. You can add all changed files to the staging area with a single command of git add -u. And of course, if you are using a remote system via SSH, you probably do only have the Git CLI available.

So, I use both a Git GUI, and the CLI, based on what tasks I'm doing.

I primarily use Atlassian SourceTree (Win, Mac). It's very powerful, with a lot of options, and has a good built-in UI for interactive rebasing. It also happens to be free. The biggest downside is that it doesn't have a way to view the contents of the repo file tree as of a given commit.

Other Git tools I've used in some form include:

Git Extensions for Windows (Win): integrates with Windows Explorer to let you perform Git operations from the filesystem. I mostly use this to do a quick view of a given file's history if I happen to be browsing the folder contents of the repo.
Git Fork (Win, Mac): excellent UI design, and does have an interactive rebase UI. Recently switched from being free to $50, but likely worth paying for.
Sublime Merge (Win, Mac, Linux): from the makers of Sublime Text. Fewer options and tool integrations, but very snappy. Tells you what CLI operations it's doing when you try to push or pull, so it expects familiarity with the CLI. $100, but will run for a while with nag messages.

There's also Tower (Win, Mac) and Git Kraken (Win, Mac, Linux), which have slick UIs but require yearly subscriptions, and a laundry list of other smaller Git GUIs. There's even "text-based UI" tools like lazygit, gitui, and bit.

All major IDEs have Git integration. JetBrains IDEs like IntelliJ and WebStorm have excellent Git capabilities. VS Code has adequate Git integration, but really needs additional extensions like Git History and GitLens to be useful.

I also really prefer using external diff tools for comparing complete files, or fixing merge conflicts. I personally use Beyond Compare as my external diff tool, and DiffMerge as my conflict resolution diffing tool.

Git Techniques

Improving CLI Logging

The default git log output is ugly and hard to read. Whenever I start using Git on a new machine, the very first thing I do is browse to https://coderwall.com/p/euwpig/a-better-git-log and copy-paste the instructions for creating a git lg alias to set up a much pretter CLI logging view that shows the branch and commit message history:

git config --global alias.lg "log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit"

That gives us this view whenever we run git lg:

Note that git log accepts a variety of filtering options, including text strings, dates, branches, etc.

Preparing Commits in Pieces

I've seen comments that complain that the Git staging area is confusing. To me, the staging area is one of the most valuable features of Git - it lets me carefully craft commits that contain only the code that belongs together.

When I work on a task, I frequently end up modifying multiple files before I'm ready to make a commit. However, the changes might logically belong in several smaller commits instead of one big commit. If I do git add some-file, it adds all the current changes in the file to the staging area. Instead, I often want to stage just a couple sections from file A, and a couple sections from file B, and maybe all of file C, because those are the changes that should go together in one commit.

You can do this from the commandline using the git add -p flag, which brings up a text UI that lets you view each "hunk" of changes in a file, and decide whether to stage that hunk or not. However, I strongly recommend using a Git GUI tool like SourceTree for adding pieces of files, because it's easier to click "Add Hunk" or CTRL-click a couple lines and click "Add Lines" than it is try to decipher what the command abbreviations in the text UI actually mean:

Once you've got these pieces added, you can make a commit with just chose changes, and repeat the process for the next commit. This is a key part of the "making small commits" practice that I cover below.

On the flip side, sometimes you do just want to add everything that's been changed at once. In that case, the fast way is to run git add -u from the command line, which adds all modified files to the staging area.

Stashing Changes

Stashes are most useful when you've got some modified files that aren't committed, and need to set those aside to work on a different branch for a while. Git's list of stashes acts like a stack data structure, but you can also supply names for stash entries when you create them. Creating stash entries normally resets the modified files back to the latest commit, but you can choose to leave the modifications in place.

From the CLI, the main options are:

git stash: save a copy of local changes for later reuse, and clears the working directory/index
- git stash push: creates a new stash entry
- git stash pop applies changes from the top stash entry and removes it
- git stash apply stash@{2}: applies changes from the third stash entry
- git stash -p: choose specific pieces to stash
- git checkout stash@{2} -- someFile: retrieve a specific file contents from the stash

But, this is another situation where it's particularly useful to use a GUI instead. It's easier to just click a "Stash" button in a toolbar and type in a name for the entry to create one, or to expand a "Stashes" section of a treeview, right-click an entry, and "Apply Stash" to apply a stash.

Working with Branches

Creating and Switching Branches

Git has a bunch of different commands for working with branches. The most common way to create a branch is actually with git checkout -b NAME_OF_NEW_BRANCH. That creates a new branch, starting from the latest commit on the current branch, and switches to it.

You can also use git checkout NAME_OF_EXISTING_BRANCH (without the -b flag) to switch to an existing branch.

There's many other branching commands - see the Git docs and other pages like this Git branching cheatsheet for lists of commands and options.

Fetching, Pushing, and Pulling Branches

Most Git network operation commands accept the name of the remote repo to talk to, but assume that you want to talk to the origin remote repo by default if you don't specify a remote name.

git fetch tells Git to contact another repo, and download copies of all commits that the local repo doesn't have stored. This includes information on branches in the remote repo as well.

Once your repo has downloaded the list of remote branches, you can create a local branch based on the remote branch's name, with git checkout NAME_OF_REMOTE_BRANCH. Git will create a new branch that points to the same commit. It also sets up the local branch to track the remote branch, which means that any pushes from the local branch will update the remote branch.

Later, you can update the remote branch with the new commits you made locally, with git push. You can also push local branches taht the remote repo doesn't know about yet.

If the remote branch has commits you don't have in your local branch, git pull will both fetch the set of new commits into your local repo, and update your local branch to contain those commits.

If you rewrite history on your local branch so that it's different than the remote branch, a git push attempt will fail with an error. You can force push , which will hard-update the remote branch to use these commits instead. Force pushing is semi-dangerous, depending on workflow. If someone else pulled the old history, and you force-push, now they have a conflict to deal with. Force pushing is a valuable tool if you need it, and can be a legitimate solution to fixing problems or repeatedly updating a PR, but should be used cautiously. Think of it as a chainsaw - if you need it, you need it, you just have to be very careful when using it :)

Merging Branches

Merging allows you to take changes and history that exist on branch B, and combine them into the changes on your current branch A. The assumption is that both branches have a common set of ancestor commits, and two different sets of changes (either to different files, or even the same files). Merging creates a new "merge commit" on the current branch that has all of the changes together in one spot. This is used to let developers collaborate by writing code separately, but combine their changes together.

Merging is done with git merge OTHER_BRANCH_NAME, which tells Git to merge from the other branch into the current branch.

If the changes on the two branches interfere with each other, there's a merge conflict. Git will mark the file with text strings indicating the two mismatched sections. It's up to you to fix the problem, save the corrected file, add it, and finish the merge commit. I like using SourceGear DiffMerge as a GUI tool for fixing conflicts, but VS Code also does a nice job of highlighting conflict markers in files and offering hover buttons to pick one side or the other.

Feature Branch Strategies

Most teams use some kind of a "feature branch" strategy for development. They have a primary development branch such as main, master, or develop. Any time a developer starts work on a new task, they create a new branch based on the primary branch, and often using the name and ID of a task/issue as the branch name: git checkout -b feature/myapp-123-build-todos-list.

The developer works on their feature for a while. Once the work is complete, they push the branch up to the team's central repository, other team members review the changes, the developer makes any needed fixes from the review, and then the feature branch is merged back into the primary development branch.

Developers may need to pull down changes that have been added to the primary branch, then "merge down" from the primary branch into their feature branch. Merging the feature branch back into the primary branch is referred to as "merging up".

Pull Requests

If you've worked with Git at all, you've probably heard the term "pull request" (also know as a "PR" for short, or occasionally "merge request") before. Strictly speaking, a "pull request" isn't even a Git concept - it's a merging workflow that is built on top of Git by repository hosting sites and tools like Github, Gitlab, and Bitbucket.

Pull Requests are an approach to doing code reviews and handling merging at the central Git repo/server level. This is typically associated with using feature branches. A developer pushes up their completed feature branch, and creates a PR that will merge some-feature into main. Other devs can look at the page for the PR, see the file diffs, and leave comments on specific lines suggesting changes. The feature dev makes more commits based on those suggestions, pushes them up, and the PR is updated to reflect the changes. After other team members approve, the PR can be merged and the feature branch can be deleted.

Updating Branches in the Background

Normally, the main way to update a local copy of a branch is to git checkout some-branch and then git pull. But, if I'm working on a feature branch, I often have unsaved changes and don't want to switch over to the main branch just to do a pull.

There's a really useful trick for doing a "background pull" of a branch without checking it out:

git fetch <remote> <remoteBranch>:<localBranch>

So, say I'm on features/some-feature, and I want to update my main branch without switching to it. Typically the local branch and remote branch have the same name. So, I can run:

git fetch origin main:main

and Git will download any new commits on the remote origin/main branch, then update my local main branch to have those commits too.

Rewriting Git History

There's a variety of ways to alter the history in a Git repository. Each technique is useful in different situations, and these are often valuable for fixing earlier problems. As mentioned earlier, Git commits are immutable, so you can never actually modify them - you can only replace commits with new ones. So, when we "rewrite history", we're actually creating an "alternate history" instead.

It's critical that you should only ever rewrite history that is still local to your own repository and has never been pushed up to another repository! As long as commits haven't been pushed, no one else cares about them, and you can rewrite them to your heart's content. But, once they've been pushed, someone else's Git repo clone may be relying on the old history, and changing that history will likely cause conflicts for them.

Amending Commits

The easiest technique for rewriting history is to "amend" the latest commit.

Amending a commit really means replacing it with a slightly different one. This can be done via git commit --amend, or a corresponding option in a GUI tool:

Technically, the old commit still exists in Git's storage, but the current branch ref now points to the newly created commit instead.

Resetting Branches

Since branch refs are pointers to a given commit, we can reset a branch by updating the ref to point to an earlier commit. This is typically used to roll back some of the commits you made.

When you reset a branch, you have three options for what happens to the files on disk and in the staging area:

git reset: move a branch pointer to point to a different commit
- --soft: keep the current files on disk and in the staging area
- --mixed: keep the current files on disk, but clear the staging area
- --hard: clear the staging area and make the working directory look exactly like this specific commit

So, git reset --soft is fairly "safe" to do, because it doesn't change any files on disk. git reset --hard is "dangerous", because it will wipe out any files that were changed during these commits or that haven't been committed yet, and replace them all with the files from this exact commit.

git reset requires a commit identifier as an argument. This could be a specific commit hash ( git reset ABCD1234 ), or some other revision identifier. You can even update your current branch to point to the same commit as a different branch ( git reset --hard some-other-branch ).

Rebasing Branches

"Rebasing" is a technique that is an alternative to merging for updating one branch with another's changes. Instead of combining the two sets of changes directly, rebasing rewrites history to act as if the current branch was created now, off the latest commits on the source branch, instead of starting from the earlier commits. Similar to merging, this is done with git rebase OTHER_BRANCH_NAME.

Imagine that the main branch has commits A <- B to start with, and we make a feature branch starting from commit B. Now, someone else merges some more work into main, giving it commits A <- B <- C <- D. If we rebase our feature branch against main, it's kind of like cutting off the line of our feature branch, transplanting it to the end, and pretending we really started this branch after commit D instead of B:

Reverting Commits

Resetting a branch effectively throws away the newer commits. What if we want to undo the changes in an earlier commit, but keep the history since then?

Reverting a commit with git revert creates a new commit that has the opposite changes of the commit you specified. It doesn't remove the original commit, so the history isn't actually modified - it just inverts the changes.

Cherry-Picking

Cherry-picking allows you to copy the changes in specific commits, and apply those as new commits onto a different branch. For example, maybe there's an urgent patch that has to be created directly onto a hotfix branch and deployed to production, but you need to also make sure that main has that commit as well. You can cherry-pick the individual commit from the hotfix branch over onto main.

git cherry-pick accepts either a single commit reference, or a commit range. Note that the range excludes the first commit you list. if I run git cherry-pick A..E, then it will copy commits B,C,D,E over onto this branch. This creates new commits with new hashes (because the timestamps and parent commits are different), but preserves the diffs and commit metadata.

Interactive Rebasing

"Rebasing" involves rewriting the entire history of a branch. There is a variation on this called "interactive rebasing", which allows you to selectively modify earlier commits on a branch. This is done with git rebase -i STARTING_COMMIT.

Interactive rebasing lets you perform several different types of modifications. You can:

Edit the message for a commit
Reorder commits
Squash multiple commits together
Remove commits

After you specify the desired changes to the commit history, Git will execute the modifications you listed, and update all commits after the starting point accordingly. As with other history rewriting operations, this always produces a new set of commits after any changed commit, with new hashes even if the rest of the contents haven't changed due to the parent commits changing.

Running an interactive rebase from the CLI brings up a list of all commits after the starting commit in your text editor, along with a column of odd command names like "pick" and "squash". You rework the commits by actually modifying the text in the file, and then saving and exiting. For example, if you want to swap a couple commits, you'd cut one of the text lines and paste it in a different location.

I find this very unintuitive to work with, so I strongly recommend using a Git GUI for any interactive rebase operations. SourceTree and Fork have pretty good UIs for performing interactive rebasing.

Reflog

It's actually very hard to completely wipe out your Git commits and permanently lose work. Even if you do a git reset --hard and the commits appear to have vanished, Git still has a copy of those commits saved internally.

If you do end up in a situation where you can't see those commits referenced from any tag or branch, you can use the Git reflog to look back and find them again. The reflog shows all commits, no matter what branch they're on or whether there's still a meaningful pointer to that commit. That way you can check them out again, create a new tag or branch pointing to those commits, or at least see the diffs.

Advanced History Rewriting

Finally, Git supports some very advanced tools for rewriting history at the whole repository level. In particular, git filter-branch lets you perform tasks like:

rewriting file names and paths in the history (example: changing files in ./src so that they now appear to be in the repo root)
creating a new repo that contains just certain folders from the original, but with all their history
rewriting actual file contents in many historical commits

git filter-branch is notoriously slow, so there's other external tools that can perform similar tasks. While I haven't used it, https://github.com/newren/git-filter-repo claims to be able to run the same kinds of operations much more quickly, and is apparently now even recommended by the actual Git docs.

Sometimes repos end up with very large files cluttering the history, and you want to rewrite the history to pretend those files never existed. A tool called the BFG Repo Cleaner does a good job of that.

If these existing tools don't do what you need, you can always write your own. I once wrote a set of Python-based tools to rewrite the JS source for for an entire repository with multiple years of history, including optimizing it to run in just a few hours.

These tools are very powerful and should not be something you use for day-to-day tasks. Think of them as fire extinguishers. You hope you never need to use them, but it's good to have it sitting around in case something happens.

Git Patterns and Best Practices

So now that we've covered a bunch of commands and technical details, how do you actually use Git well? Here's the things that I've found to be most helpful:

Write Good Commit Messages

It's critical to write good commit messages. It's not just a chore to satisfy the Git tools. You're leaving notes to any future developers on this project as to what changes were made, or even more importantly, why those changes were made. Anyone can look at a set of diffs from a commit and see the changed lines, but without a good commit message, you may have no idea what the reason was to make those changes in the first place.

There's lots of good articles out there discussing rules for writing commit messages, with plenty of good advice. I personally don't care so much about things like "max 72 characters per line" or "use present tense for the top line and past tense for other lines", although there's valid reasons to do those things. To me, the critical rules are:

Always start the commit message with a relevant issue tracker ID number if there is one. That way you can always go back to the issue tracker later to see more details on what the specific task was supposed to be.
First line should be a short high-level summary of the intent of the changes. This line of the message is what will be shown in any Git history log display, so it needs to both fit in one line, and clearly describe the commit overall. Aim more for the purpose of the changes than a specific list of "what changed" in this line.
If you have any further details, add a blank line, then write additional paragraphs or bullet points. Write as much as you want here! This section will usually be collapsed by default in a Git UI, but can be expanded to show more details. I've seen some excellent commit messages that were multiple paragraphs long, and they provided important context for why changes were being made.

A typical example of this format would look like:

MYAPP-123: Rewrite todos slice logic for clarity

- Added Redux Toolkit
- Replaced handwritten reducer with `createSlice`.  This simplified the logic considerably, 
  because we can now use Immer and it auto-generates the action creators for us.
- Updated `TodosList` to use the selectors generated by the slice.

Make Small, Focused Commits

This goes hand-in-hand with the advice to write good commit messages.

Commits should be relatively small and self-contained, conceptually. One commit might touch several files, but the changes in those files should be closely related to each other. There's multiple reasons for this:

It makes it easier to describe the changes in the commit
It's easier to look at that one commit and see what changes it includes
If the commit needs to be reverted later, there's fewer other changes that would be affected
When someone looks at the line-by-line history, there will be more specific comments associated with each line ("Fixed bug X" instead of "Made a bunch of changes", etc)
It's easier to bisect the commit history and narrow down what changes might have caused a particular bug

For example, say I'm adding a new JS library to a project. I would make one commit that just updates package.json and yarn.lock, then put the initial code changes using that library into a separate commit. You can see an example of this commit approach in the commits for the "Redux Fundamentals" tutorial example app I wrote.

To me, the commit history should "tell a story" of how a given task was accomplished. Someone should be able to read through the series of commits, whether it be during the PR review process or years down the road, and be able to understand my thought process for what changes I made and why I made them.

Clean Up Commit History Before Pushing

I frequently have to make "WIP" commits as I'm working on a task. Maybe I've just made a bunch of edits, the code is now mostly working, and I want to record a checkpoint before I keep going. Or, maybe I forgot to commit a particular bit of code, added it in another commit later, but it doesn't really belong as part of the "story" that I'm telling.

I often use interactive rebase to clean up my commits before I push a branch for a PR. Just because I have some junk commits in my history locally doesn't mean that the rest of the world needs to know or care that was part of my actual progress for this task. The "story" that I'm telling with my commits is sort of the idealized version - ie, "let's pretend that I did this task perfectly without any mistakes along the way".

Only Rewrite Unpushed History

As mentioned earlier: as long as a branch is still local and hasn't been pushed, it's fair game - rewrite it all you want! Once it's been pushed, though, you should avoid rewriting it.

The one main exception to that is if the branch is still up for PR, and you redo the history. At that point, most likely no one depends on it yet, so you can get away with force-pushing the branch to update the PR. (The React team does this frequently.)

Keep Feature Branches Short-Lived

There's no hard rule about how many lines of code or commits can be in a branch. In general, though, try to keep feature branches relatively short-lived. That way the size of the changes to merge in a PR is smaller, and it's less likely that you'll need to pull down changes from someone else.

Some people argue about whether it's better to merge feature branches back into the primary branch, or rebase them when they're done to keep the main branch history "clean and linear". I kinda like having merge commits, personally - I prefer seeing when things got merged in. The important thing is to pick a convention as a team and stick with it.

Code Archeology with Git

So why do all these good commit practices matter?

Say you're working in a codebase with multiple years of history. One day, you're assigned a task to work on some portion of the codebase. Maybe it's fixing a bug that just popped up, or adding a new feature. You open up a file, and there's hundreds of lines of code inside. You read through it, and it's kind of ugly - there's a bunch of extra conditions in the logic, and you're really not sure how it ended up this way.

Reading through that file tells you what the code does, now. Unless the file has a lot of good comments, there may not be much information for why the code is like that, or how it got that way. We naturally have a tendency to assume that "whatever code is there currently must be correct", but that's not always true :)

That's where having a good Git history is critical. Digging through a file's history can show you:

Who wrote each line of code
When that code was changed, and what other code changed at the same time
What task the changes were part of
What the intent was behind the change
What the author was thinking at the time
When a bug was introduced

These can all be extremely valuable pieces of information when tracking down a bug or working on a feature.

Displaying Historical File Changes

There's a variety of ways to view the history of changes to a file.

git log lets you look at the commits that affected a specific file. IDEs and Git GUIs let you explore the history of a file as well, showing each commit its diffs, often including the ability to diff two arbitrary versions of a file. Some Git GUIs also let you explore the entire repo file tree as of a specific commit.

Git has a feature called git blame, which prints the commit ID, author, and timestamp for each line. The CLI output is hard to read, but every good IDE has the ability to show file blame information next to the actual code in a file. IDEs typically enhance the blame information to show you more details on the author, the commit message, and the commits before and after this one:

Github offers a "blame" view as well, and makes it easy to jump back to view an earlier version of the repo. Github also lets you browse specific file versions and trees. For example, https://github.com/reduxjs/react-redux/tree/v7.1.2 shows the React-Redux codebase as of tag v7.1.2, and https://github.com/reduxjs/react-redux/blob/5f495b23bcf3f03e0bee85fa5f7b9c2afc193457/src/components/connectAdvanced.js shows that exact file version. (Press y while browsing a file on Github to change the URL to the exact file hash.)

Bisecting Bugs

Git has a really neat command called git bisect, which you can use to help find the exact commit where a bug was introduced. When you run git bisect, you can give it a commit range where you think the problem started. Git will then check out one commit, let you run whatever steps you need to with the app to determine if the bug is present or not, and then say git bisect good or git bisect bad. It then jumps to another commit and lets you repeat the process. It follows a splitting pattern that lets you narrow down the potential problem commit in just a few steps.

Final Thoughts

As software developers, we use lots of tools. Everyone has their own preferences for things like text editors and such, but everyone on a team is going to use the same version control system. In today's world, that's inevitably Git.

Given how critical Git is to modern development, anything you can do to use it more effectively will pay dividends down the road, and anyone reading your commits will appreciate the effort you put into clearly describing what changes are happening and why. It might be a teammate reading your PR, an intern exploring the codebase next year, or even yourself revisiting code that you wrote many years ago.

Ultimately, good Git practices are a key part of long-term codebase maintainability.

Further Information

Git Tutorials
- Atlassian Git tutorials
- Git for Humans slides
Git Internals
Commit Messages
- How to Write a Git Commit Message
- Git Commit Message Good Practices
Operations
- Git Beyond the Basics
- Advanced Git Commands You Will Actually Use
Cheat Sheets
- A Visual Git Guide
- Tower Git Cheat Sheet
- akras14 Git Cheat Sheet
- DevHints Git cheat sheets: revision syntax, log syntax, branching commands
Other Resources
- Git Flight Rules: a guide for when things go wrong
- React/Redux Links: Git Resources

Idiomatic Redux: Redux Toolkit 1.0

Wed, 23 Oct 2019 10:00:00 -0500

Today I am extremely excited to announce that:

Redux Toolkit 1.0 is now available!

Note: Redux Toolkit was formerly known as "Redux Starter Kit", but we renamed it to clarify the intended usage and audience.

As part of that, I'd like to look back at how Redux Toolkit came to be, talk about my goals and vision for the project, and look at how we've designed the API to fulfill those goals.

As a TL;DR, Redux Toolkit is designed to:

Make it easier to get started with Redux
Simplify common Redux tasks and code
Use opinionated defaults guiding towards "best practices"
Provide solutions to reduce or eliminate the "boilerplate" concerns with using Redux

I'm proud to say that we've accomplished all those goals!

Origins

The Rise of Redux

Redux came out in the summer of 2015, at the tail end of the "Flux Wars". Within a year, it had swept aside all other Flux implementations, and became the de-facto standard state management tool for React applications.

Redux was deliberately designed to be extensible, and it succeeded brilliantly. The design led to a thriving ecosystem of addons and related libraries, ranging from action/reducer generation utilities to store persistence to dozens of immutable update utilities to middleware for manipulating dispatched actions. In particular, since Redux deliberately did not include a built-in solution for managing async behavior and side effects, dozens of side effect addons appeared.

In short, if you needed to do something with Redux, odds were an addon already existed for your use case, and it was possible to build something yourself otherwise.

Facing the Hype Cycle

Most major technologies go through the hype cycle. A new product comes out, and early adopters fall in love with it and tout it as the solution to every problem. As more people use it, they begin to run into limitations, and the negative feedback becomes noticeable. Eventually, enough people figure out the best ways to use it, and it settles down as a viable tool with known tradeoffs.

Redux is no exception to this. It quickly became associated with React, and many tutorials and discussions assumed that "if you're using React, you must be using Redux too". This became something of a self-fulfilling prophecy, as new learners would read those tutorials and assume they had to use Redux with React. Or, in many cases, senior devs might tell other devs that they had to.

As a result, many people began to learn Redux without understanding the context of why it was originally created, what problems it was meant to solve, and what the tradeoffs were. Blindly using any tool without understanding is a recipe for trouble. While many people were happy using Redux, many others began to complain about flaws, both actual and perceived.

Dealing with Complexity

There are several factors that combine to make it relatively harder to use Redux in comparison to other options:

Redux deliberately adds a level of indirection to the idea of updating state, in the form of actions. As Dan said early on:

[Redux] is not designed to be the most performant, or the most concise way of writing mutations. Its focus is on making the code predictable.

Common conventions around Redux add additional levels of indirection and code that "needs" to be written (action creators, action type constants, selectors, thunks, switch statements, immutable updates).
There's also a lot of "rules" around Redux that are described in written form, but not obviously enforced by Redux itself, particularly immutability, serializability, and avoiding side effects.
JS is a mutable language, and writing correct immutable code is hard, both in terms of avoiding accidental mutations and the amount of code that has to be written to handle updates immutably
Redux was always meant to be a small core supporting a larger ecosystem of addons. This means that the core is deliberately tiny, and it's up to the user to choose how to configure the store. This is great for flexibility, but means that even the most common tasks require multiple semi-complex lines (such as adding thunk middleware for async logic, and also setting up the DevTools Extension). It also means the user has to make decisions early on about which addons they want to use, even in a simple app.
Functional programming concepts are foreign to most people and hard to grok, especially if you're coming from an OOP background

Like most early adopters, many early Redux users were "power users". Configuration, customization, and "implementing things by hand so I know exactly what's happening" were key selling points. However, Redux took off across a larger userbase, those early selling points became weaknesses. Many people saw the patterns and practices as simply "boilerplate" that was a pain to deal with, and reasons to look for other options.

In addition to the minimum required complexity of dispatching actions and writing reducers, the incidental complexity became a problem. For example, the original Redux docs divided code up into files and folders by "actions", "constants", "reducers", and "containers". There are valid reasons to split your code into files by type. But, at the same time, there's no requirement that you must write actions, reducers, and constants in separate folders/files. Because it was in the docs, most people copied what they saw, and this did become the standard pattern.

Similarly, writing action types as constants like const ADD_TODO = "ADD_TODO", or the idea of having action creator functions for every single action type, are not required either. But, those were the recommended patterns, and users followed the examples in the docs.

On top of that, the need to make decisions about what addons to use has frequently resulted in decision fatigue. The plethora of side effects libs (each with their own terminology) has been a particular pain point. Even within a year of Redux's release, it was evident that the rapidly growing list of side effects libs was confusing.

While the ecosystem has largely settled on thunks, sagas, and observables as the standard approaches, the situation has made "how do I choose a side effects middleware?" one of the most frequently asked questions. (Ironically, thunks were originally built into the core, but removed to make things flexible and unopinionated). Clearly, there's a need for an official blessing on how to handle side effects.

In Search of a Solution

I got involved with Redux when I volunteered to write the Redux FAQ in early 2016. Dan handed over the maintainer keys to myself and Tim Dorr that summer, and I found myself answering Redux questions everywhere across the internet.

By spring 2017, I realized that the programming landscape had changed enough that most new Redux users didn't understand the context behind its creation or why the typical usage patterns existed. I attempted to remedy that by writing a pair of comprehensive blog posts: The Tao of Redux, Part 1: Implementation and Intent, and The Tao of Redux, Part 2: Practice and Philosophy, which covered how Redux was originally designed and how it was meant to be used.

Around the same time, I filed Redux issue #2295: Request for Discussion: Redux "boilerplate", learning curve, abstraction, and opinionatedness . In that issue, I asked:

What would idiomatic Redux usage with "less boilerplate" look like? How can we provide solutions to those complaints?

What possible abstractions could be created that simplify the process of learning and usage, but without actually hiding Redux (and would hopefully provide a migration/learning path to "base" Redux)?

The very first reply was from Justin Falcone (@modernserf), who suggested:

official redux-preset package containing redux, react-redux, redux-thunk already wired together

built-in jumpstate-style reducer creators, e.g. createReducer({[actionType]: (state, payload) => state}, initState)

That suggestion was the genesis for what would eventually become Redux Toolkit.

The thread continued on for over a hundred comments, eventually devolving into users just pasting links and advertisements for their own Redux abstraction libs. In the middle of the thread, Dan Abramov dropped by, and laid out his own wishlist for an opinionated layer on top of Redux:

If it were up to me I’d like to see this:

A Flow/TypeScript friendly subset that is easier to type than vanilla Redux.

Not emphasizing constants that much (just make using string literals safer).

Maintaining independence from React or other view libraries but making it easier to use existing bindings (e.g. providing mapStateToProps implementations).

Preserving Redux principles (serializable actions, time travel, and hot reloading should work, action log should make sense).

Supporting code splitting in a more straightforward way out of the box.

Encouraging colocation of reducers with selectors, and making them less awkward to write together (think reducer-selector bundles that are easy to write and compose).

Instead of colocating action creators with reducers, getting rid of action creators completely, and make many-to-many mapping of actions-to-reducers natural (rather than shy away from it like most libraries do).

Making sensible performance defaults so memoization via Reselect “just works” for common use cases without users writing that code.

Containing built-in helpers for indexing, normalization, collections, and optimistic updates.

Having built-in testable async flow support.

On top of core, of course, although it’s fine to brand it in as an official Redux thing.

The thread eventually fizzled out, but the discussion stayed in the back of my mind.

Jumpstarting the Process

In February 2018, Nick McCurdy filed an issue suggestion adding freezing of state in dev to avoid accidental mutations. In that thread, I commented:

This goes back to my comments in #2295 , about wanting to provide a "starter preset" of packages that would provide a Redux-team-recommended simplified store setup. I've seen dozens of variations on the theme, as well as "CRA + Redux" pieces out there already, but we haven't "blessed" any of them.

The real issue here is that the Redux core comes completely unconfigured out of the box. We make no assumptions about what middleware you might add in, or even if you're going to use middleware at all. So, we can't just add a dev middleware out of the box, because nothing is configured out of the box.

In response, Tim Dorr opened Redux issue #2859: RFC: Redux Starter Kit, and said:

Based on comments over in #2858 and earlier in #2295, it sounds like a preconfigured starter kit of Redux core + some common middleware + store enhancers + other tooling might be helpful to get started with the library.

As for what goes in a starter package, the short list in my mind includes one of the boilerplate reduction libraries (or something we build), popular middleware like thunks and sagas, and helpful dev tooling. We could have a subset package for React-specific users. In fact, I started a Twitter poll to find out if that's even necessary (please RT!).

I don't think we need to build a Create React App-like experience with a CLI tool and all that jazz. But something out of the box that gives a great dev experience with better debugging and less code.

Seeing the issue got me incredibly excited, and that led to a very long discussion on Reactiflux with Nick McCurdy and Harry Wolff, as we tossed around ideas for what this "starter kit" thing might look like. Afterwards, Harry summarized the ideas:

wrapper around createStore with better defaults, i.e. easier way to add middleware, redux dev tools

built in immutable helper (immer?)

createReducer built in (the standard "reducer lookup table" utility)

reselect built-in

action creator that adheres to FSA

async action support / side effect support (flux, saga, what?)

The discussion continued in the issue. I had some time free the next day, and put together a notional first sample implementation of configureStore and createReducer.

Two days later, I published @acemarke/redux-starter-kit as an experimental package, and Redux Starter Kit (the original name) was born!

Designing the Toolset

Early Implementation and Naming

Work on RSK advanced intermittently over the next few months. Nick filled out the build setup and initial unit tests, added CI integration, and enabled the Redux DevTools in configureStore. I added re-exports from Redux and the selectorator library, and oversaw discussion and improvements to middleware setup. But, after the initial burst of activity in March and April 2018, RSK just sort of sat around for the next few months.

I came close to burning out in the summer of 2018 due to too many self-assigned responsibilities, such as maintaining my links lists. After taking some time to evaluate, I decided I really needed to focus on "maintainer" responsibilities. As part of that, I decided it was time to start pushing RSK forward seriously.

The initial name of "Redux Starter Kit" was notional, and part of why I'd originally published it as a personal scoped package. In late August, I opened up an issue to bikeshed over what the final name should be. We debated various options, including using a @redux/ scoped prefix.

The discussion trailed off until early October, when someone suggested we just go with plain redux-starter-kit. The problem was that the unscoped redux-toolkit package name was already taken on NPM.Just for kicks, we tried pinging the original owner (@shotak) and asked if they would be willing to donate the redux-toolkit package name. To my surprise, they replied within a few hours and said yes. They transferred ownership of the NPM package, I moved the repo over to the reduxjs org, and published the first official version as v0.1.0.

Filling Out Functionality

The createReducer and createAction functions we'd already added were useful, but I wanted to find more ways to simplify things.

We'd already been discussing possibly bringing in helper functions from packages like redux-utilities.

I'd seen dozens of various similar libs. One that had stuck out to me was Eric Elliott's autodux package. In particular, autodux allowed users to auto-generate action creators and action types based on an object full of reducer functions and a "slice name" prefix.

I'd met Shawn Wang at ReactRally in August and told him about my ideas for RSK, and he wrote up an assessment of autodux and several other packages, concluding:

using autodux creates a fairly powerful set of creators and selectors and reducers out of the box, and you can further write actions for business logic easily. i like autodux.

Ultimately, autodux served as the direct inspiration for the most useful part of Redux Toolkit: the createSlice function.

Eric Bower (@neurosnap) was participating in those discussions, and made an initial attempt to port autodux into our own createSlice. The PR stalled a bit. Shortly thereafter, I saw someone on Reddit say they'd written a library called robodux that used Immer, and after some confusion, I realized that Eric had opted to try putting together his own version as a separate lib written in TypeScript.

After some further discussion, Eric agreed to contribute his code back to RSK in whatever form worked best for us. I ended up compiled his TS code to plain JS and porting that back over to RSK.

Defining a Roadmap and Clarifying the Vision

In early December, we set up a Docusaurus-based docs page for RSK as a testbed for converting the Redux docs.

Meanwhile, I added default dev middleware for detecting mutations and non-serializable values, and enabled the new Redux DevTools Extension "action stack trace" feature that I'd helped to implement.

In late January, I filed RSK issue #82: Roadmap to 1.0, and began adding the ability for slices to listen to other actions. In that PR, Denis Washington (@denisw) left a comment suggesting that createSlice be dropped entirely.

In response, I put together a very long comment, laying out My Vision for Redux Starter Kit. I'll skip quoting the entire thing (which is worth reading), but here's the key section:

Objectives

Per the README, other specific inspirations are Create-React-App and Apollo-Boost. Both are packages that offer opinionated defaults and abstractions over a more complex and configurable set of tools, with the goal of providing an easy way to get started and be meaningfully productive without needing to know all the details or make complicated decisions up front. At the same time, they don't lock you in to only using the "default" options. They're useful for new learners who don't know all the options available or what the "right" choices are, but also for experienced devs who just want a simpler tool without going through all the setup every time.

I want Redux Starter Kit to be the same kind of thing, for Redux.

Speed Up Getting Started

... I want to give people a way to get working Redux running as fast as possible.

Simplify Common Use Cases and Remove Boilerplate

I want to remove as many potential pain points from the equation as possible:

Reduce the amount of actual physical characters that need to be typed wherever possible

Provide functions that do some of the most obvious things you'd normally do by hand

Look at how people are using Redux and want to use Redux, and offer "official" solutions for what they're doing already

Opinionated Defaults for Best Practices

Redux Toolkit should set things up in ways that guide the user towards the right approach, and automatically prevent or detect+warn about common mistakes wherever possible.

No Lock-In / Add It Incrementally

If someone starts an app using Redux Toolkit, I don't want to prevent them from choosing to do some parts of the code by hand, now or later down the road.

On the flip side, someone should be able to add Redux Toolkit to an existing Redux application, and begin using it incrementally, either by adding new logic or replacing existing logic incrementally.

Become the "Obvious Default" Way to Use Redux

Long-term, I'd like to see RSK become the "obvious default" way for most people to use Redux, in the same way that Create-React-App is the "obvious default" way to start a React project.

Don't Solve Every Use Case, and Stay Visibly "Typical Redux"

Redux is used in lots of different ways. Because of that, and because the core is so minimal, there's thousands of different Redux addon libraries for a wide spectrum of use cases. We can't solve all those ourselves, and we won't solve all those ourselves.

Reasonable TypeScript Support

I want to support people using TypeScript, same as I want to support anyone else using Redux in specific ways. So, RSK should be reasonably well typed. At the same time, I don't want "perfect typing" to become the enemy of "good functionality". I'm fine with trying to shape things so they work well in TS, but I'm very willing to ship a feature that has less-than-ideal typing if that's what it takes to get it out the door.

Honing the API

Simultaneous with this discussion, Denis was busy converting the RSK codebase to TypeScript, based on a fork from @Dudeonyx. @Jessidhia also chimed in with several good pieces of TS advice.

The React-Redux v7 rewrite completely occupied my attention for the next couple months. The hooks API design process kept me mostly occupied after that.

Meanwhile, we had ongoing discussions on things like possible additions to createSlice, like maybe a special combineSlices() function, or whether the auto-generated selector functionality was really useful.

By summer 2019, the React-Redux work was finally done, and I was ready to turn my attention back to RSK. I began writing an actual set of tutorial docs pages, finishing in early September.

Meanwhile, Lenz Weber (@phryneas) had started contributing a number of key improvements, adding a prepare callback to createAction, and rewriting the TS type definitions for readability.

From there, I began nailing down the final API. This included new functionality like allowing customization of the default middleware and exporting the case reducers, and breaking changes such as removing selectors from createSlice and renaming the slice field to name.

There were still plenty of other ideas for more additions, but I concluded that anything else could be added post-1.0.

And that leads us up to the official release of Redux Toolkit 1.0!

Changing the Name

I've been going back and forth between the names "Redux Starter Kit" and "Redux Toolkit" in this post. We did call it "Redux Starter Kit" up through the actual 1.0 release. But, after the release, we began getting feedback that the word "Starter" was confusing people. They assumed that it was a CLI project creation tool, a cloneable boilerplate project, something that was only suitable for beginners, or something that you'd outgrow later in a project.

I didn't want to rename the package, but it was clear that this confusion was going to continue.

I opened up another thread to discuss naming and marketing. After some additional discussion, we concluded that "Redux Toolkit" was the best option, and that we should publish it as a scoped @reduxjs/toolkit package name.

With that decision made, I was able to push through the rename process fairly quickly, and published the renamed package as Redux Toolkit v1.0.4.

The Future of Redux Toolkit

Review

Looking back at the original ideas and goals that came up in those early discussion issues, I think we've pretty much nailed a majority of the desired capabilities.

RTK is:

an official opinionated package, with our recommended best practices built in
that drastically simplifies Redux apps and reduces "boilerplate" for the most common use cases
while still remaining "visibly Redux" in the process
that can all be added to a new project on day 1, or used for incremental migration of an existing project
is useful for both new learners and experienced developers
and provides solid TypeScript support without requiring a painful amount of type declarations

I'd say we've hit the sweet spot with that list!

Reception

I've been actively advertising Redux Toolkit for the past year. It's linked on the front page of the Redux docs, it's recommended in the "Configuring Your Store" docs page, and I've been telling almost literally everyone I talk to that they should try out RTK :)

I've been excited to see that as a direct result of my efforts, Redux Toolkit has been picking up some actual adoption. Starting at effectively 0 a year ago, it's now at 100K NPM downloads a month and climbing at a solid pace. This is even more impressive given that it's been pre-1.0 until now.

I've also been thrilled at the reception for RTK. I've received numerous comments telling me things like:

Honestly your starter thing should have been the advertised api all along. It’s 💯

hijacking this to confirm how the redux-toolkit is literally indispensable, from a beginner to an advanced level, the combination of createSlice and immer is just so good

I am in the business of building things and having less code to maintain, RTK checks all my boxes.🙏 thank you for making it

I'll admit to a certain amount of pride, in that RSK has basically been my idea, I've driven it, and I've written a very large portion of the code. So, yeah, it feels great to hear all the compliments.

But, even more than that, I'm excited for how much this is going to help people learning and using Redux.

Next Steps

We're currently planning a major revamp of the Redux docs, and I'm hoping to finally begin working on that now that RTK 1.0 is out.

As part of that docs revamp, we plan to actually teach using RTK as part of the tutorial sequence, and emphasize that we recommend its use throughout the rest of the docs. We're also going to be reworking the tutorials and other docs sections to emphasize simpler patterns wherever possible, like avoiding use of action constants, dropping the emphasis on writing React "container components", and recommending the "ducks pattern" for organizing Redux logic.

Feature-wise, there's also plenty more we could consider adding, such as maybe having a built-in way to define async side effects in slices.

If you've got any feedback, I'm always interested. Ping me at @acemarke on Twitter and in the Reactiflux chat channels.

Further Information

Idiomatic Redux: The History and Implementation of React-Redux

Thu, 22 Nov 2018 23:50:00 -0500

Intro

React-Redux is conceptually pretty simple. It subscribes to the Redux store, checks to see if the data your component wants has changed, and re-renders your component.

However, there's a lot of internal complexity to make that happen, and most people aren't aware of all the work that React-Redux does internally. I'd like to dig through some of the design decisions and implementation details of how React-Redux works, and how those implementation details have changed over time.

Note: This post was originally written before the release of React-Redux v6. It has since been updated to cover the final release of v6, and the development and release of v7.0 and v7.1.

Integrating Redux with a UI

Understanding Redux Store Subscriptions

It's been said that "Redux is just a [dumb] event emitter". There's actually a fair amount of truth in that statement. Earlier MVC frameworks like Backbone would allow triggering any string as an event, and automatically trigger events like "change:firstName" in models. Redux, on the other hand, only has a single event type: "some action was dispatched".

As a reminder, here's what a miniaturized (but valid) implementation of the Redux store looks like:

function createStore(reducer) {
    var state;
    var listeners = []

    function getState() {
        return state
    }
    
    function subscribe(listener) {
        listeners.push(listener)
        return function unsubscribe() {
            var index = listeners.indexOf(listener)
            listeners.splice(index, 1)
        }
    }
    
    function dispatch(action) {
        state = reducer(state, action)
        listeners.forEach(listener => listener())
    }

    dispatch({})

    return { dispatch, subscribe, getState }
}

Let's focus in particular on the dispatch() implementation. Notice that it doesn't check to see if the root state actually changed or not - it runs every subscriber callback after every dispatched action, regardless of whether there was any meaningful change to the state or not.

In addition, the store state is not passed to the subscriber callbacks - it's up to each subscriber to call store.getState() if desired to retrieve the latest state. (See the Redux FAQ entry on why the state isn't passed to subscribers for more details.)

The Standard UI Update Cycle

Using Redux with any UI layer requires the same consistent set of steps:

Create a Redux store
Subscribe to updates
Inside the subscription callback:
1. Get the current store state
2. Extract the data needed by this piece of UI
3. Update the UI with the data
If necessary, render the UI with initial state
Respond to UI inputs by dispatching Redux actions

Here's an example with a vanilla JS "counter" app, where the "state" is a single number:

// 1) Create a store
const store = createStore(counter)

// 2) Subscribe to store updates
store.subscribe(render);

const valueEl = document.getElementById('value');

// 3. When the subscription callback runs:
function render() {
    // 3.1) Get the current store state
    const state = store.getState();

    // 3.2) Extract the data you want
    const newValue = state.toString();

    // 3.3) Update the UI with the new value
    valueEl.innerHTML = newValue;
}

// 4) Display the UI with the initial store state
render();

// 5) Dispatch actions based on UI inputs
document.getElementById("increment")
    .addEventListener('click', () => {
        store.dispatch({type : "INCREMENT"});
    })

Every Redux UI integration layer is simply a fancier version of those steps.

You could do this manually, in every React, component, but it would quickly get out of hand, especially once you start trying to cut down on unnecessary UI updates.

Clearly, the process of subscribing to the store, checking for updated data, and triggering a re-render can be made more generic and reusable. That's where React-Redux and the connect API come in.

`connect` in a Nutshell

About a year after Redux came out, Dan Abramov wrote a gist entitled connect.js explained. It contains a miniature version of connect to illustrate how it works conceptually. It's worth pasting that here for emphasis:

// connect() is a function that injects Redux-related props into your component.
// You can inject data and callbacks that change that data by dispatching actions.
function connect(mapStateToProps, mapDispatchToProps) {
  // It lets us inject component as the last step so people can use it as a decorator.
  // Generally you don't need to worry about it.
  return function (WrappedComponent) {
    // It returns a component
    return class extends React.Component {
      render() {
        return (
          // that renders your component
          <WrappedComponent
            {/* with its props  */}
            {...this.props}
            {/* and additional props calculated from Redux store */}
            {...mapStateToProps(store.getState(), this.props)}
            {...mapDispatchToProps(store.dispatch, this.props)}
          />
        )
      }
      
      componentDidMount() {
        // it remembers to subscribe to the store so it doesn't miss updates
        this.unsubscribe = store.subscribe(this.handleChange.bind(this))
      }
      
      componentWillUnmount() {
        // and unsubscribe later
        this.unsubscribe()
      }
    
      handleChange() {
        // and whenever the store state changes, it re-renders.
        this.forceUpdate()
      }
    }
  }
}

// This is not the real implementation but a mental model.
// It skips the question of where we get the "store" from (answer: `<Provider>` puts it in React context)
// and it skips any performance optimizations (real connect() makes sure we don't re-render in vain).

// The purpose of connect() is that you don't have to think about
// subscribing to the store or perf optimizations yourself, and
// instead you can specify how to get props based on Redux store state

We can see the key aspects of the API here:

connect is a function returning a function returning a wrapper component.
The wrapped component's props are a combination of the wrapper component's props, the values from mapState, and the values from mapDispatch.
Each wrapper component is an individual subscriber to the Redux store.
The wrapper component abstracts away the details of which store you're using, how it's interacting with the store, and optimizing performance so that your own component only re-renders when it needs to.
You simply specify how to extract the data your component needs based on the store state, and the functions it can call to dispatch actions to the store

But, this miniature example hand-waves a lot of the details. In particular, as the comments point out:

Where does the store come from?
How does connect check to see if your component needs to actually update?
How does it implement the optimizations?

And beyond that, how did we even end up with an API that looks like this in the first place?

Development History of the React-Redux API

Early Iterations

The first few versions of Redux included React bindings as part of the main package. I won't go through those versions specifically, but you can see the changes in the release notes and READMEs for these early releases:

Fun side note: amazingly, all that iteration took place over a single week!

It took another three weeks to get up to v1.0.0-rc, which is where the meaningful history begins.

Original API Design Constraints

React-Redux was split out as a separate repo in July 2015, just before Redux v1.0.0-rc came out.

Dan filed React-Redux issue #1: Alternative API Proposals to discuss what the final API should look like. In that issue, he listed a number of design constraints that should guide how the final API worked:

Common pain points:
- Not intuitive how way to separate smart and dumb components with <Connector>, @connect
- You have to manually bind action creators with bindActionCreators helper which some don't like
- Too much nesting for small examples (<Provider>, <Connector> both need function children)

Let's go wild here. Post your alternative API suggestions.

They should satisfy the following criteria:
- Some component at the root must hold the store instance. (Akin to <Provider>)
- It should be possible to connect to state no matter how deep in the tree
- It should be possible to select the state you're interested in with a select function
- Smart / dumb components separation needs to be encouraged
- There should be one obvious way to separate smart / dumb components
- It should be obvious how to turn your functions into action creators
- Smart components should probably be able to react to updates to the state in componentDidUpdate
- Smart components' select function needs to be able to take their props into account
- Smart component should be able to do something before/after dumb component dispatches an action
- We should have shouldComponentUpdate wherever we can

These criteria form the basis for the React-Redux API that we have today, and help explain why it works the way it does.

The biggest points that came out of that discussion were using connect as a function instead of a decorator, and how to handle binding action creators.

Another fun side note: the "object shorthand" for binding action creators was one of Dan's first suggestions for the API:

Perhaps we can even go further and bind automatically if an object is passed.

Finalizing the API

The next few releases continued iterating on the connect API. Highlights of the changes:

v0.5.0: introduced the mapState, mapDispatch, and mergeProps arguments
v0.6.0: used immutability and reference checks to determine if re-rendering is necessary
v0.8.0: added the ability to pass store as a prop
v0.9.0: added the ownProps arguments to mapState/mapDispatch
v1.0.0: <Provider>s single child had to be a function that returned an element
v2.0.0: no longer supported "magically" hot reloading reducers
v2.1.0: added an options argument
v3.0.0: moved the initial mapState/mapDispatch calls to be part of the first render, to avoid state staleness issues

Continuing the observations, at one point Dan warned against connecting leaf components:

We do warn in the documentation that we encourage you to follow React flow and avoid connect()ing leaf components.

This is particularly amusing, given that we now recommend connecting components anywhere in the tree you feel it would be useful.

That brings us up to what I'd say is the "modern era" of the React-Redux API, starting with version 4.

v4.x

React-Redux v4.0.0 came out in October 2015, and had four major changes:

React 0.14 as a minimum and a peer dependency
No more React native-specific entry point
<Provider> no longer accepted a function as a child, but rather a standard React element
Refs to the child component now required the withRef option, instead of always being enabled

In addition, v4.3.0 added the "factory function" syntax of mapState/mapDispatch to allow per-component-instance memoization of selectors.

I would consider 4.x the first "complete" version of the React-Redux API. As such, it's worth digging in to some of its implementation details, as well as some common points that define the overall behavior of the API across versions.

API Behavior

Every Wrapper Component Instance is a Separate Store Subscriber

This one's pretty simple. If I have a connected list component, and it renders 10 connected list item children, that's 11 separate calls to store.subscribe(). That also means that if I have N connected components in the tree, then N subscriber callbacks are run for every dispatched action!.

Every single subscriber callback is then responsible for doing its own checks to see if that particular component actually needs to update or not, based on the store state and props.

UI Updates Require Store Immutability

We've already established that the Redux store will run all subscriber callbacks after every dispatched action, regardless of whether the state actually changed or not.

In order to implement efficient UI updates, React-Redux assumes you have updated the store state immutably, so it can use reference comparisons to determine if the state changed.

This occurs in three stages:

When a connect wrapper component's subscriber callback runs, it first calls store.getState(), and checks to see if prevStoreState !== storeState. If the store state did not change by reference, then it will stop right there and bail out of any further update work, because it assumes that no other part of the store state changed.
If the root state has changed, the wrapper component then runs your mapState function, and does a "shallow equality" comparison between the current result and the last result. If any of the fields have changed by reference, then your component probably needs to be updated.
Assuming that there was some change in the mapState or mapDispatch results, the mergeProps() function is run to combine the stateProps from mapState, dispatchProps from mapDispatch, and ownProps from the wrapper component itself. A final check is done to see if the merged props result has changed since the last time, and if there's no change, the wrapped component will not be re-rendered.

Note: The root state comparison relies on a specific optimization in combineReducers, which checks to see if any state slices were changed while processing an action, and if not, returns the previous state object instead of a new one. This is a key reason why mutating your state results in your React UI components not updating!

The Store Instance is Passed Down via Legacy Context

In v4, rendering <Provider store={store}> puts that store instance into the legacy context API, as context.store. Any nested class component could then request that context.store be attached to the component.

The biggest reason why the entire store was put into context was because context itself was flawed. If you put an initial value into context in a parent, and some child requested that value, it would receive the correct value. However, if the parent component put an updated value by that name into context, and there was an intervening component that skipped rendering using shouldComponentUpdate -> false, the nested component would never see the updated value. That's one of the reasons why the React team always discouraged people from using legacy context directly, suggesting that any uses should be wrapped up in an abstraction so that when a "future replacement for context" came out, it would be easier to migrate. (See the React "Legacy Context" docs page and Michel Westrate's article How to Safely Use React Context for more details.)

So, one workaround was to put an event emitter instance into context, instead of an actual value. The nested components could grab the emitter instance on first render and subscribe directly, thus bypassing the sCU "roadblocks" to receive updates. React-Redux did this, as did libraries like react-broadcast.

Connected Components Accept a Store as a Prop

In addition to checking for the store instance in context, the wrapper components can optionally accept a store instance as a prop named store instead, like <MyConnectedComponent store={store}>. This works because each component subscribes to the store individually, so this component just gets that store a different way.

This is mostly useful for rendering connected components in a test without putting a <Provider> around it, but does mean you could have a connected component in the middle of your tree that uses a different store than all the other components around it.

Implementation Details

Update Logic was Directly in the Wrapper Component

Up through v4, all of the logic was part of the connect component class itself, including handling store updates, calling mapState, and determining if the wrapped component needed to re-render.

There's a couple interesting observations out of this. First, the wrapper component always called setState() whenever the root store state had changed, before it tries to do anything else.

Second, as a result, the real work of running mapState and determining if anything has changed was actually done directly in the render method.

This meant that any update to the Redux store required all wrapper components to re-render themselves in order to determine if their wrapped components actually need to update or not. That's a lot of components re-rendering every time, and also meant React was always getting called on every meaningful store update.

Child Component Rendering was Optimized Via Memoized React Elements

It's not well documented, but React has a particular performance optimization built-in. Normally, when a component renders, it creates new child elements every time:

render() {
    // Turns into: React.createElement(ChildComponent, {a : 42})
    return <ChildComponent a={42} />;
}

Every call to React.createElement() returns a new element object like {type : ChildComponent, props : {a : 42}, children : []}. So, normally every re-render results in all-new element objects being created.

However, if you return the exact same element objects as before, React will skip updating those specific elements as an optimization. (This is the basis for @babel/plugin-transform-react-constant-elements, and the behavior is discussed some in React issue #3226).

The v4 implementation takes advantage of this, by memoizing the element for the child component to skip updating it if not needed. This is necessary because the wrapper component is itself already re-rendering, so it needs some way to not have the child re-render.

v5.x

Up through v4, React-Redux was still primarily the work of Dan Abramov, albeit with a number of external contributions. Dan had given me commit rights after I wrote the Redux FAQ, and I'd spent enough time working on issues and looking the code that I felt comfortable giving more feedback beyond just how to use Redux. However, by mid 2016 Dan had joined the React team at Facebook, and was getting busy there, so he told Tim Dorr and I that we were now the primary maintainers.

About that time, a user named Jim Bolla filed an issue asking about an unusual use of connect. During the discussion, Jim commented that he was working on "an alternate version of connect", and I mentally dismissed that.

A few days after that, though, Jim filed a follow-up issue asking for feedback on his alternate implementation. We discussed some of the complexities in connect's implementation, and how those related to the use cases it was trying to solve, but I again otherwise didn't think much of it.

To my surprise, a couple days later Jim created issue #407: Completely rewrite connect() to offer advanced API, separate concerns, and (probably) resolve a lot of those pesky edge cases as a precursor to filing an actual PR. I was still really skeptical and began pointing out concerns and edge cases, but to my (pleasant) surprise, Jim kept taking my feedback and improving his WIP branch. This included producing some benchmarks which indicated that his version was noticeably faster than v4 in some particular scenarios.

Jim's efforts eventually won me over, and we began seriously collaborating on pushing his rewrite forward. That became PR #416: Rewrite connect() for better performance and extensibility .

The rewrite was released as v5.0.0 in December 2016. The biggest changes were:

Logic was moved from the wrapper component into memoized selectors
Enforceed top-down subscription updates
Added a new connectAdvanced API
More customization of comparison options
Overall performance improvements

All of this while keeping the same public connect API compatible with v4.

v5 also resolved a large number of existing issues as well.

There were various bugfixes up through v5.0.7, and v5.1.0 recently added support for passing React's new built-in component types like memo and lazy into connect.

Let's dig through some of the details.

API Behavior

Top-Down Updates

The connect wrapper components subscribe to the store in componentDidMount. However, because that lifecycle fires bottom-to-top in a new component tree, it was possible for child components to subscribe to the store before their parents did. Up through v4, that resulted in some nasty recurring bugs.

As an example, imagine a connected list with 10 connected list item children. If they all render right away, the list items will subscribe before the list parent. If you then delete the data for one of the items from the store, the list item component's mapState would run before the parent's did. This usually meant that the list item's mapState would throw an error and break the component tree.

v5 enforced the idea of top-down updates. Components higher in the tree always subscribe to the store before their children do. That way, in a scenario like the connected list, deleting an item from the store will result in the parent updating first, and re-rendering without that list item child before the child even has a chance to run its own mapState. This gives much more predictable behavior, and aligns with how React itself works.

We'll discuss the specifics of how this is implemented separately.

`connectAdvanced`

connect is fairly opinionated. It lets you extract data from the store via mapState, and prepare functions that dispatch actions via mapDispatch, but it doesn't let you use store state data in mapDispatch to prevent performance footguns. It does provide the mergeProps argument as an escape hatch, but that's separate.

However, for users that want more flexibility (such as Jim himself), v5 adds a new connectAdvanced API. Rather than taking (mapState, mapDispatch), it asks you to pass in a "selector factory". A selector instance will be created for each component and given a reference to dispatch, and the selectors will be called with (state, ownProps) on all future updates from the store or the wrapper component. That way, you can customize exactly how you want to handle derived props based on those inputs.

The original connect API is now actually implemented as a specific set of selector functions and options to connectAdvanced.

Implementation Notes

Logic is Implemented in Memoized Selectors

v5 moves all of the state derivation logic out of the wrapper component and into a separate set of homegrown memoized selector functions. These selectors specifically implement all of the connect API behavior, like:

Checking if the root state has changed
Handling the various forms of mapState and mapDispatch ( (state) vs (state, ownProps), mapDispatch as an object vs function, etc)
Calling mapState, mapDispatch, and mergeProps
Calculating the new child props and determining if a re-render is actually necessary

As a result, the subscriber callbacks can run extremely quickly, without involving React at all. In fact, React will only get involved once the wrapper component knows the child should re-render, and it uses a dummy this.setState({}) call to queue up that re-render. (We probably could have used forceUpdate() instead, but I don't think it makes any difference in this case.)

This is the biggest reason why v5 is generally faster than v4.

Custom Top-Down Subscription Logic

In order to enforce top-down subscriptions, v5 introduced a custom Subscription class. Internally, connect actually puts both the store instance and an instance of Subscription into legacy context. If no subscription exists in context, that component will subscribe to the store directly, as it must be high up in the tree. Otherwise, it subscribes to the Subscription instance. This means that each connected component is effectively subscribing to its nearest connected ancestor.

When an action is dispatched, the uppermost connected components will have their callbacks be triggered right away. If they do need to re-render, they call setState(), and wait until componentDidUpdate to trigger notification of the next tier of connected components. If no update is necessary, the next tier is notified right away.

This works, but it also requires some very tricky logic in both the Subscription class and the wrapper component itself (including dynamically adding and removing a componentDidUpdate function to micro-optimize perf).

v6.0

Motivations

v5 was great. It performed faster than v4 in almost all scenarios we've seen, and it added more flexibility.

However, the React team has continued to innovate. In particular, React 16.3 introduced the new React.createContext() API, which is an officially supported replacement for the legacy context API, and encouraged for production use. With createContext now available, they've been encouraging the community to migrate away from legacy context.

They've also been working on "concurrent React", an umbrella that describes future capabilities like "time-slicing" and "Suspense". Long-term, there are questions about how synchronous external stores like Redux will work correctly when React is running in concurrent mode.

With that in mind, we had several discussion threads about how React-Redux should work with concurrent React (#890, #950), as well as how to deal with the deprecation warnings when used in <StrictMode>

We originally planned on releasing a 5.1.0 release to fix <StrictMode> issues, but that test release turned out to be very broken. When we tried to fix the breakage, our attempts turned out to drastically hurt performance, as well as add way too much complexity.

We ultimately decided to not put out a direct fix for <StrictMode> warnings in 5.x, and instead moved on to work on v6.

The primary drivers for v6 development were:

Use createContext instead of legacy context
Fix <StrictMode> warnings
Be more compatible with concurrent React behavior going forward

We went through several experimental PRs (particularly #898 and #995) before settling on PR #1000: Use React.createContext() as the best approach. Another contributor named Greg Beaver had been working with us on the <StrictMode> issues, and he and I submitted "competing" candidate PRs for v6 with varying internal implementations. His approach turned out to be slightly faster than mine, so we went with that PR, and I was then able to further optimize the PR.

API Changes

We released React-Redux v6.0 in early December 2018. The primary changes were:

Internal: Uses createContext internally instead of legacy context
Internal: Changes to how the components subscribe and receive the updated state from the store
Breaking: The withRef option has been removed in favor of using React's forwardRef capability
Breaking: Passing a store as a prop is no longer supported

Note that there were only two minor breaking changes to the public API!. React-Redux has a fairly comprehensive suite of unit tests for connect and <Provider>, and v6 passed the same unit tests as v5 (with appropriate changes in the tests to match some of the implementation changes). v6 also ran safely inside a <StrictMode> tag without any warnings.

Because of that, for most apps, React-Redux v6 was a drop-in upgrade! We did require React 16.4 as a minimum because of using createContext, but other than that, many apps were able to just bump to the new version. The biggest set of issues came from various community libraries that relied on accessing the store instance directly from legacy context, which broke.

However, the implementation changes did result in different behavior tradeoffs. Let's look at the changes in detail.

Implementation Notes

v6: The Store State is Passed Down via `createContext`

In every version up through v5.x, the Redux store instance itself was put into context, and every connected component subscribed directly. In v6, that changed drastically.

In v6:

The Redux store state is put into an instance of the new createContext API
There is only one store subscriber: the <Provider> component

This had all kinds of ripple effects across the implementation.

It's fair to ask why we chose to change this aspect. We certainly could have put the store instance into createContext, but there's several reasons why it made sense to put the store state into context instead.

The largest reason was to improve compatibility with "concurrent React", because the entire tree will see a single consistent state value. The very short explanation on this is that React's "time-slicing" and "Suspense" features can potentially cause problems when used with external synchronous state management tools. As one example, Andrew Clark has described "tearing" as a possible problem, where different parts of the component tree see different values during the same component tree re-render pass. By passing down the current state via context, we can ensure that the entire tree sees the same state value, because React takes care of that for us.

The long-term goal was to hopefully prevent weird bugs when React-Redux is used with concurrent-mode React. (We do have other questions we need to solve regarding how to fully make use of Suspense - I wrote an extensive Reddit comment describing the aspects we might need to solve.)

Related to this, React-Redux has previously faced numerous problems around dispatching in constructors and componentWillMount (see some related issues ). Switching to passing the state via context was meant to eliminate those edge cases.

Another big reason was that we got "top-down updates" for free! Context inherently propagates top-down and ties into the render process. So, if the data for a list item is deleted, the list parent will naturally re-render before the list item does. As a result, for v6 we were able to delete that custom Subscription logic - we no longer needed it! That was less code that we had to maintain, and a slightly smaller package size as a result.

In addition, the original reason for passing the store instance no longer exists, because createContext correctly propagates value changes past shouldComponentUpdate blockers.

Finally, while this would have changed either way we handled the state-vs-store question, switching to createContext fixed bugs when mixing old and new context together. There's already been a number of bugs filed that indicate weird things happen if you use both forms of context in the same component, and Dan has also said that having old context being used anywhere in a component tree slows things down some.

Putting the store state into context did have some interesting implications around performance, which we'll get to in a bit.

Update Logic is Selectors, Used In Rendering

The new context API relies on a "render props" approach for receiving the values put into context, like:

<MyContext.Consumer>
{ (contextValue) => {
    // render something with the new context value here
}}
</MyContext.Consumer>

This means that context updates are directly tied to the wrapper component's render function.

v6 still used the exact same set of selector functions for connect as v5 did. However, there was also some additional memoization logic built into the wrapper component itself to help with the rendering process. (I initially tried adding a second inner wrapper component and doing tricks with getDerivedStateFromProps, but adding an additional selector in the one wrapper component proved to be more efficient.)

As part of that, v6 re-used the "memoized React child element" trick to indicate that the wrapped component shouldn't be re-rendered. As with v4, this is because updates are tied to the wrapper component re-rendering, so we need a way to bail out if the child doesn't need to update. (In fact, v6 doesn't even actually implement shouldComponentUpdate, because this trick is equivalent in terms of when the child updates.)

The `withRef` Option is Replaced by `forwardRef`

One of the acknowledged downsides to Higher-Order Components is that they don't easily allow you to get access to the wrapped component inside. React 16.3 introduced a new React.forwardRef API as a solution. Libraries can use this to allow end users to put a ref on the HOC, but actually get back the real wrapped component instance.

We've added that in v6, which means that the old withRef option is no longer needed. Since this does add an additional layer of wrapping (and therefore a bit more work for React to do), it's still opt-in via the new {forwardRef : true} option.

No More `store` as a Prop

This was a consequence of changing from individual subscriptions per component, to a single subscription in <Provider>. Since the components no longer subscribe, passing a store directly as a prop was meaningless, and so it was removed.

As mentioned earlier, the two main use cases for this were avoiding rendering <Provider> in tests, and allowing portions of the component tree to read from another store. The unit test use case was something that did require changes in your codebase, for folks who were actually rendering connected components in their unit tests. For the "alternate store" use case, we added the ability to pass your own custom context object as a prop to <Provider> and connected components, allowing them to read from a different store if desired, hoping that would be a sufficient alternative.

(I originally intended the API to involve passing a Context.Provider as a prop to <Provider> and passing a Context.Consumer as a prop to a connected component. However, the useContext() hook requires an entire context object, not just a consumer, despite my requests to the React team to allow it to work with just a consumer. So, the thought was that if we ever switched to using hooks internally to read from context, we'd need the whole context object in the wrapper component, so best to just require that as a prop now.)

Accessing the Store via Context Has Changed

While it's never been part of our public API, it's common knowledge that any component could get a reference to the Redux store by declaring the appropriate contextTypes and using this.context.store. Many community libraries took advantage of this. For example, connected-react-router adds an extra subscription to handle location changes, while react-redux-subspace intercepts the store and passes down a wrapped-up version that presents an altered view of the state.

Obviously, this is unsupported, and any library that does this kind of thing is risking things breaking... and in v6, that all broke because we weren't using legacy context any more. However, we wanted to allow the community the ability to build customized solutions on top of React-Redux if desired.

Each connected component needs the current store state, and a reference to the store's dispatch function so that it can implement mapDispatch correctly. In one early PR, I had the <Provider> putting {storeState, dispatch} into context to handle that.

However, in v6 final, we actually put both the store state and the store instance into context, so the context value actually looks like {storeState, store}. That way, the components could reference store.dispatch. In addition, we're exporting our default instance of ReactReduxContext. If someone wants to, they can render that context consumer, retrieve the store instance, and do something with it.

Again, it wasn't an official API, but the goal was to make it possible for folks to build on that if needed.

Performance Implications

When we were working on the attempt to fix the initial failed 5.1.0 version, we ran some benchmarks to see how the altered version compared to 5.0.7. The large perf slowdown was a big reason why we abandoned that attempt.

In response, I set up a benchmarks repo that could compare multiple versions of React-Redux together. We used that throughout the development of v6, comparing our various WIP builds against each other and v6.

Based on those benchmarks, we expected that React-Redux v6 would be sufficiently fast enough for almost all real-world apps.

Having said that, there's some caveats.

When I first envisioned switching over to createContext, I had hoped that it would be a potential boost to performance. After all, we would be going from N subscriber calls on every action down to just 1. Unfortunately, that isn't the case.

In artificial stress test benchmarks, v6 was generally slower than v5.... but the amount varies, and the reasons are complex.

Understanding the Performance Differences

In v5, a dispatched action would result in N subscriber callbacks executing. But, thanks to the heavily memoized selector functions used with connect, only wrapper components whose data had changed would actually call this.setState() to trigger a re-render. That meant that React only got involved when updates were needed.

In v6, <Provider> has the only subscriber callback. However, in order to safely handle state changes, it immediately calls setState() using the functional updater form:

    this.unsubscribe = store.subscribe(() => {
      const newStoreState = store.getState()

      if (!this._isMounted) {
        return
      }

      this.setState(providerState => {
        // If the value is the same, skip the unnecessary state update.
        if (providerState.storeState === newStoreState) {
          return null
        }

        return { storeState: newStoreState }
      })
    })

It does try to optimize some by skipping any further work if the store state hasn't changed, but this means that React will immediately and always get involved after each dispatched action.

The next issue is that React has to trace through the component tree to find all of the matching context consumers. In a simple app structure, React would sort of do this automatically anyway, because calling setState() in the root component would recursively cause the entire component tree to be re-rendered.

However, many components in the tree may be blocking updates, whether it be manually-implemented shouldComponentUpdate -> false, instances of PureComponent or React.memo(), or connect wrappers that are skipping re-renders of their children. Let's assume for sake of the example that the topmost <App> component simply has shouldComponentUpdate -> false, thus blocking updates further down when <Provider> calls setState(). In this case, React still has to traverse the entire rendered component tree to find all the consumers.

React is fast, but that work does take time. The speed of context updates affects more than just React-Redux. The maintainer of react-beautiful-dnd opened up React issue #13739: React Context value propagation performance to discuss some of the perf implications. In that thread, Dan and Dominic suggested that the current handling of nested context updates is somewhat naive, and could potentially be optimized further down the road.

Performance Benchmarks

When I finished cleanup and optimization on the PR that became v6 beta, I did a final set of runs against our benchmarks. You can view those benchmark results here. Summarizing:

Both an earlier WIP v6 iteration and the final version of the v6 PR were slower than v5, in all benchmark scenarios
That said, the final v6 build was by far the fastest of the v6 versions
The amount of relative slowdown varied based on the benchmark scenario. It was most pronounced with a totally flat tree of rapidly-updating components (~20% slower), much less so with deeper trees and other update patterns (2% slower).

I'd like to re-emphasize that these are totally artificial stress-test benchmarks!. We needed some way to objectively compare different builds for performance, and so we set up scenarios that deliberately cranked up the numbers of components and frequency of dispatched actions until all the builds began slowing down.

(Note: I'd happily accept more help from the community in fleshing out the benchmarks suite, to help us come up with some more realistic scenarios. Also, anyone ought to be able to clone the benchmarks repo, drop in a particular build of React-Redux, and replicate the approximate results on your own machine.)

v7.0

Motivations

I've observed a number of times that the single best way to get feedback on a piece of software is to release it as a final version. Doesn't matter how much you advertise alphas and betas, and beg people to try them out, A) most people won't try them out, and B) the few folks who do simply can't match the breadth of ways that people are using your code.

v6 fit that pattern to a T. Soon after we released it, folks began filing issues listing various issues they encountered trying to upgrade. The most common issue was actually with third-party libraries that were trying (and now failing) to access the store directly. Notable examples included connected-react-router, react-redux-firebase, react-redux-subspace, and even redux-form. We couldn't do anything about that, beyond poke maintainers and offer some suggestions on an upgrade path.

Other issues, however, were more concerning. The biggest one was performance. Despite my hopes that v6 would be "fast enough" in real-world apps, several users complained of noticeable slowdowns in various scenarios.

Another major issue was that our use of context turned out to be a bad foundation for eventually creating a hooks-based API. There was a giant React discussion issue on potential ways to bail out of updates caused by context in function components. Despite some early promising comments, it ultimately resulted in the React team saying that they weren't going to address that particular use case any time soon. In addition, Sebastian Markbage specifically described new context as "not meant to be used for Flux-like state propagation".

Finally, a few specific (but vocal) users raised concerns about the removal of store as a prop. The combination of Enzyme's implementation and limitations, and our use of context, effectively made it impossible for them to continue shallow-testing connected components.

In response, in early February 2019 I filed issue #1177: React-Redux Roadmap: v6, Context, Subscriptions, and Hooks. It's a monster post, laying out those concerns in greater detail, and trying to establish a direction for what a potential v7 might look like that would solve those issues. Rather than duplicate it here, please read through it to understand the challenges and how the process evolved.

In particular, the React team (and specifically Dan Abramov and Sebastian Markbage) encouraged us to move back to using direct store subscriptions in components to improve performance. Dan also encouraged us to make use of React's unstable_batchedUpdates() API as well.

Development

When I filed that roadmap issue, I didn't have any specific ideas on how we were going to reimplement connect to achieve them. Fortunately, I had some free time on my hands, and jumped right into experimenting with ideas.

Given the constraints we'd seen with accessing context in class component lifecycles, I opted to try a brand-new implementation using React's new hooks APIs. I ended up bringing back the custom Subscription class from v5, but the initial benchmark results weren't promising.

A day later, though, I made a breakthrough: wrapping connect in React.memo() drastically improved performance! Comparisons showed it was at least as fast as v5, and even faster in some scenarios.

I began publishing alphas for people to play with. Over the next few weeks, the community poked and prodded, and found several issues that we fixed. At one point, I wrote an insanely detailed step-by-step data flow analysis comparing how v5, v6, v7-alpha.1, and v7-alpha.2 process updates and re-renders.

The mile-long issue thread had a number of digressions and debates on various topics, including the need for tiered subscriptions, if a class component approach could still work, whether peer dependency bumps require a new major version of this package, and much more.

Finally, in early April, we published React-Redux v7.0 as final. The reception has been universally positive. Folks who were seeing slowdowns with v6 reported that v7 was vastly faster across the board.

Implementation Notes

`connect` Is Implemented Using Hooks

The connect wrapper component had always been a class component. As of v7, connect is now a function component that uses hooks inside. This made some aspects simpler (access to values from context, easy memoization of child elements), but complicated others (timing of effect callbacks).

We initially executed subscriptions in useEffect(), but later concluded we needed to use useLayoutEffect() to ensure they're added synchronously. Unfortunately, the React team chose to print warnings when useLayoutEffect() is used in an SSR environment. We had to do some hacky environment detection and call useEffect() in SSR instead. Neither of them run, but at least useEffect() doesn't warn.

The Return of Direct Component Subscriptions

As with v5 and earlier, v7 wrapper components all subscribe to the store directly, and only get React involved when the selector logic determines that the wrapper component needs to re-render. This was the first key step in bringing performance back to the level of v5.

Use of React's Batched Updates API

React has always had an API called unstable_batchedUpdates(). Internally, React wraps all your event handlers inside of that, which is what allows React to batch together multiple state updates from one event tick into a single render pass.

The React team urged us to use unstable_batchedUpdates() directly in React-Redux. This was tricky, because it's actually exported from renderers like ReactDOM and React Native, not the core React package. React-Redux should work with any React renderer, so we couldn't add a direct dependency on either of those. We had to write some different wrapper files so that the "react-dom" import would get loaded in a web environment, and the "react-native" import when used with RN. For apps that might be using React-Redux with an alternate renderer, we added an additional entry point that falls back to a dummy batching implementation.

There have been other Redux addons that make use of batched updates. I didn't want to dictate anything about what enhancers and store setup were needed. Instead, I was able to use the batching inside our custom Subscription class, and updated <Provider> to create a root subscription.

Use of `React.memo()` for Prop Optimizations

connect has always implemented optimizations similar to what React.PureComponent now does, but more extensive. It does checks on incoming props from the parent component, but ultimately only renders if the merged stateProps + dispatchProps + ownProps have changed.

React 16.6 introduced React.memo() as an alternative to use of shouldComponentUpdate or PureComponent. Like PureComponent, it checks to see if updates are necessary by doing a shallow comparison of previous and current props. Unlike PureComponent, which is an alternate base class component, React.memo() is a new component type that can wrap around either class or function components. In addition, it returns a very special object that looks like {$$typeof: REACT_MEMO_TYPE, type : WrappedComponent, compareFunction} (see implementation reference). This is interesting, because up until now all React components have been functions of some kind (as JS classes are actually functions). For the first time, a React component type may actually be an object instead, and so code that tries to determine if a value is a component by seeing if it's a function is now wrong.

Naturally, this meant that once we released v7, we began getting issues saying that people's code had broken because there were checks expecting all components are functions.

API Changes

Return of `store` as a Prop

Now that components are subscribing to the store themselves again, it was easy to re-add the ability for connected components to accept store as a prop once more. That resolved the issues from it being removed.

New `batch()` API

Since we'd already gone to the trouble of ensuring that we could import unstable_batchedUpdates() in both web and RN environments, we decided to re-export it as a public API named batch(). This allows end users to wrap parts of their code that are triggering multiple state updates outside of React's event handlers (such as an async function or a thunk), and minimize the number of re-renders that occur.

v7.1: Hooks?!?!

Motivations

As soon as React hooks were announced, people began asking when React-Redux would include a hooks-based public API. (The React Hooks FAQ even mentioned "useRedux()" as a hypothetical hook.)

By the time React hooks were officially released, there were already numerous third-party Redux hooks libraries. (I later put together a spreadsheet comparing the various library APIs and their popularity.)

Clearly, it was a question of when we would develop and publish a hooks API, not "if".

Development

The v7.0 work derailed the hooks API discussions for a while. As we found out in React #14110: Provide more ways to bail out inside Hooks, there's no way to stop updates caused by useContext(). That incredibly long discussion led to the React team advising us to switch back to direct subscriptions, and that meant that getting v7.0 out the door was a prerequisite for any kind of a hooks API.

After the initial hooks discussion thread got derailed, I opened up a new API discussion thread in February based on some of the v7 discussions. The discussion continued there for a while, but I didn't pay much attention. In fact, as of late March 2019, I replied to an "ETA?" question saying I was busy, and it was unlikely to be any time soon.

But, with v7 in beta and nearing release, my mind started poking at the hooks question. About that time, I published a Twitter poll asking what I should focus my time on next, and 82% selected "Hooks". Clearly, the people had spoken :)

I started actively participating in the discussion thread, and we soon figured out there were some major issues we'd have to deal with. In particular, we couldn't enforce top-down updates in a hooks environment, because v7 relies on overriding context values to pass down nested Subscription instances, and you can't render context values from a hook. This meant users would potentially encounter the "zombie child" issue again.

I finally spent a couple days doing some analysis of all the third-party hooks libs I'd seen and the various approaches that had been offered, and wrote a summary of how I thought we might be able to move forward.

There were some ensuing digressions on topics such as use of Proxies to track updates and whether we could use the v6 and v7 approaches in parallel. After much extended debate, we concluded that we'd basically just have to give up on a technical solution to the "zombie child" issue, document the potential concerns, and move on.

A user named Jonathan Ziller had put together a package implementing his proposed set of hooks APIs, and I finally suggested we should take that implementation as a PR. After more bikeshedding over hook names, we finally published our first alpha, which contained five hooks:

useSelector
useActions
useDispatch
useRedux
useStore

The alpha cycle led to another monster discussion thread (250 comments). During that process, we made three major changes:

v7.1-alpha.3 dropped useRedux, on the grounds that it didn't provide anything useful
v7.1-alpha.4 dropped useActions, after Dan Abramov strongly argued that it added too much complexity (including naming and variable scoping clashes)
v7.1-alpha.5 changed from shallow to reference equality for determining updates, under the idea that users would be more likely to select specific values rather than returning grouped objects

After sitting in alpha for a couple months, we finally zoomed through the RC stage and put out React-Redux v7.1.0 in early June. (I was about to give a talk based on this post at ReactNext, so we ended up publishing v7.1 the night before the talk so I could announce it was live.)

API Changes

Hooks!

As mentioned, we ultimately wound up shipping three hooks:

useSelector: subscribes to the store and returns the selected value
useDispatch: returns the store's dispatch function
useStore: returns the store instance itself

Due to the lack of top-down subscription enforcement, we made sure to document potential edge cases so that people are aware of those issues.

We also ended up adding copy-pasteable recipes for useActions and useShallowEqualSelector to the docs as well.

The Future

The existing connect API has been incredibly successful overall, and there's hundreds of thousands of apps using it. Our hooks API is new and certainly not as battle-tested, but we iterated on it sufficiently that I'm confident in how it works.

I'm truly hoping that React-Redux has stabilized after all the v6/v7 churn. I'm thankful that we managed to keep the public APIs consistent, since that meant that most folks were able to upgrade v5->v6->v7 without any real breaking changes. Still, I dislike that we've had to bump through a couple major versions, and hopefully we can leave things alone for a while.

But, a maintainer's work is never done. Some possible future considerations:

Alternate APIs

Prior to the announcement of hooks, we were frequently asked to provide a "render props" form of connect. Now that hooks are a thing, that's unlikely to ever happen.

Beyond that, perhaps there's some alternative API approach that would be easier to use and work better with concurrent React down the road, and we just haven't thought of it yet.

Concurrent React

The React world has been eagerly awaiting the release of React's "Concurrent Mode" since Dan helped publicize it in his JSConf Iceland 2018 talk "Beyond React 16". The React team published a roadmap in late 2018 suggesting they hoped to have it out by mid-2019, but as of June that hasn't happened yet, and recent comments indicate it'll still be a while.

Flarnie Marchan (formerly on the React core team) did a great talk at ReactNext in June entitled "Ready for Concurrent Mode?", where she laid how a summary of how Concurrent Mode works and some potential issues with existing code. It's very worth watching.

Long-term, we don't know how React-Redux will be able to fully interact with Concurrent Mode, largely because it isn't out and documented yet for us to be able to understand it. Someone just filed an issue asking about our Concurrent Mode compat status, and the answer is: "we dunno, we'll figure it out down the road". Keep an eye on that issue for future discussions.

Future Context Improvements?

The most disappointing thing about v6 is that it worked, it just wasn't fast enough for real-world usage. It's possible that React might someday make changes to the context API that would allow us to reconsider context-based state propagation as a viable approach again.

As a potential example, Josh Story just filed React RFCs describing two potential rewrites of context: lazy context propagation for faster updates with less overhead, and context selectors to determine if contexts should update, as a replacement for observedBits. He also filed a proof-of-concept React-Redux PR to rewrite connect to use that context selectors implementation. Obviously, a lot would have to happen before we could ever actually use that (RFCs accepted, changes merged into React, new versions published), and it would require a new major version of React-Redux, but there's some potential there.

Magic?

Back when v6 was in development, I wrote a long post on ways we could potentially use Proxies for tracking state dependencies and optimize context updates based on that info: React-Redux issue #1018: Investigate use of context + observedBits for performance optimization.

Since then, Daishi Kato has been experimenting with various similar approaches, and currently has a small library called reactive-react-redux that implements a Proxy-based useTrackedState() hook as an alternative to React-Redux. Long-term, that kind of approach is very intriguing.

I'd certainly love to hear feedback from the community on what forms of "magic" are acceptable, especially around optimizing component updates.

Final Thoughts

Hopefully this journey through time and release notes has been informative. As you've seen, React-Redux has never been "magic" - just smart about implementing optimizations so you don't have to. For all the internal complexity, it's still just a matter of subscribing to the store, checking to see what data your component needs, and re-rendering it when necessary. The implementations have changed, but the goals haven't.

This should also help explain why you should use React-Redux instead of trying to write subscription logic yourself in your components. The Redux team has put countless hours into optimizing performance, handling edge cases, and dealing with changes in the ecosystem. You should take advantage of all the hard work that's gone into this API! :)

As always, if you've got questions, please leave a comment, file an issue, or ping me @acemarke on Reactiflux and Twitter.

Further Information

ReactNext 2019 talk: A Deep Dive into React_Redux

Rewriting Your Git History and JS Source for Fun and Profit

Sat, 17 Nov 2018 20:00:00 -0500

Intro

I just completed a large-scale rewrite and cleanup of our team's Git repository. I learned a lot working on this task and came up with some nifty and useful techniques in the process, so I'd like to share what I've learned.

In order to keep this post from getting any longer that it already is, I'll be referencing a bunch of external articles and assuming that you, the reader, have taken the time to read them and understand most of the concepts involved. That way I can go into more detail on the stuff I actually did.

To summarize the rest of the cleanup task:

I filtered out junk files from the repo's history, shrinking the base repo size by over 70%
I automatically rewrote our ES5 Javascript source files to ES6 syntax throughout the entire history, as if they had "always been written that way"

I wrote a bunch of scripts and code for this task. I've created a repo with slightly cleaned-up copies of these scripts for reference . I'll show some code snippets in this post, but point to that repo for the full details.

Note: This post is absurdly technical and deep, even for me :) Hopefully people find this info useful, but I also don't expect it to be widely read. This is mostly a chance for me to publicly document all the stuff I did, as a public service.

Background

Why Rewrite History?

My current project got started about six years ago. We used Mercurial for the first year, then migrated to Git. The repo's .git folder currently takes up about 2.15GB, with about 15,000 commits. There's several reasons for that. We've historically "vendored" third-party libs by committing them directly to the repo, including a lot of Java libraries. We've also had some random junk files that were accidentally committed (like a 135MB JAR file full of test images).

Unfortunately, because of how Git works, any file that exists in the historical commits has to be kept around permanently, as long as at least one commit references that file. That means that if you accidentally commit a large file, merge that commit to master, and then merge a commit that deletes the file, Git still keeps the file's contents around.

So, we had junk files in the history that should never have been committed, and we had old libraries and other binaries that were not going to be needed for future development. We just wrapped up a major development cycle, and I wanted to clean up the repo in preparation for the next dev cycle. That way everyone's clones would be smaller, which would also help with CI jobs.

Dealing with History Changes

However, Git commits form an immutable history. Every commit references its parents by their hashes. Every commit references its file tree by a hash. Every file tree references its files by the hashes of their contents. That means that if you literally change a single bit in one file at the start of the repo's history, every commit after that would have a different hash, and thus effectively form an "alternate history" line that has no relation to the original history. (This is one of the reasons why you should never rebase branches that have already been pushed - it creates a new history chain, and someone else might be relying on the existing history.)

My plan was to create a fresh clone of our repo, and rewrite its history to filter out the files we didn't need for future work. We'd archive the old repo, and when the next dev cycle starts, everyone would clone the new repo and use that for development going forward.

The Codebase

Our repo has two separate JS client codebases which talk to the same set of services. The services are written in a mixture of Python and Java.

The older JS client codebase, which I'll call "App1", started in 2013, and the initial dev cycle resulted in a tangle of jQuery spaghetti and global script tags. I was able to convert those global scripts to AMD modules about halfway through that first dev cycle, and spent the second half of the year refactoring the codebase to use Backbone. We continued to use Backbone for new features until early 2017, when we began adding new features in React+Redux, and refactoring existing features from Backbone to React. We didn't have a true "compile" step in our build process, so we were limited in what JS syntax we could use based on our target browser environment. I finally upgraded the build system from Require.js to Webpack+Babel in late 2017, and that allowed us to finally start using ES6 modules and ES6+ syntax. Since then, all of our new files have been written as ES6 modules, and we've had to do back-and-forth imports between files in AMD format and files in ES6 module format.

"App2" was originally written using Google Web Toolkit (GWT), a Java-to-JS compiler. We completed a ground-up rewrite of that client in React+Redux during this dev cycle (and I took great joy in deleting the GWT codebase, particularly since I'd written almost all of that myself). This codebase was written using Wepack, Babel, and ES6+ from day 1.

Because of its longer and varied dev history, App1's codebase is a classic example of the "lava layer anti-pattern" (and I will freely admit that I'm responsible for most of those layers). It's currently about 80% Backbone and 20% React+Redux, and we hope to finish rewriting all the remaining Backbone code to React over the next year or two. In the meantime, the mixture of AMD and ES6 modules is a bit of a pain. Webpack will let you use both, but you have to do some annoying workarounds when importing and exporting between files in different module formats (like adding SomeDefaultImport.default when using an ES6 default export in an AMD file)

The Plan Grows

Our team hasn't exactly been consistent with our code styling. In theory we have a formatting guide we ought to be following, but in practice... eh, whatever :)

I've been planning to set up automatic formatting tools like Prettier for our JS code and Black for our Python code. However, the downside of adding a code formatter to an existing codebase is that you inevitably wind up with a "big bang" commit that touches almost every line and obscures the actual change history of a given file. If you do a git blame (or "annotate"), at some point every line was last changed by "Mark: REFORMATTED ALL THE THINGS", which isn't helpful. There's ways to skip past that, but it's annoying.

At some point I realized that if I was going to be rewriting the entire commit history anyway by filtering out junk files, then I could also apply auto-formatting of the code at every step in the history, to make it look as if all our code had "always been formatted correctly". That led me to another bigger realization: I could do more than just reformat the code - I could rewrite the code!.

I'd seen mentions of "codemods" before - automated tools that look for specific patterns in your code and transform them into other patterns. The React team is especially fond of these, and has provided codemods for things like renaming componentWillReceiveProps to UNSAFE_componentWillReceiveProps across an entire codebase.

it occurred to me that I could automatically rewrite all of our AMD modules to ES6 modules, and upgrade other syntax to ES6 as well. And, as with the formatting, I could do this for the entire Git history, as if the files had been written that way since the beginning.

Naturally, I ran into a bunch of complications along the way, but in the end I accomplished what I set out to do (yay!). Here's the details of how I did it.

Filtering Junk Files from History

Short answer: use The BFG Repo-Cleaner. Done.

Slightly longer answer: it does take work to figure out which files and folders you want to delete, and set up a command for the BFG to do its work.

Note: most of these commands are Bash-based and require a Unix-type environment. Fortunately, Git Bash will suffice on Windows.

Finding Files to Delete

There's a couple ways to approach this.

The first is to look for specific large files in the history. Per this Stack Overflow answer, here's a Bash script that will spit out a list of the largest files and their sizes:

git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| sed -n 's/^blob //p' \
| awk '$2 >= 2^20' \
| sort --numeric-sort --key=2 \
| cut -c 1-12,41- \
| numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

The 2^20 searches for files larger than 1MB, and can be adjusted to look for other sizes.

Anoher approach is to look at the names of every file path that has ever existed in the repo, to get an idea what might not be relevant any more. Thanks again to Stack Overflow for this answer:

git log --pretty=format: --name-only --diff-filter=A | sort -u > allFilesInRepoHistory.txt

Skimming through the list of all files turned up a bunch of junk from early in the development history that wasn't in our current tree, and could safely be removed.

Preparing the Filtering Command

The BFG Repo-Cleaner supports deleting top-level folders by name, but not nested folders. For that, I had to write a script that looked for all files matching a given folder/path prefix, and write all matching blob hashes to a file that could be read by the BFG. (Pretty sure this started from another SO answer, but can't find which one atm.)

First, write a text file called nestedFoldersToRemove.txt with each path prefix to delete on a separate line:

ParentFolder1/nestedFolder1/
ParentFolder2/nestedFolder2/nestedFolder2a/
ParentFolder3/someFilePrefix

Then, run this script inside a repo to generate a file containing just the blob IDs that match those prefixes

readarray -t folders <  "../nestedFilesToRemove.txt"

for f in "${folders[@]}"
do
    echo "Finding blobs for ${f}..."
    git rev-list --all --objects | grep -P "^\w+ ${f}*" | cut -d" " -f1 >> ../foundFilesToDelete.txt
done

If you want to see which files are getting deleted, remove the | cut -d" " -f1 portion to generate lines like ABCD1234 Path/to/some/file.ext

Finally, put together a list of the top-level folders you want deleted as well.

In some cases, I wanted to nuke all the old files in a folder, and only keep what was in there currently. In those cases I went ahead and specified the whole folder as a search path, because the BFG by default will preserve all files in the current HEAD.

Running the BFG

Once you know all the files you want cleaned up, you need to make sure that your original repo does not have those in the tip of its history. If any of them still exist, add commits that delete those files.

Once that's done, it's time to nuke stuff!

# Clone the existing repo, without checking out files
git clone --bare originalRepoPath filteredRepo

# Run the BFG, deleting specific top-level folders and nested folders/files
java -jar bfg-1.13.0.jar --delete-folders "{TopLevelFolder1,TopLevelFolder2}" --strip-blobs-with-ids ./fileIdsToDelete.txt filteredRepo

After the BFG has rewritten the filtered repo, you need to have Git remove any blobs that are no longer referenced:

git reflog expire --expire=now --all && git gc --prune=now --aggressive

You should now have a much smaller repo with the same commit authors and times, but a different line of history.

Rewriting JS Source with Codemods

Codemods for Converting to ES6

There's plenty of tools out there for rewriting JS source code automatically. My main concern was finding codemod transforms to do what I needed: convert our AMD modules to ES6 modules, and then fix up a few more bits of ES6 syntax on top of that.

I found most of what I needed in two repos:

5to6/5o6-codemod
- amd: converts AMD module syntax to ES6 module syntax
- named-export-generation: Adds named exports corresponding to default export object keys. Only valid for ES6 modules exporting an object as the default export.
cpojer/jscodemod
- no-vars: Conservatively converts var to const or let
- object-shorthand: Transforms object literals to use ES6 shorthand for properties and methods.
- trailing-commas: Adds trailing commas to array and object literals

There's many other transforms I could have used, but this was sufficient for what I wanted to do.

Writing a Custom Babel-Based Codemod

I mentioned that we had some funkiness in the JS code as a result of cross-importing between AMD and ES6 modules. One common pattern was that an AMD module would do:

define(["./someEs6Module"], 
function(someEs6Module) {
    const {named1, named2} = someEs6Module;
});

Transformed into ES6, this would be:

import someEs6Module from "./someEs6Module";

const {named1, named2} = someEs6Module;

However, the ES6 module in question might actually only have named exports, and no default export. This only worked because of Webpack magic.

When I tried running the transformed code, Webpack gave me a bunch of errors saying that these imports didn't exist. So, I opted to write a codemod that found all named exports, and if there was no default export, generated a fake default export object containing those, as a compatibility hack.

I couldn't find anything related to this that worked with jscodeshift. However, I did find jfeldstein/babel-plugin-export-default-module-exports, which almost did what I wanted. I figured I could hack together some custom changes to it. Since it was a Babel plugin, I needed a different tool to run that codemod. Fortunately, square/babel-codemod lets you run Babel plugins as codemods.

Thanks to the AST Explorer tool and some assistance from Twitter, I was able to hack together a plugin that did what I needed.

Initial Conversion Testing

I started off by trying to run each of these transforms as a separate step. I created a shell script that called jscodeshift multiple times, each with the path to a single transform file. I also called Prettier to do some formatting.

While doing that, I ran across some issues with Babel not recognizing the dynamic import() syntax, so I added a couple sed commands to rewrite those temporarily:

# Before anything else, replace uses of dynamic import
sed -i s/import/require.ensure/g App1/src/entryPoint.js

# Do other transforms
yarn jscodeshift -t path/to/some-transform.js App1/src

yarn codemod -p my-custom-babel-plugin.js App1/src

# Format the code
yarn prettier --config .prettierrc --write "App1/src/**/*.{js,jsx}"

# Undo replacement
sed -i s/require.ensure/import/g App1/src/entryPoint.js

Using this script, I was able to process a current checkout of our codebase automatically.

Formatting Python Code

While the majority of my focus was on our JS code, we've also got a bunch of Python code on our backend. I figured I'd take this chance to do some auto-formatting on that as well. I reviewed the available tools, and settled on Black, largely because its highly-opinionated style is almost identical to how we were writing our Python anyway.

Iterating through Git History

My ultimate goal was to iterate through every commit in the history of the already-filtered Git repo, find any relevant JS and Python files, transform the versions of the files as they existed in that commit, and create new commits with the same metadata but updated file trees containing the transformed files.

I'd done some prior research and read through a bunch of articles. The most relevant was A tale of three filter-branches, which compared three different ways to iterate using the git filter-branch command, and showed that the third way was the fastest.

Understanding the Index Filter Logic

In general, git filter-branch will iterate over the commit history, and run whatever additional commands you want at each commit. These could be "inline" shell commands, or separate scripts / tools.

The third approach shown in that post involved using git filter-branch --index-filter, and checking each commit to see which files had actually been added/changed/deleted. Fortunately, the author of that post chose to write the per-commit logic as a Ruby script. Reading that was extremely helpful in understanding what was going on.

I'll summarize the steps:

Retrieve the current commit ID from the filter-branch environment
Look up the original parent commit ID
Look up the ID for the rewritten form of the parent commit
Reset the Git index to the tree contents of the rewritten parent commit
Diff the original parent tree and original commit tree to see which files changed
For each added/changed/removed file:
- If it's a file we're interested in, transform it, and add the transformed version to the Git index
- Any other added/changed files we don't care about should just be added to the index as-is
- If it was removed, delete it from the index
Create a new commit with the original metadata and the transformed tree

Note that the Ruby script was specifically interacting with Git via low-level "plumbing" commands like git cat-file blob and git update-index. In addition, note that it was shelling out to call Git's commands as external binaries.

Speeding Up the Filtering Process: Iterating Commits in Python

I started by trying to port some of the Ruby script's logic to Python. My first attempt was just to run the same Git commands, capture the list of changed files, filter them based on the source paths of the JS and Python files I was interested in, and print those. I used the great plumbum toolkit to let me easily call external binaries from Python.

When I tried running git filter-branch --index-filter myScript.py, it worked. But, Git estimated that it would take upwards of 16 hours just to iterate through 15K commits and print the list of changed files. My guess was that a large part of that had to do with kicking off so many external processes (especially since I was doing this in Git Bash on Windows 10).

I knew that the libgit2 library existed, and that there were Python bindings for libgit2. I figured the process would run faster if I could somehow do all of the Git commands in a single process, using pygit2 to iterate over the history.

I experimented with pygit2 and figured out how to iterate over commits using commands like:

for commit in repo.walk(repo.head.target, GIT_SORT_TOPOLOGICAL | GIT_SORT_REVERSE):
    # do something with each commit

Since pygit2 also has APIs to manipulate the index, I was able to put together a script that replicated the logic from the Ruby script, but all done in-process. I think the initial script I wrote was able to loop over the history and print every JS/Python file that matched my criteria, in about 30 minutes or so. Clearly a huge improvement.

Speeding Up the Filtering Process: Using `pylter-branch`

I was curious if anyone else had done something like this already. I dug through Github, and came across sergelevin/pylter-branch. Happily, it already did what I wanted in terms of iterating over commits, doing some kind of rewrite step, and saving the results, and provided a base "repo transformation" class that could be subclassed to define the actual transformation step.

I switched over to using that, and it actually seemed to iterate a bit faster.

Optimizing the Transformation Process

My original plan for the rewrite was to run these steps for each commit:

Filter the list of added/changed files for the JS and Python files I was interested in
Write each original file blob to disk in a temp folder, with names like ABCD1234_actualFilename.js
Run all of the JS codemods and Python formatting on all files in that folder
Write the changed files to the Git index and commit

However, I knew that all of the file access and external commands would slow things down, so I began trying to find ways to optimize this process.

Speeding Up JS Transforms: Combining Transforms

I had six different JS codemods I wanted to run. Five of them required jscodeshift, the other required babel-codemod. Originally, this would have required six separate external processes being kicked off for every commit.

A comment in the jscodeshift repo pointed out that you could write a custom transform that just imported the others and called them each in sequence. Using that idea, I was able to run all five jscodeshift transforms in one step.

That left the one Babel plugin-based transform as the outlier. There was a jscodeshift issue discussing how to use Babel plugins as transforms, and I was able to adapt Henry's example to run my plugin inside of jscodeshift. That meant all six transforms could run in a single step.

I ultimately copied the transform files locally so I didn't have to try installing them as separate dependencies.

Speeding Up Python Formatting

That cut down on a lot of the external processes, but I was still writing files to disk for every commit. Since my script was written in Python, and the Black formatter is also Python, I realized I could probably just call it directly.

I set up some logic to put all matching Python files into an array that looked like [ {"name" : "ABCD1234_someFile.py", "source" : "actual source here}], and just directly call Black's format_str() function on each source string. That eliminated the need to write any Python file to disk - I could read the source blobs, format them, and write the formatted blobs back to the repo, all in memory without ever writing any temp files to disk.

Speeding Up JS Transforms: Creating a Persistent JS Transform Server

Since the JS transforms were all done using tools written in JS, calling that code directly wasn't an option. To solve that, I threw together a tiny Express server that accepted the same kind of file sources array as a POST, directly called the jscodeshift and prettier APIs to transform each file in memory, and returned the transformed array. This meant I didn't have to have any temp files written to disk. It also meant there were no other external processes starting up for every commit. All I had to do was start the JS transfom server, and kick off the commit iteration script.

I later realized I could speed up things even further by parallelizing the JS transforms step to handle multiple files at once. The workerpool library made that trivial to add. I set up the pool of workers, and for every request, mapped the array to transform calls that returned promises, and did await Promise.all(fileTransforms). This was another great improvement.

Handling Transform Problems

I ran into a bunch of problems along the way. Here's some of the problems I found and the solutions I settled on.

Python String Conversions

Both Black and pylter-branch required Python 3.6, so I was using that. pygit2 reads blobs as bytes, but some of the work I needed to do required strings. I also found a few files that had some kind of a "non-breaking space" character instead of actual ASCII spaces. Fixing this required doing some unicode normalization:

def normalizeEntry(fileEntry):
    newText = normalize("NFKD", str(fileEntry["source"], 'utf-8', 'ignore'))
    return update(fileEntry, {"source": newText})

Handling JS Syntax Errors

Turns out our team had made numerous commits over the years with invalid JS syntax. This included things like missing closing curly braces, extra parentheses, wrong variable names, Git conflict markers, and more. Because the JS transforms are all parser-based, both jscodeshift and prettier could throw errors if they ran across bad syntax, causing the transforms for that file to fail.

I first had to know which files were broken, at which commits. I added error handling to the JS transform server to catch any errors, return the original (untransformed) source, and continue onwards. I then added handling to write both the current string contents and the error message to disk, like:

- /js-transform-errors
    - /some-commit-id
        - ABCD1234_someFile.js    // bad source file contents
        - ABCD1234_someFile.json  // serialized error message

I'd run the conversion script for a while, let it write a bunch of errors, then kill it and review them.

I wound up writing dozens of hand-tested regular expressions to try to fix those files whenever they came up. Using an interactive Python interpeter (in my case, Dreampie), I'd read the original blob into a string, fiddle with regexes until I got something that matched the problematic source, create a substitution that fixed the issue, and then paste the search regex and the replacement into a search table in my conversion script. The conversion logic would then check to see if any file in a given commit matched the bad file paths, and run each provided regex in sequence on the source in memory. This would fix up the source to be syntactically valid before it was sent to the JS transform server, allowing it to be transformed correctly.

I eventually gave up on fixing every last issue. So, a few files in the history wound up with commit sequences like:

FIXED_SOURCE_C    // ES6 syntax
BROKEN_SOURCE_B   // original AMD syntax
FIXED_SOURCE_A    // ES6 syntax

Fortunately, most of those were far enough back in the history to not really cause issues with the blames.

This was probably the most annoying part of the whole task.

Mistaken Optimization of ES6 Files

All of the source files for App2 were already ES6 modules, and about 20% of App1's files were also ES6. Five of the six codemods were about converting AMD/ES5 to ES6, so I figured I could speed things up by not running those on files that were already ES6.

I modified the JS transform server to accept a {formatOnly : true} flag in the file entries, in which case it would skip the transforms and just run Prettier on the source. That was fine.

I then tried to have the Python script detect which files were already ES6, and did so by checking to see if the strings "import " or "export" were in the JS source. That turned out to be a mistake, as we had a bunch of AMD files with those words in comments or source already. I did one conversion run that I thought would be "final", but realized afterwards that many files hadn't been transformed at all.

I eventually settled for just checking to see if the file was part of App2's source, and ran the complete transform process on everything in App1.

Running the Conversion

I did lots of partial and complete trial runs to iron out the issues. The actual final conversion run, with all of the transforms and formatting, took right about 5 hours to complete. That's certainly a huge improvement over the base git filter-branch command, which probably would have taken upwards of 20-some hours. (This proved to be particularly helpful when I realized that I'd screwed up a "final" run with the bad "skip ES6" optimization, and had to re-run the whole process again.)

I had added a bunch of logging to the conversion script, including running values of elapsed time, time per processed commit, and estimated time remaining. This was really useful, and it was extremely satisfying to watch the commit details fly by in the terminal. (Fun side note: a couple hours after kicked off the actual final conversion run in the background, a popup window informed me that IT had pushed through a forced Windows reboot due to updates. That meant it was a race for the conversion to complete before the reboot, and at one point it was 3.5 hours remaining for the conversion, 3 hours left until the reboot. Fortunately, the conversion sped up considerably about halfway through thanks to smaller files to process.)

However, we were still doing some fixes and work in our existing repo, and had made several commits since I made the clone to start the file filtering process. Those needed to get ported to the new repo, and that turned out to be trickier than I expected.

Attempt #1: Bundles

I've used Git "bundles" before, which let you export a series of commits as a single file. That file can then be transferred to another machine, and used as a source for cloning or pulling commits into another repo.

I had assumed I could export the latest commits into a bundle and pull those directly into the newly-rewritten repo. I was wrong :( Turns out that git bundle always verifies that the target repo already has the parent commit before the first commit in the bundle, and of course since the rewritten repo had a completely different history line, that specific parent commit ID didn't exist in the new repo.

I poked at various forms of pulling, cloning, and banging my head against a wall before giving up on this approach.

Attempt #2: Cherry-Picking

I then figured I could add the original repo as a remote, git fetch the original commits into the new repo, and cherry-pick them over into the new history. Didn't go as I planned.

First, the new repo had to copy over every old blob and commit into itself. Then, when I tried cherry-picking the first "new" commit, it brought along not just the changed files from that commit, but the old versions of every other file in the old commit's file tree.

I probably could have done some kind of surgery to make that work, but I gave up.

Attempt #3: Patch Files

Git was originally intended for use with an email-based workflow, since that's what the Linux kernel team does. It has built-in commands for generating patch files, writing emails with those patches, and applying patches from emails.

I was able to generate a series of patch files with git format-patch COMMIT_BEFORE_FIRST_I_WANT..LAST_COMMIT_I_WANT. I then copied those to the new repo, and tried to apply them.

Git has two related commands. git apply takes a single standalone patch file and tries to update your working copy. git am reads an email-formatted patch file, applies the diff, and then creates a new commit using the metadata that was encoded in the email header.

When I tried to use git am, it failed. The generated patch files have lines like index OLD_BLOB_HASH NEW_BLOB_HASH for each changed file, and those lines caused Git to try to find the old file blob IDs in the new repo. Again, those didn't exist.

I finally resorted to manually deleting those index lines from each patch file, then running git am --reject --ignore-whitespace --committer-date-is-author-date *.patch. Git would try to apply a patch file, and if any hunks failed to apply correctly, write them to disk as SomeFile.js.rej and pause. I could then do manual hand-fixes to match the changes in the patch, and run git am --reject --ignore-whitespace --committer-date-is-author-date --continue, and it would pick up where it left off.

So, patch files aren't ideal, but they at least allow copying over newer changes semi-automatically while reusing the commit metadata. I probably could have found other ways to do this programmatically, but oh well.

Conversion Results

Nuking unneeded files from the repo history knocked the new repo down from 2.15GB as a baseline to "only" 600MB, a savings of 1.5GB. We still unfortunately have a bunch of vendored libs with the current codebase, but that's still a significant improvement. (Yeah, yeah, we'll look at maybe using something like Git-LFS down the road.)

The JS transformation process worked out exactly as I'd hoped. All of the versions of the JS files in our history were transformed and formatted, as if they'd originally been written using ES6 module syntax and consistent formatting from day 1. This meant that each commit retained the original metadata and relative diffs to its parent.

As an example, App1's entry point was initially a global script, but a couple months into development I converted it into an AMD module. The original diff looked like:

+define(["a", "b"],
+function(a, b) {

// Actual app setup code here

+   return {
+       export1 : variable1,
+       export2 : function2
+   }
+});

Afterwards, the equivalent commit was still by me, still on the same day, but the diff looked like:

+import a from "a";
+import b from "b";

// Actual app setup code here

+export {export1 : variable1, export2 : function2};

I did have to do a few hand-fix commits after the conversion process was done, mostly around our mixed use of AMD/ES6 modules (like removing any of use of SomeDefaultImport.default). Once I did that, the code ran just fine.

Final Thoughts

As I said, that was incredibly technical, but hopefully it was informative.

I had a fairly good grasp on how Git worked and how it stored data before this task began, but this really solidified my understanding.

I suspect there may have been some other approaches I could have used, particularly rewriting each file blob in parallel. I'm pretty happy with how this worked out, though.

The sanitized conversion scripts are available on Github. If you've got questions, leave a comment or ping me @acemarke on Twitter or Reactiflux.

Further Information

Git internal data structures: blobs, trees, commits, and hashes

Git from the Inside Out
Git from the Bottom Up
Git - What a Branch Is (good diagram showing how commits point to prior commits and file trees)
Git isn't magic

Mark's Dev Blog

Coding Career Advice: Using Git for Version Control Effectively

Introduction

Table of Contents

Git Fundamentals

Git Terms and Concepts Overview

Git Basics

Sharing Data Between Repositories

Branches

Understanding Git Internals

Git Tools

Git Techniques

Improving CLI Logging

Preparing Commits in Pieces

Stashing Changes

Working with Branches

Creating and Switching Branches

Fetching, Pushing, and Pulling Branches

Merging Branches

Feature Branch Strategies

Pull Requests

Updating Branches in the Background

Rewriting Git History

Amending Commits

Resetting Branches

Rebasing Branches

Reverting Commits

Cherry-Picking

Interactive Rebasing

Reflog

Advanced History Rewriting

Git Patterns and Best Practices

Write Good Commit Messages

Make Small, Focused Commits

Clean Up Commit History Before Pushing

Only Rewrite Unpushed History

Keep Feature Branches Short-Lived

Code Archeology with Git

Displaying Historical File Changes

Bisecting Bugs

Final Thoughts

Further Information

Idiomatic Redux: Redux Toolkit 1.0

Table of Contents

Origins

The Rise of Redux

Facing the Hype Cycle

Dealing with Complexity

In Search of a Solution

Jumpstarting the Process

Designing the Toolset

Early Implementation and Naming

Filling Out Functionality

Defining a Roadmap and Clarifying the Vision

Objectives

Speed Up Getting Started

Simplify Common Use Cases and Remove Boilerplate

Opinionated Defaults for Best Practices

No Lock-In / Add It Incrementally

Become the "Obvious Default" Way to Use Redux

Don't Solve Every Use Case, and Stay Visibly "Typical Redux"

Reasonable TypeScript Support

Honing the API

Changing the Name

The Future of Redux Toolkit

Review

Reception

Next Steps

Further Information

Idiomatic Redux: The History and Implementation of React-Redux

Intro

Table of Contents

Integrating Redux with a UI

Understanding Redux Store Subscriptions

The Standard UI Update Cycle

connect in a Nutshell

Development History of the React-Redux API

Early Iterations

Original API Design Constraints

Finalizing the API

`connect` in a Nutshell

`connectAdvanced`

v6: The Store State is Passed Down via `createContext`

The `withRef` Option is Replaced by `forwardRef`

No More `store` as a Prop

`connect` Is Implemented Using Hooks

Use of `React.memo()` for Prop Optimizations

Return of `store` as a Prop

New `batch()` API