Mark's Dev Blog

How Web Apps Work: AJAX, APIs, and Data Transfer

Sun, 22 Nov 2020 11:00:00 -0500

Web development is a huge field with a vast array of concepts, terms, tools, and technologies. For people just getting started in web dev, this landscape is often bewildering - it's unclear what most of these pieces are, much less how they fit together.

This series provides an overview of fundamental web dev concepts and technologies, what these pieces are, why they're needed, and how they relate to each other. It's not a completely exhaustive reference to everything in web development, nor is it a "how to build apps" guide. Instead, it's a map of the territory, intended to give you a sense of what the landscape looks like, and enough information that you can go research these terms and topics in more depth if needed.

Some of the descriptions will be more oriented towards modern client-side app development with JavaScript, but most of the topics in the series are fundamental enough that they apply to server-centric applications as well.

Other posts in this series cover additional topics, such as:

New terms will be marked in italics. I'll link references for some of them, but encourage you to search for definitions yourself. Also, some of the descriptions will be simplified to avoid taking up too much space or dealing with edge cases. This post in particular does not attempt to be a "how to program" tutorial or complete reference to JS, but will point out common gotchas and differences from other languages like Java, C++, and Python.

The MDN JavaScript docs have a complete set of resources on everything related to JavaScript, AJAX, and browser functionality

The Modern JavaScript Tutorial has information on built-in AJAX APIs and browser storage.

My JavaScript for Java Developers slides also cover much of this post's content as well, showing examples of JS syntax and concepts in a cheatsheet-type format. I'll link relevant sections of these slides throughout this post rather than copy entire large code blocks.

JSON

JSON Overview

JSON is the most widely used data transfer format for web apps. JSON is short for "JavaScript Object Notation", and is an adaptation of JS object/array syntax for use as a data format. All major programming languages have built-in support for reading and writing JSON data, but it naturally fits into JavaScript. JSON has become the de-facto standard data transfer format for HTTP-based APIs, and some databases even support storing and querying JSON document contents. (slides: JSON)

JSON syntax consists of normal JS objects and array definitions, including primitives like numbers, strings, booleans, and null. However, JSON does have some additional rules and restrictions compared to normal JS syntax:

No variables
No comments
All object keys must be quoted with double-quotes
The final key in objects and the final item in arrays may not have a trailing comma

The top level of JSON data may be either an object or an array (slides: (slides: JSON syntax examples))

A typical JSON document for a list of todo objects might look like:

[
  { "id": 0, "text": "Buy milk", "completed": false, "colors": ["red"] },
  { "id": 1, "text": "Clean yard", "completed": true, "colors": [] },
  {
    "id": 2,
    "text": "Read book",
    "completed": false,
    "colors": ["red", "blue"]
  }
]

Because JSON is a subset of JS syntax, it lacks several things that normal JS has. The inability to write comments is a frequently cited pain point. It also doesn't have native support for date values, so they have to be serialized as strings. You can write numbers in JSON of any size, but because JavaScript itself is limited to 64-bit floating point numbers, in practice that also limits the size of numbers you can pass via JSON.

Working with JSON

JSON content must be serialized by converting it into a single string for transfer, and then deserialized back into actual objects after it has been received.

The JS language supports this with a pair of built-in functions: JSON.stringify(value) will convert any valid value (object, array, primitive) into a complete string. JSON.parse(string) will deserialize a string of JSON text back into its corresponding values.

When sending JSON content over HTTP, the request or response should use the header content-type: application/json.

Most server frameworks have shortcuts built in for returning a JSON-formatted response. These methods often allow returning an object or array directly, and the framework takes care of serializing the data into JSON format and setting the right HTTP headers. For example, the Express server framework for Node has res.json(obj).

AJAX

AJAX is a term used to describe making HTTP requests, via JavaScript, in the background of a page, without reloading it. It's an acronym that originally stood for "Asynchronous JavaScript And XML". At the time the phrase was created, XML was the standard data transfer format used for most requests. Since then, JSON has completely replaced XML as the standard data transfer format. Today, the term "AJAX request" normally refers to fetching data in JSON format from a web server over HTTP.

Browsers have a couple built-in APIs that are used for making AJAX requests: XMLHttpRequest (XHR) and fetch. XHR is generally considered to be a clunky, low-level, and hard to use API design. fetch is intended to be a more modern replacement, but still has some noticeable annoyances in its design. Because of those, most apps use a third-party library that wraps one of those two built-in methods to provide an alternate API that is easier to use.

Note: The term "API" always means "other code that you call to get something done", but depending on context, it can mean "an actual function you call in your code", or "a remote request to a server to fetch some data". In the "AJAX" section, "API" refers to actual functions you call in your JS code. In the Data Transfer Protocols section below, it refers to remote network requests to fetch data.

Browsers can cache some HTTP response contents. This is most common with static files like HTML, CSS, and images, but other HTTP data responses can be cached as well. This is most likely to happen if the HTTP request is using the HTTP GET method. Several HTTP headers also have an effect on caching as well.

AJAX APIs and Libraries

AJAX API usage typically falls into three categories:

The built-in XMLHttpRequest and fetch APIs
Third-party wrapper libraries like axios and jQuery that wrap the core APIs
Higher-level data fetching abstractions that use the other APIs internally

There are many third-party AJAX wrapper libraries, so we'll briefly look at a couple of the most popular options here.

`XMLHttpRequest`

XMLHttpRequest, typically referred to as "XHR", is the original browser API for making AJAX requests. It was introduced by Internet Explorer around 1999, and added by other browsers around over the next few years. Despite its name, it can be used to transfer any kind of content over HTTP, not just XML.

XHR's API is based on creating an XHR object instance, and assigning callbacks to it for various request lifeycle events. That approach is notoriously hard to work with. (slides: XHR examples)

XHR is supported in all browsers, and in Node.js. Because it's available everywhere and has been around longer, it's often used as the core implementation of other AJAX libraries.

`fetch`

The fetch API is a newer built-in browser API, meant to be a replacement for XHR. It has a nicer syntax, and uses Promises for handling the responses. (slides: fetch examples)

A typical fetch request might look like:

fetch('http://example.com/api/users')
  .then((response) => response.json())
  .then((users) => {
    // do something with users data
  });

However, fetch has some behaviors that frequently surprise or annoy developers:

fetch only returns a rejected promise if the actual network request failed completely. As long as the server returns a response, fetch will return a successfully resolved promise, even if the HTTP status code was a 4xx or 5xx HTTP error.
The initial fetch('/some/url') call returns a promise containing a Response object. That object is not the actual serialized data from the server. Instead, a second call is required to ask the response to return the request body content in a specific format, such as JSON.
fetch does not send cookies to the server by default - you must include a credentials: 'same-origin' option
fetch does not automatically stringify objects or set headers when sending JSON - you must do those steps yourself

Also, fetch is not supported in Internet Explorer or Node.js.

Because of this, it's very common to use a library that wraps XHR or fetch. For cases where you want to use fetch but it's not available in your target environment, there are polyfills to add a reimplementation of fetch in environments that don't support it, such as isomorphic-fetch.

Axios

Axios is a widely popular AJAX wrapper library. Its API design is based on the "Resource" classes that were part of AngularJS 1.x, particularly the structure of its response objects.

Axios makes common tasks fairly easy. The default axios object has .get() and .post() methods, which return promises. The .post() method accepts a data value as an argument, and if provided, automatically stringifies that into JSON and adds headers to the request. Axios response objects automatically deserialize JSON content, and the body content is always available in response.data. A typical request might look like:

axios.get('http://www.example.com/api/users').then((response) => {
  // do something with response.data
});

Because it uses XHR internally, it works across all browsers as well as Node.

Axios also provides some powerful capabilities like "interceptors", which allow modification of requests and responses. This is often used for adding auth headers or transforming data.

jQuery

jQuery was historically used to enable powerful DOM manipulations, while smoothing over DOM API differences between browsers. jQuery also includes an AJAX wrapper, $.ajax(). Given the difficulties in working with the plain XHR API, jQuery's AJAX wrapper was another major reason for its popularity. (slides: jQuery AJAX)

The jQuery AJAX API is nicer to work with than XHR, but definitely still shows that it was designed in an earlier era of JS usage. It primarily takes success and error callbacks, rather than working with promises. It does have the ability to handle responses by returning a precursor to Promises, called a "deferred", and can automatically deserialize JSON data.

If you're working on a simple site and already are using jQuery for other things, it's reasonable to use jQuery to make AJAX calls. Otherwise, prefer using a modern option like fetch or axios. Definitely do not add jQuery to a site just to make AJAX calls.

Other Request Libraries

Many tools and frameworks come with their own HTTP request wrappers. For example, AngularJS 1.x had $http for general requests and a higher-level $resource class for working with REST APIs, while modern Angular has an HttpClient service.

Since the fetch API has several annoyances in its design, many users opt to write their own fetch wrapper to customize the behavior, typically by auto-converting JSON data and rejecting promises on HTTP 4xx/5xx responses.

Alternately, redaxios is a reimplementation of much of the axios API as a small wrapper around fetch.

HTTP Request Variations

HTTP is inherently a request/response protocol. That means that a server can never initiate sending a message to a client by itself. However, there are many cases when it's necessary for a server to send data to a client based on an event that occurred in the server.

Over the years, the community has come up with several workarounds. The modern solution to this problem is Websockets, but it's useful to know about these other techniques as well.

Polling

Polling is the simplest option to implement. The client continually makes requests to the server every few seconds, on a timer. However, this is usually inefficient - most of the time the server won't have anything new to report, so the new request is a waste of time and bandwidth.

Long Polling

Long polling is a variation on the polling approach. Instead of continually making requests, the client opens one request to the server. If the server doesn't have any messages to report, it sleeps the request. The client still has the connection open, and continues to wait for a response from the server. Eventually, the server either has something new to report, or a timer expires. At that point, it wakes up the request, and finally sends back a response to the original open request connection.

The client then immediately makes another request, and the server again either has something to send back immediately, or sleeps on the request.

This approach requires a bit of coordination inside the server, but is more bandwidth-efficient.

HTTP Streaming

Once an HTTP connection is opened by the client, the server may continue to send data over the open connection without closing out the response, leaving it open indefinitely.

Server Sent Events

Server-Sent Events are a specific form of HTTP streaming that is actually part of the HTTP spec. The client creates an EventSource object and adds an event listener callback, and the server writes specifically formatted content to the response over time.

Websockets

All of those HTTP-based approaches to 2-way message and data transfer have distinct limitations. Websockets were created to address those limitations. Websockets allow opening a persistent connection between a client and a server, where either side may send messages over the connection at any time. (slides: websockets)

A websocket starts out as a standard HTTP request, but the client adds a header telling the server it wants to "upgrade" the connection to a websocket. A websocket is similar conceptually to an actual OS-level networking socket, and like all HTTP requests, the data is sent over an actual socket. But, it's a higher-level abstraction, and is much more limited API-wise compared to a low-level OS socket.

Once created, either side can add event listeners to process received messages, and send messages over the connection. Messages can contain text or binary content.

Today, websockets are the standard approach used to provide live updates to a page after it's been updated, such as updates to sports scores.

CORS

HTTP requests often involve security and authorization concerns. As part of this, browsers specifically implement restrictions such as the Same-Origin Policy, so that scripts can only interact with resources loaded from the exact same server URL. Additionally, a server may only want to allow interactions with its own scripts and client code that it has served, instead of requests from any arbitrary site or client.

Browsers enforce this via Cross Origin Resource Sharing (CORS). By default, an AJAX request to any URL other than the original URL for a host page will cause the browser to make a pre-flight request to ask the server "are you okay with other sites making data requests to you?". The server may then respond with a set of URL patterns that are allowed to successfully make requests, or a value indicating "all requests are okay". If the current client matches that pattern, the browser will actually make the real request.

CORS is a frequent source of confusion for developers who are just trying to make a request and are surprised when the browser blocks it.

Note that only browsers implement CORS - HTTP requests made outside a browser, such as from another server or another non-browser environments do not enforce CORS. This can also be confusing - "it worked in my CLI tool / API test client, why doesn't it work in the browser?".

Data Transfer Protocols

HTTP provides the standard mechanism for making some kind of request and response to a server. However, there are many ways that a server may structure its URLs and the expected format of requests and responses. In order to write code that makes requests to a server, you must know what structure and request formats that server expects your client to use.

There are numerous variations on how server data APIs are structured, but there are a few common categories of API structures you'll see frequently. (slides: AJAX API design approaches)

With all these approaches, the server defines specific endpoints - combinations of URLs, HTTP methods, and expected request/response formats.

REST

REST stands for "Representational State Transfer". A REST API is an HTTP-based API that uses URLs and HTTP methods as the primary approach for determining what kind of request the client is making, and how the server should handle that request.

REST APIs are often used for CRUD (Create/Retrieve/Update/Delete) apps, where the app is conceptually doing operations to update data in a database on the server ("create this thing", "give me a list of these things", "update that thing", "delete that thing").

There are frequent arguments over what exactly constitutes a true REST API, but in general, a REST API:

Offers different URLs for each kind of "resource" that it supports
Expects different HTTP methods to be used for different kinds of operations on the same resource URL. Typically, HTTP POST/GET/PUT/DELETE map to relevant CRUD operations.
Returns different HTTP status codes to indicate success or failure of the request
Expects key pieces of data like item IDs to be included as part of the actual URL

Example REST API usage might look like:

Get list of users: GET /users
Get one user: GET /users/42
Create one user: POST /users/42 (body: {name : "Mark"})
Update user: PUT /users/42 (body: {name : "Mark"})

Many server web frameworks have built-in support for defining a set of REST API endpoints based on metadata.

RPC

RPC stands for "Remote Procedure Call". This is a general style of network request that is intended to mimic a normal function call in your code, but with the actual logic running on some other server. There have been many RPC tools over the years for many languages. Here, the focus is on RPC-style HTTP requests.

An RPC-style approach describes "methods" to be "called", instead of "resources" to be "operated on". Unlike a REST API, an RPC-style API will likely define URLs whose paths look like verbs or function names instead of nouns.

For example, an HTTP-based RPC server approach might look like:

Get a list of users: GET /getUsers
Get one user: POST /getUser (body: {userId : 42})
Create one user: POST /createUser (body: {userId : 42, name : "Mark"})
Update one user: POST /updateUser (body: {userId : 42, name : "Mark"})

Alternately, there might be a single URL endpoint, and the request could have the name of the "method to call" in the body. The JSON-RPC spec uses this approach:

POST /jsonrpc (body: {"method": "createUser", userId : 42, name : "Mark"})

Either way, the response is likely to have an HTTP 200 status code indicating that the server returned a response successfully, but the contents of the response might have some kind of a field indicating whether the actual operation succeeded, like {status: 'failed'}.

GraphQL

GraphQL is a relatively recent data transfer protocol created by Facebook. Conceptually, GraphQL is a defined format for API request and response content. In addition, there is an ecosystem of client and server tools that are typically used that abstract the details of making a GraphQL-formatted query, processing the request on the server and fetching requested data, formatting the response correctly, and caching the data on the client. Most discussions of "using GraphQL" assume use of the most popular libraries and tools for working with GraphQL.

With GraphQL, the server offers a single URL endpoint, and defines schemas that describe what data types it supports and how they relate to each other. The client sends a GraphQL query request that asks for some subset of those data types, and defines the expected structure of the response format. The server then resolves the requested data types, extracts the requested fields for each type, formats the response as requested by the client, and sends the data back. The server may also support mutations, which allow the client to create/update/delete data, and subscriptions for changes to query results over time.

GraphQL requests are almost always HTTP POSTs, and subscriptions are typically built on top of websockets.

GraphQL query syntax looks somewhat like JSON, but without commas or values:

{
  hero {
    name
    friends {
      name
    }
  }
}

which might result in this JSON response from the server:

{
  "data": {
    "hero": {
      "name": "R2-D2",
      "friends": [
        {"name": "Luke Skywalker"},
        {"name": "Han Solo"},
        {"name": "Leia Organa"}
      ]
    }
  }
}

GraphQL moves a lot of the query complexity to the server, and gives clients a lot of flexibility in what data they want to ask for. This can simplify cases where different portions of an app need to fetch different subsets of the same data and reduce the need to make numerous calls to fetch related values. On the other hand, it can be more difficult to deal with things like auth and business logic as part of the data resolution on the server, and the most popular GraphQL clients are very heavyweight.

Browser Storage

In addition to all these network requests for data fetching, browsers also offer several tools for storing data within the browser itself. These are typically used to manage data related to user sessions and preferences, but can be used for other scenarios as well.

Cookies

Browsers allow servers to set cookies - small pieces of text associated with a given URL. Any cookies set by a site will then be included on future requests to that site. Cookies are typically limited to a couple KB in length, and are most often used to store a unique session ID that can be read by the server to retrieve additional session data from a database or in memory. Cookies can have expiration timestamps set by the server.

Cookies can be a source of security issues. A server can add an HttpOnly attribute when setting a cookie to ensure that JS code cannot read or modify that cookie.

`localStorage`

localStorage is a browser API that lets client code save key/value string data, then retrieve it later. localStorage persists indefinitely, so it's often used to save user settings so they can be reloaded the next time the user visits a page. An example of this might look like localStorage.setItem('userPrefs', JSON.stringify(userData)), and then reversing the process when a page is loaded. Data in localStorage can only be read by code from the same origin, and is limited to about 2MB in size.

`sessionStorage`

sessionStorage has the same basic API methods as localStorage, but is isolated on a per-tab basis. It will persist between tab reloads, but is cleared when the user closes the tab.

`indexedDB`

A more powerful database-style storage system that can hold larger amounts of data.

Client URL Routing

In HTTP and Servers, we talked about the idea of routing: determing application behavior based on the URL. For application servers, routing means looking at a URL and HTTP method, and choosing the right application handler logic to process that request.

Client-side applications can do their own form of routing. Browsers offer access to the current URL via window.location, and the browser's URL history via a history object.

Client applications can parse the URL and use that as the basis for dynamically showing and hiding portions of the UI, without needing to make an additional request to the server. Client-side routers typically offer special forms of a link tag that handle clicks by calling history.push('/new-route') to alter the URL, and then read the updated route to switch to showing a different piece of UI.

However, client routing does add additional complications. If the browser tries to reload a client-routed URL by itself, the server now has to handle that URL somehow with a valid response. Typically, the server is configured so that if a URL like /some-route is not recognized, it returns the entire index page with the full JS bundle as the response, just as if the browser requested the / URL. Once the content is loaded, the client router kicks in, sees that /some-route is the active URL, and immediately shows the relevant content on the client side.

Further Resources

JSON
- MDN: Working with JSON
- What is JSON?
AJAX
HTTP Request Variations
Websockets
CORS
- Understanding CORS
- CORS Tutorial: A Guide to Cross-Origin Resource Sharing
Data Transfer Protocols
Browser Storage
Client Routing

How Web Apps Work: JavaScript and the DOM

Sun, 15 Nov 2020 16:00:00 -0500

Other posts in this series cover additional topics, such as:

The MDN JavaScript docs have a complete set of resources on JavaScript ranging from intro-level tutorials to detailed API references.

FreeCodeCamp has multiple courses that cover JavaScript usage as well as other aspects of web dev.

The Modern JavaScript Tutorial is an excellent and thorough set of explanations of JS syntax and DOM APIs.

JavaScript Overview

JavaScript is a dynamic interpreted language that is primarily used in web browsers, but can also be used outside a browser environment. It was created in the mid-1990s by Brendan Eich at Netscape, and takes inspiration from the Java, Scheme, and Self languages. Since then, it has grown from a tiny scripting language to the world's most popular language. It's definitely not a "toy" - it can be used to write anything from small scripts to HTTP server applications to complex client-side web apps with hundreds of thousands of lines of code.

JS is multi-paradigm - it can be used for Object-Oriented Programming similar to Java or C#, but can also be used in a Functional Programming style as well.

Note: Despite its name, JavaScript has no actual relation to the Java language! Some aspects of JS's syntax were indeed borrowed from Java, but they are completely different languages. (slides: Java vs JavaScript comparison table)

Language Evolution

To understand modern JS, it helps to understand how the language has evolved over time. (slides: JS language timeline) There's a neat timeline of milestones in JS history that helps visualize some of the key events and changes over time.

JS was standardized early on by the ECMA standards organization. Since "JavaScript" was trademarked, the standardized name is "ECMAScript", and the specification is often referred to as the "ES Spec". Early revisions were numbered: "ES2", "ES3", etc. The ES spec is maintained by a committee known as TC39.

The ES spec has had several new revisions published over time, which added new language syntax and features. However, the ES4 spec revision grew so large that it was abandoned in 2003 due to disagreements over potential features.

As a result, when the ES5 spec came out in 2009, it only added some relatively minor changes to the language.

The next major spec revision, ES6, didn't come out until 2015. Because of the long wait between revisions and the rapid growth in JS popularity and usage, the ES6 spec was huge and doubled the amount of syntax and features in the language.

You can roughly divide JS history and syntax into "before ES6" and "after ES6". All the old syntax still works, but ES6 drastically changed how JS developers write code.

After the ES6 spec was finished, the TC39 committee changed how they design changes to the JS language. Now, individual features are proposed and go through a series of stages over time (proposal, early implementation, completed implementation, finalized). New spec revisions are published yearly, and any new features finalized since the last revision are added to the new spec. Revisions are now known by year: ES2016, ES2017, and so on. (slides: TC39 stages and process)

To give a sense of how these revisions have changed the language over time, here's an incomplete list of new features by revision:

ES3 (1999): regular expressions; try/catch blocks and exceptions
ES5 (2009): object and array methods; JSON parsing; function binding; syntax cleanup
ES6 (2015): let/const variable declarations; arrow functions; classes; object literal shorthand; template strings; promises; generators; default function arguments; array spread syntax; module syntax; proxies
ES2016: Array.includes(), ** exponent operator
ES2017: async/await functions; Object.values() / Object.entries(); trailing commas in function args
ES2018: async iteration; object rest/spread operators; Promise.finally()

As you can see, ES6 was huge, while ES2016 was tiny. Since then, the rate of new features has been slower, but steady. (slides: ES spec revisions and features)

Runtime Environments

JavaScript was originally invented to give web browsers the ability to have interactive logic running inside a web page. By the mid 2000s, all browsers included a JS interpreter, and developers were starting to write larger client-side applications in JS.

In 2009, Ryan Dahl announced Node.js - a JS runtime for executing JS outside of a browser environment. Node is built on top of Chrome's V8 JS engine, and adds a standard library of APIs for things like working with the local filesystem, network sockets, and more. Node also popularized the CommonJS module format and the NPM package management tool.

Today, JS is widely used in multiple environments for many kinds of projects. The primary use cases are interactivity in web pages, full web app clients, web app servers, and build tools.

So, some JS code is meant to run exclusively in a browser, some is meant to run exclusively under Node, and some code can work in both environments.

JS Development Constraints

The other key factor to understand in how JS has evolved and how it's used is the set of unique constraints web developers face. There's a great quote from one of the early browser developers that encapsulates this:

The by-design purpose of JavaScript was to make the monkey dance when you moused over it. Scripts were often a single line. We considered ten line scripts to be pretty normal, hundred line scripts to be huge, and thousand line scripts were unheard of. The language was absolutely not designed for programming in the large, and our implementation decisions, performance targets, and so on, were based on that assumption.

Eric Lippert, former IE/JS developer at Microsoft
http://programmers.stackexchange.com/a/221658/214387

Compared to many other languages, JS is very different:

No built-in module definition system (until ES6)
No built-in private / public encapsulation
Prototypal-based inheritance system unlike most languages
No static type declarations or compilation
Dynamically modified objects and data
Minimal standard library
Variations in browser capabilities
Entire codebase has to be shipped to the browser every time a page is loaded, then parsed and executed

Because of this, web client developers need to:

Minimize bytes sent over the wire
Handle browser compatibility issues
Fill in gaps in the JS standard library and language spec
Reuse and share code between apps
Build increasingly complex full-blown applications that just happen to live inside a browser

Core Language

Basic Syntax

JS uses much of the typical syntax seen in the C family of languages (C, C++, Java, and C#). Statements end with semicolons, variable names are case-sensitive, blocks are denoted with curly braces, comments can be written with // for a single line and /* */ for multiple lines, conditional logic uses if/else, and there are variations of for and while for looping.

Semicolons are actually optional - the interpreter will automatically insert them in places where they're strictly needed. That said, there's an ongoing argument between people who prefer using semis and those who avoid them. (Personally, I'm in favor of always using semicolons.)

The console.log() statement is the standard method for printing to the screen or the browser's debugging console. It accepts multiple arguments, including strings, objects, arrays, and other values, and will attempt to pretty-format any complex value. Other console methods exist, such as console.error() for error-formatted messages, as well as specialty formatting methods like console.table() and console.group(). (slides: comments and logging)

The const and let keywords are used to declare variables, and are block-scoped - they scope the lifetime of variable to the nearest curly brace block. let variables can be assigned a new value, while const variables cannot be changed to point to something else. There's also an older var keyword, which has more confusing scoping behavior - it's function-scoped, which means that no matter where you use var to declare a variable, it's hoisted and acts as if it was declared at the first line of the function it's in. Modern JS usage avoids var, since const and let behave more consistently. (slides: basic variable declarations)

As a personal recommendation, I suggest using const as the default, and let if you plan to reassign to that variable later, but there's others who suggest just using let all the time.

JS only has a single number type: a 64-bit floating point number equivalent to a double in other languages. It can hold integer values as well. (slides: numbers and math methods)

Strings can be written using three different quotes: single quotes and double quotes are equivalent, while the newer template literal strings use backticks and can have variable values interpolated in the middle. (slides: string syntax and methods)

Booleans are written as true and false. Objects and arrays are normally written using object/array literal syntax, as {} for objects and [] for arrays. (slides: basic object syntax, basic array syntax)

Object keys are always strings, but if the key name is a valid JS variable name, the quotes can be omitted (and usually are).

const text1 = 'Test A';
let text2 = 'Test B';
const value = 42;
// This is a template literal string
const text3 = `The number is ${value}`;
/* 
  This log statement can take multiple arguments
*/
console.log('First variable: ', text1, 'second variable: ', text2);

const object1 = {
  field1: 123,
  field2: true,
};

const array1 = [text1, value, object1, 99];

Data Types

JS has a variety of core data types. The basic data types are standalone values called primitives:

undefined : a variable that has not been assigned a value
null: a value that represents "no value"
String: text
Number : a 64-bit floating point number (aka IEEE double)
Boolean : a true / false value
Symbol: a unique reference value that can be used as a form of singleton identifier

All other values in JS are objects. This includes actual plain objects, as well as other types that extend from the Object type:

Object: a collection of string key / any value properties
- Function : a function that can be called
- Array : an expandable list of values
- Date : a date/time value
- RegExp : a regular expression

The difference between null and undefined is subtle. undefined represents "there is no meaningful value here", while null means "there is a value, but it's empty / not available".

Most of the other core data types have both a constructor function form and a literal syntax form, like new String('abcd') vs 'abcd', or new Array() vs []. Always prefer using the literal syntax to create new values instead of creating them via constructors.

Functions, arrays, dates, and regexps are technically objects as well.

Conditional Logic

JS uses the standard C-like if/else if/else syntax for conditions, as well as the ternary operator (const value = condition ? trueValue : falseValue). There's also a typical switch/case/break statement. Comparisons include the usual < and > operators and &&, ||, and ! boolean logic operators. (slides: conditional statements)

There are several forms of loops. You can do C-style counting loops with for(let i = 0; i < someValue; i++). The for (let key in obj) form iterates over keys, while for (let item of items) form will loop over values inside an iterable like an array. (slides: loops)

try/catch/finally allows handling errors. There's an Error class, but technically any value can be thrown.

Comparisons and Boolean Conversions

Unlike strictly-typed languages, JS frequently does implicit conversions or coercions of values in various contexts. The most common example is treating values as truthy or falsy. In addition to the actual true and false boolean values, using other values in a comparison statement will implicitly convert them to their boolean equivalent. (slides: "truthy" and "falsy")

For reference:

"Truthy" values: all objects and arrays (even if empty); non-zero numbers; non-empty strings; dates; functions
"Falsy" values: null, undefined, 0, NaN, and empty strings

(Note that the "empty object/array" behavior differs from Python, where those are considered "falsy".)

So, a comparison with an empty object like if ({}) { console.log('Truthy!')} will actually convert to true and print something, while a comparison with null like if (null) { console.log('Falsy!')} will convert to false and not print anything.

The boolean/comparison operators will all implicitly convert values to their boolean equivalent as well. A doubled negation operator is often used to convert a value to its boolean equivalent. For example, an empty string like '' is falsy, so !!'' would implicitly convert '' to false, !false to true, and !true to a final result of false.

JS also has two different comparison operators: == and ===. The double-equals == operator does loose comparisons that include implicit conversions of values. So, 0 == '' is true, because they will both end up being implicitly converted to false, and false is equal to false. The triple-equals === operator does strict reference comparisons that check to see if the two values are literally identical. For objects and arrays, it compares to see if they are the same reference in memory. So, {} === {} is false, because each of those is a separate new object reference.

You should almost always prefer using === strict reference comparisons to avoid unexpected implicit conversions. (slides: comparisons and coercion)

Functions

JS has two different ways to define standalone functions: the function keyword, and the () => {} "arrow function" syntax. They have some differences in behavior. (slides: function syntax and behavior)

function declarations are hoisted. Since arrow functions are assigned as variables, the declaration behavior is based on the keyword you use ( const, let, or var)
function declarations create their own value for the this keyword used inside. Arrow functions inherit the value of the this keyword as it existed at the time and scope they were declared in.
arrow functions can have optional parentheses if declared with only one parameter, and omitting curly braces adds an implicit return statement: const timesTwo = num => num * 2

JS functions are very flexible with arguments and parameters. No matter how many parameters were declared, you can always call a function with more or fewer arguments. If I have function myFunc(a, b, c) { }, but call it as myFunc(1, 2), then inside of the function a === 1, b === 2, and c === undefined because we didn't provide a value when we called it. If I call it as myFunc(1, 2, 3, 4), then the 4 argument is mostly ignored since we didn't declare a fourth parameter name. However, all function arguments can be accessed as an array-like value using the arguments keyword.

Function parameters can have default values provided, which will be used if the incoming value is undefined. If we change the example to function myFunc(a, b, c=42){ }, and call it as myFunc(1, 2), then a === 1, b === 2, and c === 42. This is frequently used for initialization. Function arguments can also be destructured - extracting specific fields from object and array parameters. (slides: function declarations and arguments)

Since functions are just variables, they can be passed around like any other variable.

JS in Depth

Objects

Object fields can be accessed with dot notation (obj.a.b.c) or bracket notation (obj["a"]["b"["c"]). Always use dot notation as long as the field names are known ahead of time. Use bracket notation if using a variable to look up a field ( obj[key]), or if the field name is not a valid JS identifer ( obj["some-field"]).

Reading a field that doesn't exist returns undefined, rather than throwing an error.

Objects can be modified at any time. New fields can be added, existing fields can be reassigned, and fields can be deleted with delete obj.field.

ES6 introduced a shorthand syntax for declaring objects when a key:value pair should be added based on the name and value of an existing variable - you can omit the : value portion:

// Shorthand syntax for declaring key/value pairs:
let x = 0,
  y = 1;

let es5Obj = { x: x, y: y };
// Create a key 'x' whose value is the variable 'x'
let es6Obj = { x, y };

There's also a computed property syntax that allows dynamically constructing keys based on the values of variables and expressions:

let name = 'abc';

let es5Obj = {};
es5[name] = 123;

let es6Obj = {
  // Use the value of variable 'name' as the property key


};
// {abc : 123, abc2 : 456}

Functions can be declared inside an object using function or arrow functions as the values, or directly inline as const obj = { someMethod() {} }. (slides: Object literal syntax)

Objects can be destructured, which is a shorthand for creating local variables based on the names of object fields. Destructuring statements can provide default values if that field is undefined, or change the local variable name that's created (slides: object destructuring and spreads):

let obj1 = { a: 1, b: 2, c: 3, d: 4 };

// can create local variables the long way
let a = obj1.a;
let b = obj1.b;

// or destructure to shorten it:
let { a, b } = obj1;

// Local variables can be given different names:
let { a: differentA, c } = obj1;

// And have default values in case of undefined:
let { doesNotExist = 123 } = obj1;

New objects can be created by using the object spread operator to copy values onto a new object:

let obj2 = { a: 1, b: 2, e: 5 };

let obj3 = { ...obj2, a: 99 };

There are static methods on the Object built-in type that allow getting all of the keys or values as arrays (Object.keys(someObj)), or an array of key/value pairs as 2-element arrays (Object.entries(somObj)). Object.assign(target, src1, src1) can be used to mutate existing objects by copying properties, or create new objects by passing an empty object as the target. (slides: Object static methods)

Arrays

Arrays are 0-indexed, but can be sparse - any index can be assigned at any time (arr[97] = "stuff"). Arrays may hold any value and a mixture of different types of values. Like with objects, accessing a non-existing array index returns undefined.

Like with objects, you can use destructuring to read values from arrays (const [a, , c] = ["a", "b", "c"]), and spread operators to create arrays (const arr3 = [...arr1, "b", "c", ...arr2, "d"]). (slides: array syntax, array destructuring)

Arrays are actually objects as well, and have numerous methods built in for various purposes. Some of these methods mutate the existing array, others return new arrays or other values. (slides: array methods, array iteration methods, array searching)

Name	Description	Arguments	Returns	Mutates
`slice()`	Copies an array subset	Indices to copy	A new array	No
`concat()`	Creates array with more values	New values/arrays to add	A new array	No
`splice()`	Inserts/deletes values	Index to mutate/delete, values to insert	Items removed	Yes
`join()`	Stringifies array contents	String used to separate items	A new string	No
`push()`	Inserts new items at end of array	Items to insert	Array size after	Yes
`pop()`	Removes last item from array	None	Last array item	Yes
`unshift()`	Inserts new items at start of array	Items to insert	Array size after	Yes
`shift()`	Removes first item from array	None	First array item	Yes
`sort()`	Sorts existing array	Callback to compare two items	Existing array reference	Yes
`reverse()`	Reverses order of items in array	None	Existing array reference	Yes
`map()`	New array by transforming values	Callback to transform one item	New array with callback return values	No
`filter()`	New array with values matching comparison	Callback to compare one item	New array with original items where callback returned true	No
`forEach()`	General iteration for side effects	Callback to run logic for one item	Nothing	No
`reduce()`	Calculates one result value based on all items	Callback to accumulate running result	Final callback result value	No
`indexOf()`	Finds index of exact value match	Value to search for	index or -1	No
`includes()`	Checks if exact value is in array	Value to search fo	boolean	No
`find()`	Find item by comparison	Callback to compare one item	First match or undefined	No
`findIndex()`	Find index of item by comparison	Callback to compare one item	index or -1	No
`some()`	Checks if any items match comparison	Callback to compare one item	boolean	No
`every()`	Checks if all items match comparison	Callback to compare one item	boolean	No

To emphasize, array.sort() and array.reverse() mutate the existing array!. This can be very surprising if you're not expecting it. In many cases it's a good idea to make a copy of an array using [...arr] or arr.slice() first and then sort/reverse the copy, especially if you're using something like React or Redux.

The core iteration methods each have a different semantic meaning and purpose:

map(): creates a new array, with the same size as the original array, whose values are based on a transformation of the original values
- const doubledNumbers = [1, 2, 3].map(num => num * 2)
filter(): creates a new array, with some of the values from the original, based on values where the comparison returned true:
- const evenNumbers = [1, 2, 3].filter(num => num % 2 === 0)
forEach(): general iteration over the array, usually to produce side effects
- [1, 2, 3].forEach(num => console.log(num))
reduce(): calculates a single result value based on "accumulating" a running result at each step, plus a starting value
- const total = [1, 2, 3].reduce( (previousResult, currentValue) => previousResult + currentValue, 0)

Scoping and Closures

As mentioned above, the let, const, and var keywords differ in their scoping behavior. let and const are block-scoped - they only exist within the curly braces block where they're declared, and the same variable name can be reused and shadowed inside of nested blocks. var is function-scoped - no matter where it's used, the declaration is hoisted to act as if it was written on the first line of the function it's inside, and reusing the same name will override the earlier declaration. The function declaration keyword is also hoisted. (slides: variable scope and hoisting)

There is a top level "global object" that can always be referenced, and any variables declared in the top-level global scope using var become fields on that object. In a browser, the object is window, and under Node, it's global.

Code inside functions can reference variables outside functions, because those variables are in an outer scope. Functions can be declared inside of functions, and close over variables by referencing them. A nested function that references a variable in the parent scope is called a closure. This allows nested functions to continue to refer to and modify variables long after the outer function has finished running. This is frequently used when passing callback functions for use as event handlers.

Understanding closures is a critical part of mastering JavaScript behavior. (slides: functions and closures)

// Functions can create and return functions
function createPrintName() {
  let name = 'Mark';

  function printName() {
    console.log(name);
  }

  return printName;
}

// Functions can capture references to values.
// These are called "closures".
function makeCounter() {
  let timesCalled = 0;
  return function counter() {
    timesCalled++;
    console.log(`Times called: ${timesCalled}`);
  };
}

const actualCounter = makeCounter();
actualCounter(); // "Times called: 1"
actualCounter(); // "Times called: 2"

Functions and `this`

The behavior of the this keyword in JS is one of the most confusing and difficult topics to grasp, especially if you are coming from a language like C# or Java.

Unlike other languages, the value of this can point to different references depending on how a function is declared and what syntax was used to call it. Instead of always pointing to a specific instance of a class, what this points to is based on the execution context of a function.

This is a long and complicated topic, so I'll summarize it briefly here and suggest reading up on it further.

There are four ways to call a function in JS, which each have a different effect on the value of this inside of the function (slides: understanding this):

"Function invocation": someFunction(1, 2) (slides: function invocation)
- this points to undefined or the global object, based on use of strict mode
"Method invocation": someObject.someFunction(1, 2) (slides: method invocation)
- this points to someObject because of the dot operator
"Constructor invocation": new SomeFunction(1, 2) (slides: constructor invocation)
- this points to the new function instance
"Indirect invocation": someFunction.call(thisArg, 1, 2) or someFunction.apply(thisArg, [1, 2])
- this points to the first argument of .call() or .apply()

Functions can be bound to create a new function with pre-fixed values for some of their arguments, including forcing the value of this. (slides: indirect invocation and binding)

Arrow functions inherit the value of this at the time they were defined (slides: arrow functions and this)

Classes and Prototypes

JS inheritance is unlike other most major languages - it's based on a system of prototypes for objects, and is known as prototypal inheritance. Loosely put, inheritance is dynamic and determined at runtime by tracing field lookups up a chain of references for a given object, instead of being defined strictly at definition time like in Java or C#. (slides: prototypal inheritance)

// Define a function to act as a "class"
function Animal(name, isAwake) {
  this.name = name;
  this.isAwake = isAwake;
}

// Create methods by adding functions to the "prototype"
Animal.prototype.wakeUp = function () {
  this.isAwake = true;
};

Animal.prototype.sleep = function () {
  this.isAwake = false;
};

// Create an instance
let jinx = new Animal('Jinx', true);

// Call an instance method
jinx.sleep();

ES6 added an actual class keyword to the language, but it is effectively syntax sugar over the existing prototypal inheritance behavior (slides: classes):

class Animal {
  constructor(name, isAwake) {
    this.name = name;
    this.isAwake = isAwake;
  }

  wakeUp() {
    this.isAwake = true;
  }

  sleep() {
    this.isAwake = false;
  }
}

class Dog extends Animal {
  constructor(name, isAwake, isWaggingTail) {
    super(name, isAwake);
    this.isWaggingTail = isWaggingTail;
  }

  bark() {
    console.log('Woof! Woof!');
  }
}

let spot = new Dog('Spot', true, false);
spot.sleep();

Async

Timers and the Event Loop

JS execution is based on a single-threaded event loop + queue. Conceptually, the behavior is while(queue.waitForMessage()) queue.processNextMessage().

Events are added to the queue, and have JS code attached. This includes mouse events, timers, network requests, and much more. The event loop pops the next event off the queue, and executes the entire attached code to completion. The currently executing script cannot be interrupted. The script may start timers, which will add events with scripts to the queue for later execution.

While there is only one JS execution thread, the browser itself may add events to the queue while code is executing. Rather than blocking for async requests, they will schedule events when they complete.

Note that if you have an infinite loop in your code, the event loop is "blocked" and cannot move on to process the next event!

JS logic relies heavily on callbacks - passing a function reference to be called and executed at some future point in time. Callback usage can be synchronous or asynchronous depending on the use case. (slides: callbacks and timers)

JS provides two core functions for scheduling async logic: setTimeout(callback, ms), which runs the callback function once, and setInterval(callback, ms), which runs the callback repeatedly. Both functions return a unique ID value that can be used to cancel the queued timer via clearTimeout() or clearInterval().

Understanding the JS event loop is critical to understanding how to use async logic in JS! See the video What the heck is the event loop anyway? for an excellent explanation, as well as this accompanying interactive event loop visualization tool.

Promises

Nested async calls often result in a "callback pyramid of doom":

fetchData('/endpoint1', (result1) => {
  fetchData('/endpoint2', (result2) => {
    fetchData('/endpoint3', (result3) => {
      // do something with results 1 through 3
    });
  });
});

This also makes error handling very difficult.

The JS Promise data type provides a structured way to handle future async results (slides: promise basics. Promises are objects can be in one of three states:

Pending: created, but there is no result yet
Fulfilled: completed, with a positive/successful result
Rejected: completed, with an error result

A Promise has settled or resolved once it is either fulfilled or rejected.

Promises can be chained using somePromise.then(callback) and somePromise.catch(callback). When somePromise resolves, any provided chained callbacks will be run - .then() callbacks if it fulfilled, or .catch() callbacks if it rejected. Callbacks can even be chained after the promise has resolved, in which case they will be executed almost immediately.

Promise chains effectively form a pipeline. Each .then() or .catch() returns a new promise. (slides: creating and chaining promises, combining and resolving promises)

let promise1 = new Promise((resolve, reject) => {
  // This callback executes synchronously, immediately
  // Could "resolve" the promise with a value:
  resolve('a');
});

promise1
  .then((firstValue) => {
    // "a"
    return 'b';
  })
  .then((secondValue) => {
    // "b"
    // _Not_ returning a value returns `undefined`
  })
  .then((thirdValue) => {
    // undefined
  });

Inside a promise callback, you can run whatever calculations you want, but you can only do 3 things to complete the logic:

Return a value: resolves the promise successfully, with that value. Note that returning nothing or undefined is the same as resolving the promise successfully with undefined
Return another promise. The new promise for the callback will resolve or reject based on the promise you returned.
Throw an error. This rejects the promise, with that error.

`async/await` Syntax for Promises

Chaining Promises can also be difficult. The newer async/await syntax lets you write Promise-handling logic with what appears to be synchronous-style syntax, including use of try/catch for handling errors.

A function must be declared using the async keyword in order to use the await keyword inside. Every async function then automatically returns a Promise with whatever value is returned. Rejected promises in an async try/catch will jump to the catch block. (slides: async/await)

// This function:
function function1() {
  let resultPromise = Promise.resolve(42);
  return resultPromise;
}

// Is the same as this function:
async function function2() {
  return 42; // converted to a promise!
}

Overall, async/await syntax is much nicer to read than promise chains, and should usually be preferred:

// This promise chain:
function fetchStuff() {
  return fetchData('/endpoint1')
    .then((result) => {
      return firstProcessStep(result);
    })
    .then((processedResult) => {
      console.log(`Processed result: ${processedResult}`);
      return processedResult;
    })
    .catch((err) => {
      console.error('PANIC!', err);
    });
}

// Can convert to:
async function alsoFetchStuff() {
  try {
    let result = await fetchData('/endpoint1');
    let processedResult = firstProcessStep(result);
    console.log(`Processed result: ${processedResult}`);
    return processedResult;
  } catch (err) {
    console.error('PANIC!', err);
  }
}

DOM

HTML is a text document format, organized into a hierarchy of nested tags like <h1>, <p>, and <img>, that represents the desired structure and content of a web page.

Browsers download an HTML file, and parse the HTML tags. They then create a set of internal data structures that match the requested page content, and use that to lay out and draw the actual pixels on screen.

Browsers also expose a live representation of the current page content to JS. This is known as the Document Object Model (DOM). While every browser has its own different internal data structures for a page, the DOM is a standardized set of APIs for interacting with the content of a page. The DOM APIs enable JS code to create, read, modify, and delete the actual contents of a page, dynamically, after the initial HTML has been parsed and the page was loaded.

The DOM spec defines classes that correspond to each HTML element type. For example, a <div> tag, when parsed, will result in an HTMLDivElement class instance being created and added to the DOM. Each instance is called a DOM node, because the DOM contents form a tree.

Different DOM classes support different methods - an HTMLInputElement instance has different methods available than an HTMLDivElement instance, but all DOM node instances share a common core set of base methods.

DOM Query APIs

The DOM API allows querying for nodes based on selectors, such as the type of tag, ID attribute, classnames, node relational structure, and more. (slides: DOM query and manipulation)

The root element for a page is globally accessible via the global document object. Most queries are run against document to find nodes anywhere inside a page.

The primary query APIs are:

getElementById(): finds a DOM node based on an ID attribute string. (Note that the ID string should be provided without a '#' prefix here, unlike other places where an actual selector is being used to query)
querySelectorAll(): finds all DOM nodes that match the provided CSS selector query
querySelector(): finds the first DOM node that matches the provided CSS selector query

These same query APIs also exist on the DOM node subclasses, so you can run someNode.querySelectorAll() to scope a search to that particular portion of the page's DOM tree.

Note that querySelectorAll() returns a NodeList object. This is an array-like object. It implements a couple array-type methods like .forEach(), but is not an actual array. It's common to convert a NodeList result into an actual array before doing further work with the query results.

Other query methods like getElementsByName also return a NodeList, but those results are "live" views whose contents automatically change as the DOM is updated. This behavior can be confusing if you're not ready for it.

DOM Manipulation

DOM nodes support a wide variety of manipulation operations.

New DOM nodes can be created with document.createElement('p'), which returns an instance of the requested node type, or copied with .cloneNode().

DOM nodes have .innerHTML and .innerText properties, which allow reading and writing the entire contents of a node at once. Input nodes typically have properties like .name and .value.

Nodes have a .classlist property, which is an object that supports adding, removing, and toggling the existence of individual CSS classnames on that node (divNode.classlist.toggle('active')).

Nodes have a .style property that allows reading and mutating any of the style-related properties, including values that were defined using CSS (divNode.style.backgroundColor = 'red'). Unlike CSS, the style property names are camelCased instead of kebab-cased, because that matches JS naming conventions.

Nodes have properties that point to related nodes in the tree: .children, .firstChild/lastChild, .nextSibling/previousSibling, and .parentNode.

Nodes can be inserted and moved around with with .appendChild(), .insertBefore()/insertAfter().

Event Listeners

DOM nodes implement an event listener interface, which looks like node.addEventListener(eventName, callback). There's a wide variety of available events that will be triggered by the browser as the user interacts, such as 'click', 'mousemove', and 'keydown'. (slides: DOM event handling)

Event handlers receive an event object with numerous fields describing the current event. Inside of an event handler callback, this will point to the actual DOM node as long as you declared the callback with function (arrow functions always use the same this value from the scope where they were defined).

When a DOM event is triggered, it bubbles upwards from the original target node, through all of its parent nodes, to the root document, firing any matching event handlers along the way. After the event has reached the root, it then runs back down from the root to the original node, known as the capturing phase.

Because events bubble up, it's possible to add a single event listener on some parent node that catches all events of a certain type on any children. This pattern is known as event delegation.

Any event handler may cancel the rest of the handler processing by calling e.stopPropagation(). The browser also often has default behavior that will be run regardless of any user-provided listeners, which can be canceled with with e.preventDefault().

jQuery

For many years, the DOM APIs were poorly defined, had widely varying support between browsers, and had limited query capabilities built in. The community created many libraries to fill in these gaps and provide common capabilities across browsers. jQuery became the most popular DOM manipulation library by far. Many of its capabilities have been added to the DOM APIs themselves, and the need for jQuery has lessened as the ecosystem has evolved, but it's still one of the most widely used JS libraries. (slides: jQuery examples)

jQuery is an abstraction layer for querying nodes from the DOM and quickly manipulating them via chained operator functions. It also has utilities for interacting with inputs, creating DOM nodes from HTML strings, provides a nicer abstraction over the browser's built-in APIs for making AJAX requests, adds some animation capabilities, and has a plugin system that allows developers to extend its capabilities.

jQuery is not a framework for developing entire applications - it's best suited for adding interactivity to an existing static or server-rendered web page. jQuery is not normally needed in today's modern client-rendered web app frameworks like React or Vue, because those frameworks take care of creating and updating the DOM for you, and using jQuery would interfere with their operations.

Once loaded, the jQuery API is globally accessible as a variable named $. DOM queries are run by passing a selector, like $("#demo .content"), which returns an array-like object with numerous methods that can be chained. It's a common practice to prefix jQuery result object variable names with $, like $demo.css("backgroundColor", "green").slideUp(500).

Further Resources

JS Overviews and Tutorials
- MDN JavaScript docs: extensive tutorials and reference material on all aspects of JS
- FreeCodeCamp tutorials: beginner-focused project-based tutorials
- The Modern JS Tutorial: topic-based intros to specific aspects of JS and DOM syntax
- 2ality: JS books and articles: in-depth articles and books that cover JS syntax and behavior in fine detail, including articles that summarize the new features in each ES spec revision
- JS for Java Developers slides: a cheatsheet-style summary of JS syntax, concepts, and ecosystem tools
- Tania Rascia's JS tutorials: a broad set of tutorials that cover JS topics such as syntax, events, classes, functions, async, and more
- JS 25th Anniversary milestones timeline
The this keyword
Array Methods
JS Event Loop and Async
Other Topics
DOM
- The Modern JS Tutorial: DOM
- Understanding the DOM (free ebook)

How Web Apps Work: Browsers, HTML, and CSS

Mon, 09 Nov 2020 11:00:00 -0500

Some of the descriptions will be more oriented towards modern client-side app development with JavaScript, but most of the topics are fundamental enough that they apply to server-centric applications as well.

Other posts in this series cover additional topics, such as:

There's a comprehensive set of HTML and CSS tutorials at Interneting is Hard: HTML and CSS, with beautiful diagrams and great explanations of these terms and concepts. (I've borrowed a few of their excellent diagrams for this post.)

CSS Tricks has some amazing guides to specific aspects of CSS, like their guides for Flexbox and CSS Grid, as well as numerous articles on other CSS and web dev topics.

My JavaScript for Java Developers slides also cover much of this post's content as well.

Browsers

Any application can make requests using the HTTP protocol. However, web browsers are applications whose primary purpose is to request data using HTTP, format the content, and display it on screen. That content is normally defined using the HTML format, and the appearance of the content is defined using CSS.

A browser's implementation can be divided into several areas:

The networking logic that is used to request data
The rendering engine that parses fetched content, determines how it should be laid out on screen, processes the styling rules, and actually draws the pixels on screen
A JavaScript engine that parses and executes JS code
The browser chrome that defines the visible UI around the page content (buttons, menus, address bar, etc)

Different browser implementations may share and reuse implementation pieces, but use their own unique implementations for other sections. For example, several browsers are built on top of the "Blink" rendering engine, like Chrome and Edge, but have very different UI implementations and user-visible features.

Browser Rendering Behavior Variations

Browsers all attempt to implement the same language specifications, but each browser ends up implementing those specifications differently. Most of the time the behavior is the same, but browsers frequently have differences in behavior for the exact same features. This has become less of a problem in recent years, as browser support for major specifications has become more consistent, but browser "quirks" in layout, appearance, and JS support are still a major pain point for web developers.

Browsers can also always try adding new features themselves, but in order for features to be considered "standardized", they typically need to be implemented by at least two different browsers and go through the appropriate web specification committees.

Browser Versions

Each time a browser ships a new version, that version has a fixed understanding of a certain subset of the HTML, CSS, and JS specifications. Since that browser version may stay in wide use for many years, developers need to determine what set of browsers and versions they plan to support when developing a site or application.

Browsers originally had infrequent releases of new versions, and each new version had many significant changes. Today, most modern browsers release new versions frequently (every 6-8 weeks), and each new version only has a few meaningful changes. These browsers are referred to as evergreen browsers, because there's always a new version available and most users will automatically be upgraded to the latest version.

The "Can I Use?" site is an incredibly useful resource that has a database of different HTML, CSS, and JS features, and lists which browser versions support each feature as well as how much worldwide market share those browsers have. This is a valuable tool for deciding whether you can safely use a given feature in your site, based on your target audience. For example, the CSS position: sticky layout feature currently has 95% support worldwide, but does not work in Internet Explorer. So, if your target audience still includes IE11 users, you might not want to use that feature.

Major Browsers

The majority of browser market share belongs to a few major browsers. By far the biggest one is Google Chrome.

The market share of each browser has gone up and down over time. Internet Explorer had a near-monopoly in the early 2000s until Firefox became popular. Chrome took over from both of them, and Safari has become a major factor on mobile. Browser market share also varies significantly in different countries.

The browser market share as of October 2020 looks roughly like this. Note that there are rather different usage percentages for desktop devices vs mobile devices:

Name	Company	Version	Evergreen	Rendering Engine	Desktop %	Mobile %
Chrome	Google	86	Yes	Blink	70%	63%
Safari	Apple	14	No	WebKit	8%	24%
Firefox	Mozilla	82	Yes	Gecko	7%	0.50%
Edge	Microsoft	86	Yes	Blink	6%	0%
Opera	Opera	72	Yes	Blink	2%	2%
Samsung Internet	Samsung	12	No	Blink	0%	6%
Edge (legacy)	Microsoft	44	No	EdgeHTML	1%	0%
Internet Explorer	Microsoft	11	No	Trident	2%	0%

Browser Notes

Some notes on the major browsers:

Chrome:
- Built on an open-source core called "Chromium", which is effectively Chrome minus all the actual Google account integration and branding
- The "Chrome" browser partly got its name because it tried to minimize the amount of "browser chrome" (the term for all the browser UI that surrounds the actual page content)
- Google has a tendency to announce new web features that only exist in Chrome and yet consider them "final" even if other browsers haven't added them yet
Safari:
- The only actual browser rendering engine allowed by Apple on iOS devices. Any other browsers you see on iOS are actually the Safari core with a different UI skin and branding.
- Apple tends to be slow adding new features to Safari even after they've been added to other browsers. Apple has also refused to add certain features on the grounds that they might make act as privacy leaks in some way.
Firefox:
- Open-source, and built by the Mozilla organization - it's the only major browser not built by one of the big tech companies
- Now that many browsers build on top of Chromium/Blink or Webkit, Firefox's "Gecko" engine is the main alternative rendering engine still in meaningful use.
Edge:
- There's been two different variations of the MS Edge browser. The first was actually built entirely by Microsoft, using a cleaned-up and modernized version of the rendering engine previously used in Internet Explorer. Recently, Microsoft stopped developing their own rendering engine, labeled that version of Edge as "legacy", and rebuilt Edge on top of Chromium with their own UI.
Internet Explorer:
- IE was the default browser on Windows up until recently. Many companies still have internal websites that only work correctly in IE, which keeps it as a small but persistent portion of the market
- IE has "stagnated" twice. IE6 came out in 2001 along with Windows XP, but Microsoft stopped developing it for several years until Firefox and Chrome began taking away market share. Today, IE11 is considered an obsolete browser without many modern capabilities, and many companies are finally dropping support for running their sites in IE11.

HTML

Browsers request data from servers using the HTTP protocol. The standard format that browsers read and use to display data is HyperText Markup Language (HTML). HTML is a plain-text format that uses a hierarchy of nested HTML tags to describe the content, layout, and formatting of a web page.

HTML Tags

An HTML tag is a unique name that is written surrounded by angle brackets, like <html>, <img>, or <p>. Most tags also need to have a corresponding closing tag element, and may contain content inside. For example, a paragraph tag may have text and images inside, and a list tag will have list item tags.

Some tags are also self-closing, such as <img />, <hr />, and <input />.

Tags are not case-sensitive, but are usually written lower-case: <img>, not <IMG>.

Tags may also have attributes, written as attributeName="someValue". Some attributes don't need an actual value - the existence of the attribute name is all that's needed, such as <input disabled>.

The HTML spec defines dozens of different tags, each with a different purpose. Browsers determine what should actually be displayed by parsing these tags.

HTML Page Structure

A standard HTML page structure consists of:

an outer <html> tag that contains:
- an optional <head> tag that may have metadata-related tags inside, as well as links to additional asset files that the browser should download
- a <body> tag containing the actual content of the page

A basic example would look like:

<html>
  <head>
    <title>My Website</title>
    <link rel="stylesheet" type="text/css" href="styles.css">
    <script type="text/javascript" src="logic.js"></script>
  </head>
  <body>
    <h1>Welcome to My Website</h1>
    <p>
      Here's some <b>bold</b> text, and an image:
    </p>
    <img src="party-parrot.gif" />
  </body>
</html>

HTML tags are normally written with indented spacing for each level of nesting, but that's only needed for readability by people - browsers don't care about that.

HTML pages may also start with a <doctype> tag that declares the HTML version being used by the file, although that isn't necessary.

Common HTML Tags

Core tags
- <html>: The outer container of an HTML page
- <head>: the header section with metadata
- <body>: the main content of an HTML page
- <h1> through <h6>: headers of varying sizes
- <p>: text paragraphs
- <div>: a generic block
- <span>: a generic inline piece of content
- <img src="some-image.png">: links to an image to be displayed
- <a href="http://www.example.com">: an anchor tag that may point to another page, or a specific location within this page. Clicking an anchor tells the browser to navigate to that URL.
Headers:
- <title>: Page title, shown in the current tab / browser window
- <meta>: assorted metadata attributes describing a page
- <link>: link to assets such as stylesheets
- <script>: link to JavaScript files, or containing inline JS code
Formatting
- <b>, <i>, <u>: bold, italic, underlined text
- <blockquote>: represents a formatted quotation
Lists
- <ul>, <ol>: list parents. Unordered lists show bullet points for each list item, ordered lists show numbers.
- <li>: list items inside of lists
Tables
- <table>: defines a table with rows columns
- <tr>: a single row in the table
- <thead>: the header area for the table
- <th>: header cells in a row in the header
- <tbody>: the body of the table
- <td>: a normal cell in a row in the body
Forms
- <form>: a form area with inputs
- <input>: a user-editable input. Browsers have many types of inputs built in, most of which are used by passing a different type attribute to <input>. Common inputs are:
  - "text": a one-row textbox
  - "checkbox": a clickable true/false checkbox
  - "radio": a clickable button where only one value of a group can be selected at a time
  - "file": allows the user to select and upload a file from the local computer
- <select>: a dropdown for selecting a single value from a list
- <option>: a single selectable value in a dropdown
- <textarea> a multi-line textbox
- <label>: a descriptive label associated with an input

Some tags do basically the same thing as other tags, but are considered to be more "semantic". Semantic markup is the idea that HTML tags should be meaningfully readable to both other developers and the browser itself. For example, a <div> is a generic block of content. The HTML spec now has tags like <header>, <footer>, <main>, and <section>, which behave the same way display-wise, but are intended to be more meaningful in terms of what content they will contain.

Common Attributes

There are several common attributes that can be applied to any tag:

id="someValue": gives a globally unique "ID" name to this element. There can be many IDs in a page, but each unique ID should only exist once in a page at a time
class="first second third": a space-separated list of "class names", used to help identify elements for styling and searches
title="Some description": descriptive text, usually shown in a tooltip when hovering over the element
style="color: red; background-color: blue;": a list of inline style rules to change the display of this element. Uses semi-colon-separated CSS rules
alt="Some description": descriptive text, typically supplied for images for use when the image can't be displayed, or read by a screen reader

Other attributes only work for certain tags:

<img src="some-image.jpg">: the URL for the image to show
<a href="http://www.example.com">: the URL for the link
<input type="checkbox">: the type of input element to show

Forms

HTML forms allow users to enter values into the browser, such as typing in text or selecting items from a dropdown. The browser maintains the current value of each input internally as the user interacts with the inputs.

Input elements can be put anywhere, but should normally be put inside a <form> tag, which adds some extra behavior. In particular, forms can be submitted - the browser will automatically gather up the values of each input and send them via HTTP request to the server (normally an HTTP POST request).

A basic form might look like:

<form action="/create-user" method="post"> 
    <ul>
        <li>
            <label for="name">Name:</label>
            <input type="text" id="name" name="user_name" />
        </li>
        <li>
            <label for="state">State</label>
            <select name="state">
              <option value="ca">California</option>
              <option value="oh">Ohio</option>
              <option value="ny">New York</option>
            </select>
        </li>
        <li>
            <label for="msg">Message:</label>
            <textarea id="msg" name="user_message"></textarea>
        </li>
    </ul>
    <button type="submit">Create User</button>
</form>

Side note: a <button> in a form defaults to "submit this form" when clicked. If you want to do something else like handling logic in JavaScript (which is common in UI frameworks like React), you need to declare it as a <button type="button"> instead.

CSS

HTML describes a page's overall structure and layout, and browsers have a default basic appearance for each tag. However, every page needs additional styling information added: colors, fonts, layout, and more. This is done with Cascading Style Sheets (CSS), a declarative language that allows adding styling rules to ased on queries that match some set of HTML elements.

CSS declarations can be added to an HTML page by:

Linking a separate CSS file in the <head> of the document
Adding a <style> tag to the <head> with CSS declarations inside

Alternately, styling rules can be written directly on an element in the HTML with the style attribute, referred to as inline styles.

CSS Syntax

CSS syntax consists of two major pieces:

selectors, which are queries that define what elements will be matched
rules, which are the individual styling values that will be assigned to all matching elements

The basic syntax consists of a selector followed by a curly-brace block. The block may contain one or more rules, where each rule looks like property-name: value;

Multiple selectors may be declared in front of a rule block, and again any elements that match any of those selectors will be affected by the style rules.

A basic CSS example might look like:

div.active {
  font-weight: bold;
  font-size: 24px;
  color: red;
  border: 1px solid blue;
}

CSS only allows comments using the same multi-line comment syntax as many other languages: /* comment here */

CSS Selectors

CSS selectors can contain many different pieces that uniquely identify some set of elements. Some of the most common matching pieces are:

Specific HTML tag types: by name of the tag, without angle brackets ( h1 )
Tags with a given classname: by a . in front of the classname ( .myClassname )
Tags with a given ID: by a # in front of the ID value ( #myId )
Nesting: adding spaces between selector query pieces ( main form.signup input#firstName )
Immediate children: putting a > between parent and child ( ul.items > li )
The "universal selector" uses a * to match everything

There are also pseudo-class selectors, which apply an additional modifier to the selector matching, and are usually based on some aspect of the element's behavior. For example, the :hover pseudo-class only matches an element where the mouse is currently hovering, and the :disabled pseudo-class only matches an element that is currently marked as disabled. The :nth-child pseudo-class is commonly used for striping the background color of alternating table rows based on even/odd row numbers, and there's dozens of others as well.

An element's final style is made up of all of the combined rules that apply to it. Children inherit most properties from their parents. A rule html {font-family: Courier New;} would set that font as a default for the whole document, but a nested child could have its own font-family value that would override the value it inherited.

There are rules for selector specificity, which define which values take effect when there are multiple competing rules+properties that apply to a given element. More specific rules will override less specific rules. Inline styles have much higher precedence than styles defined separately. Adding a !important at the end of a rule raises its specificity by a large amount, like color: red !important;.

CSS Styling Rules

There are hundreds of CSS properties that can be modified. They range from very simple (changing the text color) to highly complex (creating multi-point color gradients, defining polygon clip masks). Multi-word CSS property names are hyphenated (background-color, font-weight).

Many CSS properties can be grouped by concept, such as:

Element backgrounds: background-color, background-image, background-repeat
Borders: border-left, border-color, border-width
Fonts: font-family, font-weight, font-size
Spacing: margin-top, padding-bottom
Sizing: height, width
Layout: display, position, overflow, z-index

Different properties can accept different kinds of values, and values can often be given in different units. For example, font-size accepts values using "length units", which can be in pixels (px), size relative to parent ( em), size relative to the <html> element ( rem ), actual display sizes ( in, mm, cm ), and more. Layout widths and heights are often given in percentages ( % ) or pixels.

CSS examples:

div { /* will apply to all div tags */
    color: red; /* text color */
}

div span { /* only spans nested in divs */
    color : blue; /* overrides red color */
    font-weight: bold;
}

#some-id { /* only applies to tag with this ID */
    font-size: 24px;
}

.someClassname { /* any tag with this classname */
    border: 1px solid green;
}

p.first.second { /* only paragraphs with both classnames */
    margin: 5px;
}

div.classA , span.classB { /* applies to either */
    min-width: 50%;
    max-height: 300px;
}

div > p { /* only paras as direct children of divs */
    display : none; /* hide entirely */
}

div.myClass:hover { /* only applies on mouse hover */
    border: 1px dotted orange;
}

Page Layout

There are many factors that affect how elements are positioned in a page, and what their size and shape is compared to other elements. HTML has default layout before defined for each element type, but CSS properties can completely change an element's position and layout behavior.

The Box Model

The default layout behavior is called the box model. Every HTML element is treated as a rectangular box, and falls into one of two categories:

block boxes stack downwards vertically
inline boxes stack sideways horizontally

For example, <div> tags are blocks. Adding <div /><div /><div /> will produce three elements in a column going down the page. On the other hand, <span> tags are inline. Adding <span /><span /><span /> will lay out all three spans side-by-side.

An element's default behavior can be changed using the various options for the display and position properties. For example, adding display: inline; to a <div> tag will make it behave the same as a <span> normally would.

The size of a given element is made up of several pieces:

The actual content
Padding: extra spacing that is still "inside" the element
Border : extra lines around / "outside" the element
Margin: extra spacing that is considered "outside" the element

The "padding and border outside" sizing behavior can sometimes feel unexpected, The behavior can be altered using box-sizing: border-box;, which alters the sizing calculations so that the "total width" of an element includes all of its border and padding as well.

Display and Positioning

The display and position properties can be used to alter how an element's layout behavior works.

The display: block; and display: inline; values force an element to one of the standard layout behaviors. There are also specialized layout behaviors, like Flexbox and CSS Grid (described below).

Elements can be removed from the page completely using display: none;.

The position property takes several values, some of which work in combination with the top/bottom/left/right numeric properties:

position: static: the default
position: relative: offset the location by some amount
position: fixed: stays in a single location on screen, even as the document moves around
position: absolute: stays in a single location within the document, without following normal layout flow
position: sticky: a newer option that keeps items in place for a while until the page scrolls well past them

Any value other than static creates a new stacking context. Loosely put, any children will be positioned with their top and left values relative to where this element is located.

The z-index property affects how elements "stack" on top of each other when they overlap. (Frankly, the z-index property confuses just about everyone, so I'm not even going to explain more than that here because I'll probably get it wrong, but please research it further.)

Flexbox

Flexbox is a CSS layout algorithm that allows aligning items into rows or columns, with rules for aligning items and wrapping them as they overflow.

A flex parent is defined by adding a display: flex rule to an element. All of the immediate children are then laid out automatically as flex items.

Key properties:

flex-direction: the primary layout direction for the flexbox parent (row, column)
justify-content: how items are aligned in that direction (start, end, separated)
align-items: how items are aligned in the cross direction

Centering items both vertically and horizontally had been an exceptionally difficult task on the web for many years, but Flexbox allows centering things fairly easily, such as display: flex; justify-content: center; align-items: center;.

CSS Grid

CSS Grid is a newer CSS layout algorithm designed for 2D layouts with various numbers of rows and columns.

A grid container is defined by adding a display: grid rule to an element. All of the immediate children are then laid out automatically as grid items.

Grid syntax is extensive, and allows defining sets of columns and rows with varying repeated patterns.

Responsive Screen Size UI

Users may view a site using a wide variety of devices with varying screen sizes: desktops such as 1920x1080, tablets such as 768x1024, or phones such as 375x812.

Designers and devs frequently need to create layouts that look good at many different screen sizes, referred to as responsive design. Rather than make completely different sites, they may add CSS rules that only apply in certain cases, such as when the viewing device screen is in a specific size range. This is normally done using media queries, which are CSS conditions that only apply the selectors and rules inside if the condition matches. The CSS rules typically choose certain screen sizes as breakpoints so that the layout changes if your device is larger or smaller than that size. An example media query and breakpoint setup might look like:

/* Small devices (portrait tablets and large phones, 600px and up) */
@media only screen and (min-width: 600px) {
  .notice {
    color: red;
  }
}

/* Medium devices (landscape tablets, 768px and up) */
@media only screen and (min-width: 768px) {
  .notice {
    color: green;
  }
}

/* Large devices (laptops/desktops, 992px and up) */
@media only screen and (min-width: 992px) {
  .notice {
    color: blue;
  }
}

Further Resources

How Web Apps Work: Client Development and Deployment

Fri, 06 Nov 2020 10:00:00 -0500

Other posts in this series cover additional topics, such as:

My JavaScript for Java Developers slides also cover much of this post's content as well.

JavaScript Development and Build Processes

It's important to understand the evolution of JavaScript's design and usage over time to understand why and how it gets used today.

This quote sums things up well:

The by-design purpose of JavaScript was to make the monkey dance when you moused over it. Scripts were often a single line. We considered ten line scripts to be pretty normal, hundred line scripts to be huge, and thousand line scripts were unheard of. The language was absolutely not designed for programming in the large, and our implementation decisions, performance targets, and so on, were based on that assumption.
- Eric Lippert, former IE/JS developer at Microsoft
http://programmers.stackexchange.com/a/221658/214387

Today, applications routinely consist of hundreds of thousands of lines of JavaScript. This introduces a completely different set of constraints on how developers write, build, and deliver client-side applications. Those constraints result in a very different development approach than the early days. Instead of writing a few lines of JS and inlining them into an HTML page, modern web clients use complex build toolchains that are directly equivalent to compilers for languages like C++ or Java.

JavaScript Module Formats

Almost all languages have built-in syntax for declaring encapsulated "modules" or "packages". For example, a Java file might declare it is part of package com.my.project, and then declare a dependency on another package with import some.other.project.SomeClass;. C#, Python, Go, and Swift all have their own package definition and import/export syntax.

JavaScript is not one of those languages.

Unlike all these other languages, JavaScript originally had no built-in module format syntax. For many years, JS code was written as inline <script> tags directly in HTML, or as small .js files with a few shared global variables.

As developers began writing larger application codebases, the community eventually began inventing their own module formats to help provide structure and encapsulation. Each of these formats was invented to solve differing use cases.

(Note: code snippets in this section are purely to illustrate the syntax differences between the various formats - the actual code is not intended to run or do anything useful.)

Legacy Module Formats

Adding numerous script tags to a page has several issues. It can be very hard to determine the dependencies between different script files and load them in the right order. Also, since all top-level variables occupy the same global namespace, it's really easy to accidentally have variables with the same name overriding each other:

<script src="jquery.min.js"></script>
<script src="jquery.someplugin.js"></script>
<script src="./components/dropdown.js"></script>
<script src="./components/modal.js"></script>
<script src="./application.js"></script>

// dropdown.js
var delay = 2000; // in ms

// modal.js
var delay = 4000; // in ms

// application.js
// Oops - is it 2000 or 4000?
console.log(delay)

Immediately Invoked Function Expressions (IIFEs) are a pattern that rely on JS variables being scoped to the nearest function. An IIFE involves defining a new function and then immediately calling it to get a result. This provides encapsulation, and was used as the basis of the "revealing module" pattern, where an IIFE returns an object that defines its public API (equivalent to a factory function or a class constructor):

// dropdown.js
(function(){
    var delay = 2000; // in ms
    APP.dropdown.delay = delay;
}());

// modal.js
const modalAPI = (function(){
    // No name clash - encapsulated in the IIFE
    var delay = 4000; // in ms
    APP.modal.delay = delay;

    $("#myModal").show();

    function hideModal() {
      $("#myModal").hide()
    }

    // return a "public API" for the modal by exposing methods
    return {
      hideModal : hideModal
    }
}());

The Asynchronous Module Definition format (AMD) was designed specifically to be used by browsers. A specialized AMD loader library first creates a global define function. AMD modules then call define() and pass in an array of module names they depend on, and a function that acts as the body of the module. The module body function receives all its requested dependencies as arguments, and may return any single value to act as its "export". The loader library then checks to see if all of the requested dependencies have been registered and loaded. If not, it recursively downloads other dependencies waterfall-style, and works its way back up the dependency chain to initialize each module function with its dependencies.

// moduleA.js
// Loader library adds a global `define()` function
define(["jquery", "myOtherModule"],
function($, myOtherModule) {
    // Body of the function is the module definition
    const a = 42;
    const b = 123;

    function someFunction() { }

    // Return value is the "exports" of the module
    // Can do "named exports" by returning object with many values
    return {a : a, publicName : b, someFunction : someFunction}
});

// moduleB.js
define(["backbone"],
function(Backbone) {
    const MyModel = Backbone.Model.extend({});

    // Can do a "default" export by just returning one thing
    // instead of an object with multiple things inside
    return MyModel;
});

IIFEs and AMD modules are no longer actively being used for new development, but code using those patterns is still out there.

CommonJS Modules

The CommonJS module format was developed specifically for use with the Node.js runtime (a JS interpreter running outside the browser). Since Node has access to the filesystem, the CommonJS format was designed to load modules from disk synchronously as soon as they are imported.

The Node.js interpreter defines a global require function that accepts either relative paths, absolute paths, or library names. Node then follows a complex lookup formula to find a file matching the requested path/name, and if found, immediately reads and loads the requested file.

CommonJS modules do not have any outer wrapping function. The interpreter also defines a global module.exports variable, and the module defines its exported values by assigning to that variable.

// moduleA.js
// Node runtime system adds `require()` function and infrastructure
const $ = require("jquery");
const myOtherModule = require("myOtherModule");

// The entire file is the module definition
const a = 42;
const b = 123;

function someFunction() { }

// Node runtime adds a `module.exports` keyword to define exports
// Can do "named exports" by assigning an object with many values
module.exports = {  
    a : a, 
    publicName : b, 
    someFunction : someFunction 
}


// moduleB.js
const Backbone = require("backbone");
const MyModel = Backbone.Model.extend({});

// Can do a "default" export by just assigning
// one value to `module.exports`
module.exports = MyModel;

CommonJS modules allow dynamically importing other modules at any time, and imports can be done conditionally.

The flip side is that CommonJS modules cannot be used as-is in a browser - some kind of adapter or repackaging is needed.

Universal Module Definition

Some libraries need to be able to be used in multiple environments with the same build artifact: a plain global <script> tag in a browser, an AMD module in a browser, or a CommonJS file under Node. The community invented a bizarre-looking hack that allowed a module to work correctly in all three environments by feature-detecting capabilities, which was dubbed the Universal Module Definition (UMD) format. Today this is still semi-commonly used as a build output target for some libraries.

// File log.js
(function (global, factory) {
    if (typeof define === "function" && define.amd) {
      define(["exports"], factory);
    } else if (typeof exports !== "undefined") {
      factory(exports);
    } else {
      var mod = {
        exports: {}
      };
      factory(mod.exports);
      global.log = mod.exports;
    }
})(this, function (exports) {
  "use strict";
  
  function log() {
    console.log("Example of UMD module system");
  }
  // expose log to other modules
  exports.log = log;
});

ES Modules

The ES2015 language spec finally added an official module syntax to the JS language, which is now referred to as ES Modules or "ESM". It provides syntax for defining both named and default imports and exports. However, due to the differences between browsers and Node.js, the specification did not define how modules would actually be loaded by an interpreter, or what the import strings would refer to, and instead left it up to the differing environments to figure out how to load modules appropriately. Modern browsers have all implemented loading ES Modules based on URLs as the import strings. Node has had significantly more trouble determining a path forward, due to its reliance on CommonJS modules as the default format. As of Node 15, Node has some support for loading ES Modules, but there are still difficulties determining how CommonJS and ES Module files should interop with each other.

ES Modules were designed to be statically analyzable. The downside is that you cannot do dynamic or conditional imports - all imports and exports must be at the top level of the file.

// moduleA.js
// ES6 language spec defines import/export keywords
import $ from "jquery";
// Can do "default" imports - no curly braces around the variable name
import myOtherModule from "myOtherModule";

// Define "named exports" by adding `export` in front of a variable
export const a = 42;
export const b = 123;
export {b as publicName};

export function someFunction() { }


// moduleB.js
// Can do "named imports" from other modules
import {Model} from "backbone";
const MyModel = Model.extend({});

// Can do a "default" export with the `export default` keyword
export default MyModel;

Compiling

The JS language spec has added lots of additional syntax over the years. The ES2015 spec in particular effectively doubled the amount of syntax in the language.

Each time a browser ships a new version, that version has a fixed understanding of a certain subset of the JS language. Since that browser version may stay in wide use for many years, developers need to ship code using only syntax that is supported by the set of browser versions they intend to support. However, developers also want to be able to write and develop code using the latest and greatest syntax.

This means that developers need to compile the original "current" JS source code they've written into an equivalent version that only uses older syntax.

The Babel JS compiler is the standard tool used to cross-compile JS code into a different variation of JS. It has a wide array of plugins that support compiling specific pieces of newer JS language syntax into their older equivalent forms.

As an example, this ES2015-compatible snippet uses ES Module syntax, arrow functions, the const/let variable declaration keywords, and the shorthand object declaration syntax:

export const myFunc = () => {
  let longVariableName = 1;
  return {longVariableName};
}

When compiled by Babel with the ES2015 plugin enabled and targeting the CommonJS module format, it becomes:

"use strict";

Object.defineProperty(exports, "__esModule", {
  value: true
});
exports.myFunc = void 0;

var myFunc = function myFunc() {
  var longVariableName = 1;
  return {
    longVariableName: longVariableName
  };
};

exports.myFunc = myFunc;

In addition, there are many "compile-to-JS" languages in use in the industry. Some are obscure niche languages, some were popular for a few years and have since died out (CoffeeScript). The most commonly used compile-to-JS language at this point is TypeScript, which is a statically-typed superset of JS created by Microsoft. The TypeScript compiler itself strips out type annotations at compile time and outputs plain JS. Similar to Babel, it can also compile newer syntax into varying older language versions.

// Input: TypeScript syntax is JS with type annotations
const add2 = (x: number, y: number) => {
  return x + y;
};

// Output: plain JS, no type annotations
const add2 = (x, y) => {
  return x + y;
};

Bundling

There are multiple reasons why the original JS source cannot be delivered as-is to the browser:

Code written in CommonJS format cannot be loaded by browsers
Code written in ES Module format can be, but requires careful work to have all the files and path URLs line up correctly
Target browsers used by consumers likely don't support all modern syntax
Codebases may consist of thousands of separate JS files, and downloading each file separately would take too long to load
Original source contains comments, whitespace, and longer variable names, and developers need to minimize the number of bytes sent to the browser to let pages load faster
Languages like TypeScript are not supported by JS interpreters - the original source has to be compiled to plain JS syntax

Because of this, JS code is also bundled to prepare it for use in a browser. This has to happen in both a development environment, and production.

The bundling process traces the tree of imports and dependencies starting from a set of entry point files (such as src/index.js). Any imported file is then added to the list of files to be processed. The bundler resolves all requested imports, determines the necessary loading order, and outputs the module sources wrapped in some scaffolding that initializes the application when the bundle file is loaded.

Bundling tools also typically support multiple additional processing steps during the bundling process. In particular, bundlers will usually be configured to:

run a compiler like Babel or TypeScript on all JS/TS source files
If using TS, do typechecks with the TS compiler to verify the code actually compiles
Enable importing and processing additional assets like CSS and images
Optimize the size of the output to make it as small as possibly, by minifying it (also known as uglifying).

Minifying JS source involves shrinking the code as much as possible, by stripping out whitespace and comments, replacing long variable names with shorter names, and using the shortest possible versions of syntax. Finally, minifiers can detect dead code and remove it, and JS code is often written with flags like if (process.env.NODE_ENV !== 'production') to add development-only checks that will be removed in a production build.

The same Babel-compiled output above looks like this when minified:

"use strict";Object.defineProperty(exports,"__esModule",{value:!0}),exports.myFunc=void 0;var myFunc=function(){return{longVariableName:1}};exports.myFunc=myFunc;

Webpack is the most widely used JS bundler. Other tools such as Parcel, Snowpack, and ESBuild fulfill the same roles, but with different goals and constraints.

Source Maps

Because of these transformations, the code loaded by a browser has been mangled into a completely unrecognizable form that makes it impossible to actually debug as-is. To solve this, development tools also write out source maps, which map segments of the output file back to their original lines of source. This allows browser debuggers to show the original source code, even if it's a language not actually supported by the JS interpreter in the browser. The browser will show the "original source" in its individual files, and allows developers to debug that "original source" by setting breakpoints and viewing variable contents.

Development Environments and Tools

Node.js

Node.js is a runtime for executing JS outside of a browser environment. It's equivalent to the JRE for Java, the .NET Framework SDK, or the Python runtime. It consists of the V8 JS engine from Chrome repackaged for use as a standalone executable, along with a standard library of APIs for interacting with the file system, creating sockets and servers, and much more.

NPM

"NPM" has three meanings:

NPM is the publicly-hosted JS package registry that hosts third-party JS libraries and packages published by the community
npm is an open-source CLI client used for installing packages from that registry
NPM is a company that runs the registry and develops the CLI client (recently bought by Microsoft)

Libraries and packages installed off of NPM are put into a ./node_modules folder. So, npm install redux downloads a published archive from the NPM registry servers, and extracts the contents into a new ./node_modules/redux folder.

Yarn is an alternative package manager that installs the same packages from the same public NPM package registry. It has the same core capabilities as the npm tool, but provides some different configuration options.

Node Build Tools

Since most JS build tools are written by JS developers for JS developers, the build tools themselves are typically written in JS. This includes widely used tools like Babel, Webpack, ESLint, and many others. So, to run them, you must have Node.js installed in your development environment. (As described below, you do not need Node.js installed on a server machine just to run your client code in a browser, unless you also have written your server application itself in JS.)

There's been a recent trend of alternative JS build tools being written in Rust or Go, with a goal of making compilation and bundling much faster through use of native code and parallelism. Most of these tools haven't hit mainstream usage yet, but the potential speedups are big enough that those tools are likely to pick up usage.

Dev Servers

Because the original source needs to be repeatedly re-compiled and re-bundled as a developer makes changes locally, the typical development process involves launching a dev server, a separate process that detects edits to the original source files and rebuilds the client code with the changes.

The dev server process typically acts as an HTTP proxy, and forwards requests for data and assets onwards to an actual application server.

A typical example might look like:

App server process listening on port 8080
Dev server process listening on port 3000

The developer would then browse to http://localhost:3000 to load the page. The dev server on port 3000 receives the request, loads the HTML host page and bundled JS source from its memory, and returns it. When the browser requests http://localhost:3000/images/avatar.png, the dev server forwards that to the app server at http://localhost:8080/images/avatar.png instead. Similarly, a request for data by the browser to GET http://localhost:3000/items would be forwarded to the app server at http://localhost:8080/items, and the response passed back through the dev server to the browser.

Webpack has a prebuilt dev server package available, and other tools such as Create-React-App often wrap around the Webpack dev server to provide additional configuration and capabilities.

Hot Module Reloading

Normally, recompiling a web app requires completely reloading the page to see the changed code running. This wipes out any state loaded into the app when the page is refreshed.

Tools like Webpack offer a "hot module reloading" ability. When a file is edited, the dev server recompiles with the changes, then pushes a notification to the client code in the browser. The app code can then subscribe to "some file changed" notifications, re-import the new version of the code, and swap out the old code for the new code as the app is still running.

Other tools like React's "Fast Refresh" mode can then leverage that reloading capability to swap specific parts of the app, such as replacing individual React components in real time.

Deployment

Serving Build Output

The output of a normal bundler build process is a folder full of static JS, HTML, CSS, and image files. Here's the output structure of a typical React app:

/my-project/build
    - index.html
    /static
        /css
            - main.34928ada.chunk.css
            - 2.7110e618.chunk.css
        /js
            - 2.e5df1c81.chunk.js
            - 2.e5df1c81.chunk.js.map
            - main.caa84d88.chunk.js
            - main.caa84d88.chunk.js.map
            - runtime-main.d653cc00.js
            - runtime-main.d653cc00.js.map
        /media
            - image1.png
            - image2.jpg
            - fancy-font.woff2

These are simple static files that can be served up by any web server.

To deploy these files, they just need to be uploaded to an appropriate location on the machine that hosts the server application. This is often done using a file transfer protocol such as SFTP.

Polyfills

There are many browser APIs that do not exist in older browsers, but cannot be handled by backwards-compiling syntax. This includes built-in functions, classes, and data types. A couple examples of this are the String.padStart() method and the Map data structure.

However, some of these can still be polyfilled with developer-provided implementations. Polyfills are extra code that is executed when an app is loaded, detects if a given feature exists at runtime in the current environment, and adds an artificial-but-equivalent implementation dynamically. As an example, a polyfill for String.padStart() might look something like:

if (!String.prototype.padStart) {
  String.prototype.padStart = function padStart(targetLength,padString) {
    // actual logic here
  }
}

Code Splitting

Even with minification, JS bundles can get much too large (250K, 1MB, or worse). This is often due to use of many third-party libraries for additional functionality.

Code splitting allows bundlers to break very large bundles into smaller chunks. The extra chunks are either added as additional <script> tags into the host HTML page, or dynamically downloaded as the app is running.

Some chunks might contain only third-party code, so that the "vendor chunks" can be cached by the browser and only downloaded the first time a user visits a site. Or, chunks might be common logic shared between multiple parts of an app, such as common utilities used by the main user-facing page and an admin page.

Some chunks might be "lazy loaded" only when the user activates a certain feature. For example, a rich text editor implementation might add an extra 500K to the bundle, but if it's only used in a specific modal dialog, the dialog code can dynamically import the text editor library. The bundler then detects that dynamic import, splits the editor code into a separate chunk, and the chunk is only downloaded when the user opens that modal.

Further Resources

JS Module Formats
Build Tools
Deployment
- Create-React-App docs: Deployment
- Fullstack React: Deployment introduction

How Web Apps Work: HTTP and Servers

Fri, 06 Nov 2020 09:50:00 -0500

This series of posts provides an overview of fundamental web dev concepts and technologies, what these pieces are, why they're needed, and how they relate to each other. It's not a completely exhaustive reference to everything in web development, nor is it a "how to build apps" guide. Instead, it's a map of the territory, intended to give you a sense of what the landscape looks like, and enough information that you can go research these terms and topics in more depth if needed.

Other posts in this series cover additional topics, such as:

The MDN web technologies documentation is an excellent resource for reviewing many of the specific terms and concepts listed here in more detail.

HTTP

Browsers effectively only speak one language to request information: the HyperText Transfer Protocol (HTTP). This means that a browser must talk to a server that also speaks HTTP and can respond to the browser's requests for information.

HTTP Networking

All computer networking at the software level is based on an abstraction called a socket. Sockets represent a specific open connection to another computer. A server starts listening for incoming connections, and a client asks to open a new connection. Connections are defined using numeric dot-separated IP addresses like 192.168.0.255, and an additional numeric port numbers like 8080. An individual IP address segment can be from 0-255, and port numbers range from 0-65535. Think of an IP address as a street address for an apartment building, and a port number as a specific room in that building. Any data can be sent over a socket - binary or text - it's all just bytes.

Plain HTTP is an unencrypted protocol. HTTP Secure (HTTPS) is the same content, but encrypted before being sent.

Standardized networking protocols use specific well-known port numbers. Plain HTTP defaults to port 80, and HTTPS defaults to port 443. So, an address of http://192.168.0.255 implies port 80, and https://192.168.0.255 implies port 443. Other ports are frequently used depending on project setup. For example, many web server applications listen on port 8080 or 3000 in a development environment, so in that case you would use an address like http://192.168.0.255:3000 as the destination for an HTTP connection.

Since numeric URLs are hard to remember, and a site may be made up of many systems working together, the Domain Name System (DNS) maps text name URLs like www.google.com to a specific set of numeric IP addresses.

A URL like http://www.example.com:8080/some-page?id=42#intro can be split into multiple pieces:

protocol: http://
subdomain: www
domain name: example.com
port: 8080
path: some-page
query parameters: ?id=42
fragment: #intro

HTTP Requests and Responses

HTTP is a plain-text, "stateless", request/response-based protocol. In other words:

A client application has to open a connection to the server and send a request using a text-based format
The server then parses the request, runs code to process and handle the request appropriately, and sends back a text-based response. Once the response has been sent, it closes the open socket connection.
Each request/response is an individual transaction between a client and a server, and by default there is no correlation between separate requests

Both requests and responses may also contain binary data, but the HTTP-related contents of the request and response are readable text instructions.

A typical HTTP request looks like:

GET https://www.example.com/description.html HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) Chrome/16.0.912.75 Safari/535.7
Accept: text/html,application/xhtml+xml
Referer: http://www.google.com/url?&q=example
Accept-Language: en-US,en

The first line of the request is broken down into three pieces:

The HTTP method or verb, such as GET or POST
The path of the request
The HTTP protocol version

After that, the request may contain additional headers. Each header is a piece of metadata that the client can include so that the server better understands what kind of information is being requested.

The request may also include a body after the headers, with additional data.

Responses follow the same structure:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Server: Apache/2.4
Date:   Sun, 01 Nov 2020 16:38:23 GMT
Content-Length: 17151

<html>
<head>
    <title>Page Title Here</title>
</head>
<body>
  Content here
</body>
</html>

The first line contains:

The HTTP protocol version
The HTTP status code, in a numeric form
The HTTP status reason, in a text form

That is followed by additional header lines containing metadata about the response, including the total byte size of the body. Finally, the response includes the actual body contents.

HTTP Headers

There are dozens of header fields that can be sent over HTTP, and both clients and servers can send additional arbitrary headers they've defined. Some of the most common headers are:

Requests
- User-Agent: a string describing the specific type and version of the client
- Cookie: a small piece of data previously attached by the server to help track this client
- Referer: URL of the previous page where a link was clicked
Responses
- Content-Length: size of the response in bytes
- Location: redirect request to a new URL
- Set-Cookie: creates a new cookie value
Both
- Content-Type: name of the format used in the request / response body

HTTP Methods

The HTTP protocol specifies several possible methods or verbs. Each HTTP method represents a different type of request intent:

GET: the client wants to retrieve information
POST: the client wants to create or update data on the server
PUT: the client wants to update or replace data on the server
DELETE: the client wants to delete some data on the server

Each HTTP method has different variations in how the request is formatted, and when/why it should be used by a client. In addition, some types of requests can be treated as "idempotent" (can be done many times without causing additional changes).

`GET` Requests

GET requests are used to retrieve information from the server, based on a specific URL. GET requests do not contain a request body. However, clients may include additional data as query parameters options attached to the main URL. Query params start with a ?, and are formatted as key=value pairs separated by ampersands: /endpoint?a=1&b=stuff. Spaces and special characters in URLs may need to be URL-encoded, where the original value is replaced by a % and a number: ?a=Some Value might become ?a=Some%20Value.

Since GET requests are only used for retrieving data, servers should not update data in response to a GET. This means it should be safe to make the same GET request multiple times without causing side effects.

`POST` Requests

POST requests are used to tell the server to update some data or process some information. POSTs typically include all relevant information in the body of the request, and rarely include query params.

POST request bodies typically use a few common formats

"Form-encoded": the same key=value structure as query parameters, but in the body of a POST
"Multi-part form data": a delimited format that splits the body into sections
"JSON": a string representation of JavaScript data structures like objects and arrays

`PUT` Requests

PUT requests are very similar to POST requests. Both involve sending data to the server with an intent to update. The intended difference is that a PUT is intended to create or replace a value, while a POST is intended to create or update a value. Conceptually, a PUT should be safe to do multiple times in a row, while a POST is likely to cause something to happen separately for each request.

`PATCH` Requests

PATCH requests are also similar to PUT requests, but the intent is to send a partial representation of an item, while PUT is meant to send the complete representation of an item.

`DELETE` Requests

DELETE requests are used to ask a server to delete some data. Conceptually, it should be safe to make a DELETE request multiple times - if a value is already gone, the server ought to ignore the request.

HTTP Status Codes

All HTTP responses include a numeric status code, along with a text "reason" string that describes the status code value. Status codes are 3-digit numbers, grouped into these ranges:

1xx: Informational
2xx: Successful
3xx: Redirection
4xx: Client error
5xx: Server error

Common HTTP status codes include:

200 ("OK"): successful response
301 ("Moved permanently"): the resource at this URL has been moved to a new URL
304 ("Not Modified"): The value at this URL has not changed since last asked
400 ("Bad Request"): the request was not formatted properly
401 ("Not Authorized"): not authenticated to view this
403 ("Forbidden"): not allowed to view this
404 ("Not Found"): server does not recognize this URL
500 ("Internal Server Error"): something went wrong on the server while handling this
503 ("Service Unavailable"): a portion of the server is down and cannot respond

Cookies

HTTP is inherently "stateless" by default, but servers normally want some way to correlate a specific client's identity between multiple requests. For example, if I log into a forum site, the server should remember that I'm logged in as I browse through different pages.

The HTTP spec allows servers to include Set-Cookie headers in responses with specific keys and values:

HTTP/2.0 200 OK
Content-Type: text/html
Set-Cookie: yummy_cookie=choco
Set-Cookie: tasty_cookie=strawberry

Whenever a client makes future requests to that server, the cookie values will be automatically included in the request:

GET /sample_page.html HTTP/2.0
Host: www.example.org
Cookie: yummy_cookie=choco; tasty_cookie=strawberry

Typically, a server will set a "session cookie" that contains some unique ID value, and map that unique ID internally to additional data (such as "session ID 12345 is actually user mark"). That way, each time a request is made, the server can look up the additional data needed to handle that specific user ("Request is for /messages, and the session data says the user is mark - query the database for all messages sent to user mark").

Servers

The word "server" has multiple related meanings depending on the context:

The application that opened up a socket to listen for incoming requests
An application that is specifically able to handle HTTP requests and send back responses
The physical or virtual machine that is running that server application

Here, we're going to focus on the "HTTP request handling application" meaning.

Basic HTTP Request Processing

Every HTTP server application starts handling a request by accepting an incoming socket connection, and parsing the request contents into some internal data structure. The server then inspects the HTTP request path and the method to determine the intent of the request.

Depending on how the server has been written and configured, it will then handle the request by doing some combination of:

Reading files from disk
Connecting to a database and loading / updating data
Updating session tracking information internally
Running developer-specified logic

Ultimately, the server application will fill out an HTTP response message based on all of the processing, write it to the socket, and close the connection.

Routing

Routing refers to determining what code to run based on a request URL. Servers normally use a combination of the HTTP request method and the request URL path to determine what code should handle a request. For example, GET /users might be handled by one specific function, POST /users by a second function, and GET /images/header.png by another function.

Servers typically use a convention that if /some-folder/index.html exists, it will be used as the response contents for a request to GET /some-folder/. This pattern has carried over into many other areas as well. For example, a server written using the PHP language might use /some-folder/index.php to handle requests, and the Node.js CommonJS module format will handle an attempt to load require('some-folder') by opening some-folder/index.js if it exists.

Static Files

Many HTTP requests are asking for the server to return a copy of a specific file, such as GET /images/header.png or GET /app/index.html. A server will usually handle these by mapping URLs to a specific set of folders on disk, and checking to see if a file by that name exists in the specific folder path. If it does, the server will immediately handle the request by reading the entire file from disk and returning its contents as the body of the response.

Since these files on disk typically do not change often, they are known as static files. Serving static files is a baseline capability that all HTTP server applications can do, and most web server frameworks provide a built-in mechanism for automatically serving files from a designated folder. There are many sites and tools which only host and serve static files - a user just needs to upload their files, and the site is already configured to serve them automatically.

Dynamic Server Application Logic

However, servers also need to respond to incoming requests by running additional logic written by a developer. The exact syntax and techniques used to define server logic vary widely across languages and server application frameworks, but they all share some common aspects. A typical web application server framework will allow the developer to:

Specify which HTTP method is being handled
Specify what URL routes are being handled
Read an object representing the parsed HTTP request, which will contain fields with all the headers, other metadata, and body contents
Interact with an object representing the in-progress HTTP response, which will contain fields and methods to help generate the final response contents

Many web app frameworks also implement some form of middleware, which are individual chunks of logic that are combined together. Each individual middleware typically has a specific purpose, such as adding session info to the request based on a cookie. The combined middleware then form a pipeline of preprocessing steps that are executed on every incoming request before the request is handled by the specific application logic.

The request handling code may then use whatever logic it wants to dynamically generate a response. The handler logic might:

Check the URL and HTTP method
Check cookies for a session ID and look up a user's details internally
Read query parameters
Extract data from a request body
Connect to a database to retrieve information
Run some calculations
Update the database based on the calculations or the request contents
Construct an HTML document or a JSON data structure
Send that content back as the response body

Here's a (fictional!) typical example using the Express HTTP library for the Node.js runtime:

const express = require('express');
const session = require('express-session');

const db = require('./dbConnection');

const app = express();

// Add middleware to populate req.session
app.use(
  session({
    secret: 'keyboard cat',
  })
);

// Add middleware for processing incoming request contents
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

// Add middleware to serve static files
// URLs starting with '/static' will be mapped to files in the './public' folder
app.use('/static', express.static('public'));

// Define a request handler function that receives objects for request and response
function countViews(req, res) {
  // Build up an HTML body string as we go through
  let body = '';
  // Session middleware added data to the request object
  if (req.session.views) {
    ++req.session.views;
  } else {
    req.session.views = 1;
    body += '<p>First time visiting? view this page in several browsers :)</p>';
  }

  // Send back HTML content in the response
  res.send(
    body + '<p>viewed <strong>' + req.session.views + '</strong> times.</p>'
  );
}

async function addBook(req, res) {
  // JSON body contents were processed into plain JS objects
  const { author, title } = request.body;

  // Send update request to DB and wait for it to complete
  await db.query('INSERT INTO books (author, title) VALUES ($1, $2)', [
    author,
    title,
  ]);

  // Send a specific HTTP status message and JSON-formatted response
  res.status(201).json({ status: 'success', message: 'Book added.' });
}

// Define a handler for GET requests to the '/' URL path.
app.get('/', countViews);

// Define a handler for POST requests to the '/books' URL path
app.post('/books', addBook);

if (!module.parent) {
  // Use port 3000 to listen for incoming request socket connections
  app.listen(3000);
  console.log('Express started on port 3000');
}

The exact syntax for each of these steps varies based on programming language and web framework, but all frameworks share these same basic concepts and capabilities in some form.

Response Formats

Servers can send back any data they want in a response, but most responses are sent back with one of a few common formats:

HTML: the standard markup language that describes the structure and content of a web page
CSS: the standard styling language used to define the appearance of a web page
JSON: a text-based data format based on the syntax for JavaScript objects, arrays, and primitive values
XML: a customizable text-based data format based on nested tags, similar to HTML
Actual static files (images, JavaScript files, etc)

For data, JSON has become the de-facto standard data transfer format.

Further Resources

HTTP
Other Overviews

Mark's Dev Blog

How Web Apps Work: AJAX, APIs, and Data Transfer

Table of Contents

JSON

JSON Overview

Working with JSON

AJAX

AJAX APIs and Libraries

XMLHttpRequest

fetch

Axios

jQuery

Other Request Libraries

HTTP Request Variations

Polling

Long Polling

HTTP Streaming

Server Sent Events

Websockets

CORS

Data Transfer Protocols

REST

RPC

GraphQL

Browser Storage

Cookies

localStorage

sessionStorage

indexedDB

Client URL Routing

Further Resources

How Web Apps Work: JavaScript and the DOM

Table of Contents

JavaScript Overview

Language Evolution

Runtime Environments

JS Development Constraints

Core Language

Basic Syntax

Data Types

Conditional Logic

Comparisons and Boolean Conversions

Functions

JS in Depth

Objects

Arrays

Scoping and Closures

Functions and this

Classes and Prototypes

Async

Timers and the Event Loop

Promises

async/await Syntax for Promises

Other Topics

JS Module Formats

Regular Expressions

Immutability

Lodash

DOM

DOM Query APIs

DOM Manipulation

Event Listeners

jQuery

Further Resources

How Web Apps Work: Browsers, HTML, and CSS

Table of Contents

Browsers

Browser Rendering Behavior Variations

Browser Versions

Major Browsers

Browser Notes

HTML

HTML Tags

HTML Page Structure

Common HTML Tags

Common Attributes

Forms

CSS

CSS Syntax

CSS Selectors

`XMLHttpRequest`

`fetch`

`localStorage`

`sessionStorage`

`indexedDB`

Functions and `this`

`async/await` Syntax for Promises

`GET` Requests

`POST` Requests

`PUT` Requests

`PATCH` Requests

`DELETE` Requests