Iterators and generators

Introduction

Earlier we learned that we can traverse over the properties of an object. The property's [[enumerable]] attribute determines whether or not a property is visited when the object's properties are traversed. The for...in loop or methods like Object.keys enumerate the object's properties.

Traversing object properties may also use the object's iterator, such as in for...of loops. Objects with an iterator are called iterables. Iterating over iterables is different from enumerating object properties. Loops and methods that enumerate properties can generally not be used to iterate over properties using the iterator, and vice versa. In fact, the iterator does not have to iterate over properties at all! An iterator can successively return anything. In addition, unlike iterating over iterables, enumerating properties happens in an arbitrary order that is browser/platform dependent, as we have learned earlier. In this chapter we will explain iterables in detail.

Objects can also be "array-like", without having an iterator. It is still possible to traverse its indexed properties, even if they are non-enumerable, by simply using a for loop.

Iterators and Iterables

Iterators

In JavaScript an iterator is an object that defines a sequence of values. An iterator is not some special built-in kind of object. Any object can be turned into an iterator by following some conventions as described in the iterator protocol: an object is an iterator when it implements a method by the name next() that returns an object with properties value and done.


const myIterator = {
  from: 0,
  to: 3,
  step: 1,
  next() {
    if (this.from < this.to) {
	  let result = { value: this.from, done: false };
	  this.from += this.step;
	  return result;
    }
    return { value: this.from, done: true };
  },
}

/*
for (let num of myIterator) {
  console.log(num); // Uncaught TypeError: myIterator is not iterable
}
*/

let result = myIterator.next();
while (!result.done) {
 console.log(result.value);
 result = myIterator.next();
}
// logs:
// 0
// 1
// 2

In the above example the sequence of values was produced by calling the next() method in a while loop. Iterating over the object by using a for...of loop (as well as many other iteration methods) does not work because, although the object is an iterator, it is not iterable.

Iterables

Iterators are seldom useful on their own, since they are not iterable. Fortunately, it is very easy to construct an iterable object by using an iterator. Following the iterable protocol, we create an iterable by giving the object a method with the name of the well-known symbol Symbol.iterator and that returns an iterator.


const myIterable = {
  [Symbol.iterator]() {
    return {
      from: 0,
      to: 3,
      step: 1,	
      next() {
        if (this.from < this.to) {
	      let result = { value: this.from, done: false };
	      this.from += this.step;
	      return result;
        }
        return { value: undefined, done: true };
      }  
    }
  }, 
}

for (let num of myIterable) {
  console.log(num);
}
// logs:
// 0
// 1
// 2

console.log([...myIterable]); // logs: [ 0, 1, 2 ]

BTW. [...myIterable] in the example above spreads the values of the iterator into an array.

BTW. Note that in the example above the iterator does not iterate over properties. The returned values are not limited to property names or values; they can be anything.

Now iterating constructs, like the for...of loop or spread syntax, that require an iterable, "recognize" the Symbol.iterator named method and know that they should call next() in a loop returning the value of value until done is true. This is exactly what a for...of loop does. If it cannot find a Symbol.iterator method, it will throw a TypeError.

Each time the Symbol.iterator method is invoked, a new iterator is returned. Iterables like above can be iterated over as many times as desired.

`return this;`

If Symbol.iterator is a method of an object that is an iterator, and Symbol.iterator returns the iterator itself (return this), then the iterator is turned into an iterable. However, this creates an iterable that can only be iterated over once. Each time the Symbol.iterator method is invoked again, the same, instead of a new iterator is returned. The done property was set to false after the first iteration, so no more iterations will occur.


const myIterable = {
  from: 0,
  to: 2,
  step: 1,
  next() {
    if (this.from < this.to) {
	  let result = { value: this.from, done: false };
	  this.from += this.step;
	  return result;
    }
    return { value: undefined, done: true };
  },
  [Symbol.iterator]() {
    return this;
  },  
}

console.log(myIterable.next()); // logs: { value: 0, done: false }
console.log(myIterable.next()); // logs: { value: 1, done: false }
console.log(myIterable.next()); // logs: { value: undefined, done: true }

for (let num of myIterable) {
  console.log(num);
}
// logs nothing; calling Symbol.iterator returns the same iterator that already finished.
// "this.from" >= "this.to", so "done" equals "true".

Built-in iterables

Some built-in objects, like Array or Map, are built-in iterables. They (their prototype) have a built-in Symbol.iterator method returning an iterator.

In the next example a for...of loop iterates over a string, which is also an iterable (via the primitive wrapper object):


const myString = "Hello world!"

for (let char of myString) {
  console.log(char);
}
// Logs each character of the string, one by one.

BTW. The String's iterator takes surrogate pairs into account, but not assembled characters.

Array-likes

The term array-like is not to be confused with iterable. They are different kinds of objects. An array-like is an object with indexed properties and a length property, indication the total number of indexed properties. An array-like object does not necessarily have to be an iterable object, and vice versa. A String is both array-like and an iterable, a Map (see later chapter) is iterable, but not array-like and the last object in the next example is array-like, but not iterable.


const myString = "Hello world!"
console.log(myString[6]); // logs: "w"
console.log(myString.length); // logs: 12

const myArray = [[1, true], ["hello", {}], [Symbol("foo"), null]];
const myMap = new Map(myArray);
for (let elem of myMap) {} // no error
console.log(myMap.length); // logs: undefined

const arrayLike = {
  0: "pet",
  1: "pot",
  2: "put",  
  length: 3
};

for (let elem of arrayLike) {
  console.log(elem);
}
// Logs: Uncaught TypeError: arrayLike is not iterable

BTW. Maps will be discussed in more detail in a later chapter.

We can turn the arrayLike object from the example above into an iterable:


const myArray = {
  0: "pit",
  1: "pat",
  2: "pot",  
  length: 3,
  [Symbol.iterator]() {
    return {
	  arr: this,
      index: 0,
      next() {
        if (this.index < this.arr.length) {
	      let result = { value: this.arr[this.index], done: false };
	      this.index += 1;
	      return result;
        }
        return { value: undefined, done: true };
      }  
    }
  },   
}

for (let elm of myArray) {
  console.log(elm);
}

// logs:
// pit
// pat
// pot

It is not that array-like, non-iterable objects cannot be iterated. "Non-iterable" here means that they do not have an iterator, but you can iterate over the object's indexed properties using the length property. You can, for instance, use a common for loop to do this.


const arrayLike = {
  0: "pet",
  1: "pot",
  2: "put",  
  length: 3
};

for (let i = 0; i < arrayLike.length; i++) {
   console.log(arrayLike[i]);
}
// logs:
// "pet"
// "pot"
// "put"

In addition, the Array provides a number of iterative methods that do not use an iterator, e.g. the forEach() method. Array-like, non-iterable objects do not implement array methods. However, array methods can be called indirectly on array-like objects using function method call() that sets the value of the method's this to the array-like object.


const arrayLike = {
  0: "pet",
  1: "pot",
  2: "put",  
  length: 3
};

Array.prototype.forEach.call(arrayLike, (element) => {
    console.log(element);
  });
// logs:
// "pet"
// "pot"
// "put"

arrayLike.forEach(element => console.log(element)); // Uncaught TypeError: arrayLike.forEach is not a function

Generators

A Generator object is returned by a generator function. A Generator object conforms to both the iterator protocol and the iterable protocol. A function* declaration defines a generator function.


function* makeGenerator() {
  yield 1;
  yield 2;
  yield 3;  
};

const generator = makeGenerator();

console.log(generator[Symbol.iterator]() === generator); // logs: true
console.log(generator.next()); // logs: { value: 1, done: false }
for (const item of generator) {
  console.log(item);
}
// logs:
// 2
// 3

console.log(generator.next()); // logs: { value: undefined, done: true }
console.log([...generator]); // logs: [] // empty array

A Generator object is both an iterator and an iterable; it returns this from its Symbol.iterator method. Hence, a Generator object can only be iterated over once.

Each successive time the generator's next() method is called, the generator function's body is executed until the first next yield expression (or until the end). The yield expression defines the value to be set for the value property of the returned object { value: value, done: boolean } by the generator.


function* makeGenerator() {
  console.log("log 1");
  yield 1;
  console.log("log 2");
  yield 2;
  console.log("log 3");
};

const generator1 = makeGenerator();

console.log(generator1.next()); 
// logs: "log 1"
// logs: { value: 1, done: false }
console.log(generator1.next());
// logs: "log 2"
// logs: { value: 2, done: false }
console.log(generator1.next());
// logs: "log 3"
// logs: { value: undefined, done: true }

const generator2 = makeGenerator();
console.log([...generator2]);
// logs:
// "log 1"
// "log 2"
// "log 3"
// [ 1, 2 ]

const generator3 = makeGenerator();
for (const item of generator3) {
  console.log(item);
}
// logs:
// "log 1"
// 1
// "log 2"
// 2
// "log 3"

BTW. You do not explicitly have to create a generator before creating a for...of loop. You can also directly type for (const item of makeGenerator()) {}. The makeGenerator() creates a generator once for the entire loop.

A return statement or an error thrown inside the generator will finish the generator, that is, the done property will be set to true. If a value is returned, it will be set as the value property of the object returned by the generator. In the next example yield 4 will never be reached. When iterated over, value will not be returned when done is true. The return statement sets done to true, so in the example below iteration only returns [ 1, 2 ] (and not [ 1, 2, 3 ]).


function* makeGenerator() {
  yield 1;
  yield 2;
  return 3; 
  yield 4;
};

const generator1 = makeGenerator();

console.log(generator1.next()); // logs: { value: 1, done: false }
console.log(generator1.next()); // logs: { value: 2, done: false }
console.log(generator1.next()); // logs: { value: 3, done: true }

const generator2 = makeGenerator();
console.log([...generator2]); // logs: [ 1, 2 ]

Generator functions cannot be written as arrow functions and function* declarations are hoisted to the top of their scope. Generator functions are not constructable, that is, they cannot be used in combination with the new keyword to construct an object. Generator functions can take arguments:


function* makeGenerator(...args) {
  for (let i = 0; i < args.length; i++) {
    yield args[i];
  }
}

const generator = makeGenerator(1,2,3);

console.log([...generator]); // logs: [ 1, 2, 3 ]

Next an example of an endless loop generator. Iterating over it needs a break or return to prevent it from stalling the computer:


function* powers(n) {
  for (let current = n; ; current *= n) {
    yield current;
  }
};

const result = [];
for (const power of powers(2)) {
  if (power > 32) { break; }
  result.push(power);
}
console.log(result);
// logs: [ 2, 4, 8, 16, 32 ]

PS. Note that the generator function's body is executed until the first next yield expression, each successive time the generator's next() method is called, until the loop hits break.

Generators and `Symbol.iterator`

Since a generator function returns a Generator object, which is also an iterator, we can use it as a Symbol.iterator method:


const myIterable = {
  from: 0,
  to: 5,
};

function* makeGenerator() {
  for(let value = this.from; value <= this.to; value++) {
    yield value;
  }
};

myIterable[Symbol.iterator] = makeGenerator;

console.log([...myIterable]); // logs: [ 0, 1, 2, 3, 4, 5 ]

Or:


const myIterable = {
  from: 0,
  to: 5,
  [Symbol.iterator]: function* () {
    for(let value = this.from; value <= this.to; value++) {
      yield value;
    }
  }
};  

console.log([...myIterable]); // logs: [ 0, 1, 2, 3, 4, 5 ]

Or in a shorthand notation:


const myIterable = {
  from: 0,
  to: 5,
  *[Symbol.iterator]() {
    for(let value = this.from; value <= this.to; value++) {
      yield value;
    }
  }
};  

console.log([...myIterable]); // logs: [ 0, 1, 2, 3, 4, 5 ]

const myArray = [];
for (const item of myIterable) {
  myArray.push(item);
}
console.log(myArray); // logs: [ 0, 1, 2, 3, 4, 5 ]

Each time the Symbol.iterator method is invoked, a new Generator object is returned.

Passing arguments into generators

Calling a next() method with an argument will resume the generator function execution from the yield expression where the execution was paused, after first replacing this yield expression with the argument from the next() call.


function* makeGenerator() {
  console.log(yield 1);
  console.log(yield);
  console.log(yield 3);  
};

const generator = makeGenerator();

generator.next();
generator.next('arg1').value; // logs: "arg1"
generator.next('arg2').value; // logs: "arg2"
generator.next('arg3').value; // logs: "arg3"

The above example explained:

The first next() call pauses at the first yield expression. The yield expression sets value to 1, but this is not logged. yield 1 itself is nothing that can be logged. The first next() does not have an argument; execution was not on pause when the generator function started executing, so no yield expression could be replaced with an argument. If an argument would have been passed, it would have been ignored.
The second next() call starts with the yield expression where execution was previously put on pause and replaces the yield expression with the argument ('arg1') from the next() call. Then it resumes execution of the generator function, executing console.log('arg1'), until the next yield expression (console.log(yield)) where it pauses execution again.
The third next() call starts with the yield expression where execution was previously put on pause and replaces the expression with the argument from the next() call ('arg2'). Then it resumes execution of the generator function, executing console.log('arg2'), until the next yield expression (console.log(yield 3)) where it pauses execution again.
The fourth next() call finally executes console.log('arg3').

`yield*`

The yield* operator is used inside a generator function and delegates iteration to its operant, which is another generator or iterable object, and yields each value returned by it. This way you can "embed" generators in each other.


function* makeGenerator() {
  yield 1;
  yield* embedMakeGenerator();
  yield 5;
};

function* embedMakeGenerator() {
  yield 2;
  yield 3;
  yield 4;
};

const generator = makeGenerator();

console.log(generator.next()); //logs: {value: 1, done: false}
console.log(generator.next()); //logs: {value: 2, done: false}
console.log(generator.next()); //logs: {value: 3, done: false}
console.log(generator.next()); //logs: {value: 4, done: false}
console.log(generator.next()); //logs: {value: 5, done: false}
console.log(generator.next()); //logs: {value: undefined, done: true}

We can "embed" another generator or any other iterable object:


function* makeGenerator(...args) {
  yield* [1, 2, 3];
  yield* "ab";
  yield* args;  
};

const generator = makeGenerator(true,false);
console.log([...generator]); // logs: [ 1, 2, 3, "a", "b", true, false ]

Async iterators and generators

Sometimes we need to iterate over asynchronous processes, like over a stream of downloading data. For this purpose there are two protocols very similar to the iterator and iterable protocols: the async iterator protocol and the async iterable protocol.

An object is an async iterator when it implements a method by the name next() that returns a promise that resolves with an object with properties value and done. An object is an async iterable by giving the object a method with the name of the well-known symbol Symbol.asyncIterator and that returns an async iterator.

An async iterable can be used in a for await...of loop. Loops and methods that require synchronous iterables, like the for..of or spread syntax, do not work with async iterables. These loops and methods expect an object with a Symbol.iterator, not with a Symbol.asyncIterator (do not put both properties on an object).

In the next example a for await...of loop iterates over an async iterable. The used asynchronous function in this case is a setTimeout().


const asyncIterable = {
  [Symbol.asyncIterator]() {
    return {
	  from: 0,
      to: 3,
      next() {
        const done = (this.from === this.to);
        const value = done ? undefined : this.from++;
        return new Promise((resolve, reject) => {		
          setTimeout(() => {
            resolve({ value, done }); // object literal with shorthand properties
            reject(new Error(`Whoops, something went wrong.`));
          }, 1000); 
        });		
      },
    };
  },
};

(async () => {
  for await (const num of asyncIterable) {
    console.log(num);
  }
})();
//logs (with delays of 1 second): 
// 0
// 1
// 2

We also could have used async/await:


const asyncIterable = {
  [Symbol.asyncIterator]() {
    return {
	  from: 0,
      to: 3,
      async next() {
        await new Promise((resolve, reject) => {		
          setTimeout(() => {
            resolve();
            reject(new Error(`Whoops, something went wrong.`));
          }, 1000); 
        });
        const done = (this.from === this.to);
        const value = done ? undefined : this.from++;
        return { value, done };		
      },
    };
  },
};

(async () => {
  for await (const num of asyncIterable) {
    console.log(num);
  }
})();
//logs (with delays of 1 second): 
// 0
// 1
// 2

Or with a generator:


const asyncIterable = {
  from: 0,
  to: 5,
  async *[Symbol.asyncIterator]() {
    for(let value = this.from; value < this.to; value++) {	
      await new Promise((resolve, reject) => {		
        setTimeout(() => {
          resolve();
          reject(new Error(`Whoops, something went wrong.`));
        }, 1000); 
      });	
      yield value;
    }
  },
};  

(async () => {
  const myArray = [];
  for await (const num of asyncIterable) {
    myArray.push(num);
  }
  console.log(myArray); // logs (after 5 seconds): [ 0, 1, 2, 3, 4 ]
})();

The for await...of loops above are used for async generators. A regular generator (a sync generator) may also yield promises. In that case a regular for...of can be used, in which the yielded promises are awaited explicitly inside the loop.


function* makeGenerator() {
  yield 0;
  yield 1;
  yield new Promise((resolve, reject) => {		
    setTimeout(() => {
      resolve(2);
      reject(new Error(`Whoops, something went wrong.`));
     }, 1000); 
  });
  yield 3;
};

(async () => {
  for (const num of makeGenerator()) {
    console.log(await num);
  }
})();
// logs:
// 0
// 1
// 2 // with a delay of 1 second
// 3

Or with a try/catch/finally structure:


function* makeGenerator() {
  try {
    yield 0;
    yield 1;
    yield new Promise((resolve, reject) => {		
      setTimeout(() => {
        resolve(2);
        reject(new Error(`Whoops, something went wrong.`));
       }, 1000); 
    });
    yield 3;
  } finally {
    console.log("called finally");
  }	
};

(async () => {
  try {
    for (const num of makeGenerator()) {
      console.log(await num);
    }
  } catch (e) {
    console.log(e);
  }	
})();
// logs:
// 0
// 1
// 2 // with a delay of 1 second
// 3
// "called finally"

Method `next()` returns a promise

Be aware that a async iterator's next() method returns a promise and not directly the { value: value, done: boolean } object.


async function* makeGenerator() {
  yield 1;
  yield 2;
  yield 3;  
};

const generator = makeGenerator();

generator.next().then(value => console.log(value.value)); // logs: 1
generator.next().then(value => console.log(value)); // logs: { value: 2, done: false }
generator.next().then(value => console.log(value)); // logs: { value: 3, done: false }
generator.next().then(value => console.log(value)); // logs: { value: undefined, done: true }

// Or:

const generator2 = makeGenerator();
(async () => {
  console.log( (await generator2.next()).value ); // logs: 1
  console.log( await generator2.next() ); // logs: { value: 2, done: false }
  console.log( await generator2.next() ); // logs: { value: 3, done: false }
  console.log( await generator2.next() ); // logs: { value: undefined, done: true } 
})();