Guillaume Redoulès

Showing progress

  2. Python
  3. Basics

It is a small and very popular progress meter. It is easy to use, fast (<100ns per iteration overhead), has an intelligent estimated time remaining, works nearly everywhere and is dependency-free.

A minimal example is :

from tqdm import tqdm
from time import sleep
for i in tqdm(range(100)):
    sleep(0.01)

100%|██████████| 100/100 [00:01<00:00, 98.57it/s]

It works basically everywhere : * in the notebook * in the console * on MacOS * on Linux * on Windows

The trange(N) is a convinience function that can be used as a shortcut for tqdm(range(N))

from tqdm import trange
from time import sleep
for i in trange(100):
    sleep(0.01)

100%|██████████| 100/100 [00:01<00:00, 98.31it/s]

You can add a description and units to the progress bar

from tqdm import trange
from time import sleep
for i in trange(100, desc='my progress bar', unit="epoch"):
    sleep(0.01)

my progress bar: 100%|██████████| 100/100 [00:01<00:00, 98.77epoch/s]

tqdm can be used outside of python, for example in a pipe in order to show the progress

!seq 999999 | python -m tqdm | wc -l

999999it [00:00, 2908577.61it/s]
999999

or the number of bytes per second

!seq 999999 | python -m tqdm --bytes | wc -l

6.57MB [00:00, 232MB/s]
999999

we can also have a progress bar if we specify a total

!seq 999999 | python -m tqdm --bytes --total 7628000| wc -l

 90%|███████████████████████████████████▏   | 6.57M/7.27M [00:00<00:00, 225MB/s]
999999

Iterable-based use

Wrap tqdm() around any iterable (i.e. list, numpy array, pandas dataframe, etc.)

text = ""
for c in tqdm(["a", "b", "c","d"]):
    sleep(0.25)
    text = text + c

100%|██████████| 4/4 [00:01<00:00,  3.98it/s]

The progress bar can be instantiated outside of the loop

pbar = tqdm(["a", "b", "c","d"])
for c in pbar:
    sleep(0.25)
    pbar.set_description(f"Processing {c}")

Processing d: 100%|██████████| 4/4 [00:01<00:00,  3.98it/s]

Manual

tqdm can be manually controled by using a with statement. If you specify a total (or an iterable with len()), predictive stats are displayed.

with tqdm(total=100) as pbar:
    for i in range(10):
        sleep(0.1)
        pbar.update(10)

100%|██████████| 100/100 [00:01<00:00, 99.08it/s]

Desciption and additional stats

Custom information can be displayed and updated dynamically on tqdm bars with the desc and postfix arguments.

This can be useful for machine learning where we want to print the metrics or losses during the training process.

from tqdm import trange
from random import random, randint
from time import sleep

with trange(10) as t:
    for i in t:
        t.set_description(f"GEN {i}")
        t.set_postfix(loss=random(), gen=randint(1, 999), str="h", lst=[1,2])
        sleep(0.1)

GEN 9: 100%|██████████| 10/10 [00:01<00:00,  9.80it/s, gen=927, loss=0.505, lst=[1, 2], str=h]

You can customise what your bar looks like with the bar_format option.

with tqdm(total=10, bar_format="{postfix[0]} {postfix[1][value]:>8.2g}", postfix=["Batch", dict(value = 0)]) as t:
    for i in range(10):
        sleep(0.1)
        t.postfix[1]["value"] = i/2
        t.update()

Batch      4.5

Hooks and callbacks

tqdm can be integrated with other libaries. In the example, we integrate tqdm with urllib

In order to download a file in python we use the following code but it doesn't show any progress.

import urllib.request, os

eg_link = "http://mirrors.melbourne.co.uk/ubuntu-releases/19.10/ubuntu-19.10-desktop-amd64.iso"

urllib.request.urlretrieve(eg_link, filename=os.devnull, data=None)

We can create a class called TqdmUpTo to show the progress. It is recommended to use miniters=1 whenever there is a potentially large difference in iteration speed (e.g. downloading a file over a patchy connection).

tqdm expect a call to update and urllib needs an update_to method

class TqdmUpTo(tqdm):
    def update_to(self, blocks_so_far=1, block_size=1, total=None):
        if total is not None:
            self.total = total
        self.update(blocks_so_far * block_size - self.n)

with TqdmUpTo(unit='B', unit_scale=True, miniters=1, desc=eg_link.split("/")[-1]) as t:
    urllib.request.urlretrieve(eg_link, filename=os.devnull, data=None, reporthook=t.update_to)

ubuntu-19.10-desktop-amd64.iso:   0%|          | 4.38M/2.46G [00:06<56:58, 720kB/s]

The hooks can be useful for dispalying the progress of training a neural network with keras

with tqdm(total=10, unit="epoch") as t:
    def cbk(epoch, logs):
        t.set_postfix(logs, refresh=False)
        t.update()
    cbkWrapped = keras.callbacks.LambdaCallback(on_epoch_end=cbk)

    model.fit(x, y, epochs=t.total, verbose=0, callbacks=[cbkWrapped])

  0%|          | 0/10 [00:00<?, ?epoch/s]

tqdm can also be applyied to pandas

import pandas as pd
import numpy as np
from tqdm import tqdm

df= pd.DataFrame(np.random.rand(5,10))
#Registed `pandas.progress_apply`, `pandas.Series.map_apply`, etc with tqdm
#tqdm_gui, tqdm_notebook, optional kwargs can be used
tqdm.pandas(desc="my bar!")

#replace apply by progress_apply or map by progress_map
df.progress_apply(lambda x:x**2)

/home/guillaume/anaconda3/lib/python3.7/site-packages/tqdm/std.py:654: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
  from pandas import Panel
my bar!: 100%|██████████| 10/10 [00:00<00:00, 818.46it/s]
0 1 2 3 4 5 6 7 8 9
0 0.072067 0.192319 0.354137 0.226678 0.092123 0.003836 0.190529 0.129519 0.272106 0.918287
1 0.048068 0.157399 0.047842 0.024008 0.029614 0.067597 0.891104 0.909314 0.000385 0.090840
2 0.965366 0.000364 0.159109 0.550288 0.494446 0.157180 0.113372 0.427566 0.086719 0.004057
3 0.323191 0.524031 0.000673 0.037876 0.274079 0.036120 0.479730 0.095726 0.085360 0.006432
4 0.707128 0.041636 0.339444 0.105327 0.111249 0.331656 0.172311 0.940055 0.003604 0.463608

Notebook integration

Use tnrange via the tqdm_notebook submodule

```python from tqdm import tnrange, tqdm_notebook from time import sleep

for i in tnrange(3, desc="1st loop"): for j in tqdm_notebook(range(100), desc="2nd loop"): sleep(0.01)

If you are not sure if your users are using a notebook or a console you can use tqdm.auto

```python from tqdm.auto import tqdm tqdm.pandas()