---
title: Working with GPU
---

## GPU Monitoring

Here is how to poll the status of your GPU(s) in a variety of ways from your terminal:

* Watch the processes using GPU(s) and the current state of your GPU(s):

   ```
   watch -n 1 nvidia-smi
   ```

* Watch the usage stats as their change:

   ```
   nvidia-smi --query-gpu=timestamp,pstate,temperature.gpu,utilization.gpu,utilization.memory,memory.total,memory.free,memory.used --format=csv -l 1
   ```

   This way is useful as you can see the trace of changes, rather than just the current state shown by `nvidia-smi` executed without any arguments.

   * To see what other options you can query run: `nvidia-smi --help-query-gpu`.

   * `-l 1` will update every 1 sec (`--loop. You can increase that number to do it less frequently.

   * `-f filename` will log into a file, but you won't be able to see the output. So it's better to use `nvidia-smi ... | tee filename` instead, which will show the output and log the results as well.

   * if you'd like the program to stop logging after running for 3600 seconds, run it as: `timeout -t 3600 nvidia-smi ...`

   For more details, please, see [Useful nvidia-smi Queries](https://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries).

   Most likely you will just want to track the memory usage, so this is probably sufficient:

   ```
   nvidia-smi --query-gpu=timestamp,memory.used,memory.total --format=csv -l 1
   ```

* Similar to the above, but show the stats as percentages:

   ```
   nvidia-smi dmon -s u
   ```
   which shows the essentials (usage and memory). If you would like all of the stats, run it without arguments:
   ```
   nvidia-smi dmon
   ```
   To find out the other options, use:
   ```
   nvidia-smi dmon -h
   ```

* [nvtop](https://github.com/Syllo/nvtop)

   Nvtop stands for NVidia TOP, a (h)top like task monitor for NVIDIA GPUs. It can handle multiple GPUs and print information about them in a htop familiar way.

   It shows the processes, and also visually displays the memory and gpu stats.

   This application requires building it from source (needing `gcc`, `make`, et al), but the instructions are easy to follow and it is quick to build.


* [gpustat](https://github.com/wookayin/gpustat)

   `nvidia-smi` like monitor, but a compact one. It relies on [pynvml](https://pythonhosted.org/nvidia-ml-py/) to talk to the nvml layer.

   Installation: `pip3 install gpustat`.

   And here is a usage example:

   ```
   gpustat -cp -i --no-color
   ```



## Accessing NVIDIA GPU Info Programmatically

While watching `nvidia-smi` running in your terminal is handy, sometimes you want to do more than that. And that's where API access comes in handy. The following tools provide that.


### pynvml

`nvidia-ml-py3` provides Python 3 bindings for nvml c-lib (NVIDIA Management Library), which allows you to query the library directly, without needing to go through `nvidia-smi`. Therefore this module is much faster than the wrappers around `nvidia-smi`.

The bindings are implemented with `Ctypes`, so this module is `noarch` - it's just pure python.

Installation:

* Pypi:
```
pip3 install nvidia-ml-py3
```
* Conda:
```
conda install nvidia-ml-py3 -c fastai
```

This library is now a `fastai` dependency, so you can use it directly.

Examples:

Print the memory stats for the first GPU card:
```
from pynvml import *
nvmlInit()
handle = nvmlDeviceGetHandleByIndex(0)
info = nvmlDeviceGetMemoryInfo(handle)
print("Total memory:", info.total)
print("Free memory:", info.free)
print("Used memory:", info.used)
```

List the available GPU devices:

```
from pynvml import *
nvmlInit()
try:
    deviceCount = nvmlDeviceGetCount()
    for i in range(deviceCount):
        handle = nvmlDeviceGetHandleByIndex(i)
        print("Device", i, ":", nvmlDeviceGetName(handle))
except NVMLError as error:
    print(error)
```

And here is a usage example via a sample module `nvidia_smi`:

```
import nvidia_smi

nvidia_smi.nvmlInit()
handle = nvidia_smi.nvmlDeviceGetHandleByIndex(0)
# card id 0 hardcoded here, there is also a call to get all available card ids, so we could iterate

res = nvidia_smi.nvmlDeviceGetUtilizationRates(handle)
print(f'gpu: {res.gpu}%, gpu-mem: {res.memory}%')
```


### py3nvml

This is another fork of `nvidia-ml-py3`, supplementing it with [extra useful utils](https://github.com/fbcotter/py3nvml).

note: there is no `py3nvml` conda package in its main channel, but it is available on pypi.


### GPUtil

GPUtil is a wrapper around `nvidia-smi`, and requires the latter to function before it can be used.

Installation: `pip3 install gputil`.

And here is a usage example:

```
import GPUtil as GPU
GPUs = GPU.getGPUs()
gpu = GPUs[0]
    print("GPU RAM Free: {0:.0f}MB | Used: {1:.0f}MB | Util {2:3.0f}% | Total {3:.0f}MB".format(gpu.memoryFree, gpu.memoryUsed, gpu.memoryUtil*100, gpu.memoryTotal))
```

For more details see: https://github.com/anderskm/gputil

For more details see: https://github.com/nicolargo/nvidia-ml-py3

https://github.com/FrancescAlted/ipython_memwatcher



## GPU Memory Notes


### Unusable GPU RAM per process

As soon as you start using CUDA, your GPU loses some 300-500MB RAM per process. The exact size seems to be depending on the card and CUDA version. For example, on GeForce GTX 1070 Ti (8GB), the following code, running on CUDA 10.0, consumes 0.5GB GPU RAM:
```
import torch
torch.ones((1, 1)).cuda()
```
This GPU memory is not accessible to your program's needs and it's not re-usable between processes. If you run two processes, each executing code on `cuda`, each will consume 0.5GB GPU RAM from the get going.

This fixed chunk of memory is used by [CUDA context](https://stackoverflow.com/questions/43244645/what-is-a-cuda-context).

### Cached Memory

`pytorch` normally caches GPU RAM it previously used to re-use it at a later time. So the output from `nvidia-smi` could be incorrect in that you may have more GPU RAM available than it reports. You can reclaim this cache with:
```
import torch
torch.cuda.empty_cache()
```

If you have more than one process using the same GPU, the cached memory from one process is not accessible to the other. The above code executed by the first process will solve this issue and make the freed GPU RAM available to the other process.

It also might be helpful to note that `torch.cuda.memory_cached()` doesn't show how much memory pytorch has free in the cache, but it just indicates how much memory it currently has allocated, with some of it being used and may be some being free. To measure how much free memory available to use is in the cache do: `torch.cuda.memory_cached()-torch.cuda.memory_allocated()`.



### Reusing GPU RAM

How can we do a lot of experimentation in a given jupyter notebook w/o needing to restart the kernel all the time? You can delete the variables that hold the memory, can call `import gc; gc.collect()` to reclaim memory by deleted objects with circular references, optionally (if you have just one process) calling `torch.cuda.empty_cache()` and you can now re-use the GPU memory inside the same kernel.

To automate this process, and get various stats on memory consumption, you can use [IPyExperiments](https://github.com/stas00/ipyexperiments). Other than helping you to reclaim general and GPU RAM, it is also helpful with efficiently tuning up your notebook parameters to avoid `CUDA: out of memory` errors and detecting various other memory leaks.

And also make sure you read the tutorial on `learn.purge` and its friends [here](https://fastai1.fast.ai/tutorial.resources.html), which provide an even better solution.


### GPU RAM Fragmentation

If you encounter an error similar to the following:

```
RuntimeError: CUDA out of memory.
Tried to allocate 350.00 MiB
(GPU 0; 7.93 GiB total capacity; 5.73 GiB already allocated;
324.56 MiB free; 1.34 GiB cached)
```

You may ask yourself, if there is 0.32 GB free and 1.34 GB cached (i.e. 1.66 GB total of unused memory), how can it not allocate 350 MB? This happens because of memory fragmentation.

For the sake of this example let's assume that you have a function that allocates as many GBs of GPU RAM as its argument specifies:

```
def allocate_gb(n_gbs): ...
```

And you have an 8GB GPU card and no process is using it, so when a process is starting it's the first one to use it.

If you do the following sequence of GPU RAM allocations:

```
                    # total used | free | 8gb of RAM
                    #        0GB | 8GB  | [________]
x1 = allocate_gb(2) #        2GB | 6GB  | [XX______]
x2 = allocate_gb(4) #        6GB | 2GB  | [XXXXXX__]
del x1              #        4GB | 4GB  | [__XXXX__]
x3 = allocate_gb(3) # failure to allocate 3GB w/ RuntimeError: CUDA out of memory
```

despite having a total of 4GB of free GPU RAM (cached and free), the last command will fail, because it can't get 3GB of contiguous memory.

Except, this example isn't quite valid, because under the hood CUDA relocates physical pages, and makes them appear as if they are of a contiguous type of memory to pytorch. So in the example above it'll reuse most or all of those fragments as long as there is nothing else occupying those memory pages.

So for this example to be applicable to the CUDA memory fragmentation situation it needs to allocate fractions of a memory page, which currently for most CUDA cards is of 2MB. So if less than 2MB is allocated in the same scenario as this example, fragmentation will occur.

Given that GPU RAM is a scarce resource, it helps to always try free up anything that's on CUDA as soon as you're done using it, and only then move new objects to CUDA. Normally a simple `del obj` does the trick. However, if your object has circular references in it, it will not be freed despite the `del()` call, until `gc.collect()` will not be called by python. And until the latter happens, it'll still hold the allocated GPU RAM! And that also means that in some situations you may want to call `gc.collect()` yourself.

If you want to educate yourself on how and when the python garbage collector gets automatically invoked see [gc](https://docs.python.org/3/library/gc.html#gc.get_threshold) and [this](https://rushter.com/blog/python-garbage-collector/).


### Peak Memory Usage

If you were to run a GPU memory profiler on a function like `Learner` `fit()` you would notice that on the very first epoch it will cause a very large GPU RAM usage spike and then stabilize at a much lower memory usage pattern. This happens because the pytorch memory allocator tries to build the computational graph and gradients for the loaded model in the most efficient way. Luckily, you don't need to worry about this spike, since the allocator is smart enough to recognize when the memory is tight and it will be able to do the same with much less memory, just not as efficiently. Typically, continuing with the `fit()` example, the allocator needs to have at least as much memory as the 2nd and subsequent epochs require for the normal run.  You can read an excellent thread on this topic [here](https://discuss.pytorch.org/t/high-gpu-memory-usage-problem/34694).


### pytorch Tensor Memory Tracking

Show all the currently allocated Tensors:
```
import torch
import gc
for obj in gc.get_objects():
    try:
        if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
            print(type(obj), obj.size())
    except: pass
```
Note, that gc will not contain some tensors that consume memory [inside autograd](https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/22).

Here is a good [discussion on this topic](https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741) with more related code snippets.


### GPU Reset

If for some reason after exiting the python process the GPU doesn't free the memory, you can try to reset it (change 0 to the desired GPU ID):

```
sudo nvidia-smi --gpu-reset -i 0
```

When using multiprocessing, sometimes some of the client processes get stuck and go zombie and won't release the GPU memory. They also may become invisible to `nvidia-smi`, so that it reports no memory used, but the card is unusable and fails with OOM even when trying to create a tiny tensor on that card. In such a case locate the relevant processes with `fuser -v /dev/nvidia*`and kill them with `kill -9`.

This blog [post](https://jianchao-li.github.io/post/killing-pytorch-multi-gpu-training-the-safe-way/) suggests the following trick to arrange for the processes to cleanly exit on demand:
```
if os.path.isfile('kill.me'):
    num_gpus = torch.cuda.device_count()
    for gpu_id in range(num_gpus):
        torch.cuda.set_device(gpu_id)
        torch.cuda.empty_cache()
    exit(0)
```
After you add this code to the training iteration, once you want to stop it, just cd into the directory of the training program and run
```
touch kill.me
```

## Multi-GPU

### Order of GPUs

When having multiple GPUs you may discover that `pytorch` and `nvidia-smi` don't order them in the same way, so what `nvidia-smi` reports as `gpu0`, could be assigned to `gpu1` by `pytorch`. `pytorch` uses CUDA GPU ordering, which is done by [computing power](https://developer.nvidia.com/cuda-gpus) (higher computer power GPUs first).

If you want `pytorch` to use the PCI bus device order, to match `nvidia-smi`, set:

```
export CUDA_DEVICE_ORDER=PCI_BUS_ID
```

before starting your program (or put in your `~/.bashrc`).

If you just want to run on a specific gpu ID, you can use the `CUDA_VISIBLE_DEVICES` environment variable. It can be set to a single GPU ID or a list:

```
export CUDA_VISIBLE_DEVICES=1
export CUDA_VISIBLE_DEVICES=2,3
```

If you don't set the environment variables in shell, you can set those in your code at the beginning of your program, with help of: `import os; os.environ['CUDA_VISIBLE_DEVICES']='2'`.

A less flexible way is to hardcode the device ID in your code, e.g. to set it to `gpu1`:
```
torch.cuda.set_device(1)
```
