Remove unwanted files
Especially during development, it can happen that files get generated when running
your docker container that can only be removed by the root
user again. If you
do not have admin permissions (e.g., via sudo
) on the machine then you can
revert to using docker to remove them again:
-
change into the directory with unwanted files and/or directories
-
the following docker command will map the current directory to
/opt/local
:docker run -v `pwd`:/opt/local -it bash:5.1.8
-
change into
/opt/local
and remove all files/dirs (or just the files that need removing):cd /opt/local rm -Rf *
Python
Alternative Python version
In case your base Debian/Ubuntu distribution does not have the right Python version available, you can get additional versions by adding the deadsnakes ppa:
add-apt-repository -y ppa:deadsnakes/ppa && \
apt-get update
Switch default Python version
If your base distro has an older version of Python and you do not want to sprinkle the
specific newer version throughout your Dockerfile
, then you can use update-alternatives
on Debian systems to change the default.
The following commands switch from Python 3.5 to Python 3.6 for the python3
executable
and also use Python 3.6 as the default for the python
executable:
update-alternatives --install /usr/bin/python python /usr/bin/python3.5 1 && \
update-alternatives --install /usr/bin/python python /usr/bin/python3.6 2 && \
update-alternatives --set python /usr/bin/python3.6 && \
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.5 1 && \
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 2 && \
update-alternatives --set python3 /usr/bin/python3.6 && \
Install packages with pip ignoring errors
An installation with pip
will fail if a dependency is not available, even if that dependency should not be required.
The following command-line install each dependency separately, therefore continuing, even if errors are encountered (source):
cat requirements.txt | sed -e '/^\s*#.*$/d' -e '/^\s*$/d' | xargs -n 1 pip install
Screen
The screen command-line utility allows you to pick up remote sessions after detaching from them. This is especially helpful when running docker in interactive mode via ssh sessions. If such an ssh session should accidentally close (e.g., internet connection lost, laptop closed), then the docker command would terminate as well.
On the remote host, start the screen
command before you launch your docker command
(you can run multiple screen
instances as well) and skip the screen with the licenses
etc using Enter
or Space
. Now you can start up the actual command. If you want to
run multiple screen sessions on the same host, then you should name them using the
-S sessionname
option to easily distinguish them.
For detaching a session, use the CTRL+A+D
key combination. This will leave your
process running in the background and you can close the remote connection.
In order to reconnect, simply ssh into the remote host and run screen -r
. If there
is only one screen session running, then it will automatically reconnect. Otherwise, it
will output a list of available sessions. Supply the name of the session that you want to
reattach to the -r
option. In case a session got interrupted and not properly detached,
use the -d
flag to first detach it.
You can exit a screen session by either typing exit
or using CTRL+D
(just like
with a bash
shell).
If you need to increase the scrollback buffer, let us say to 100,000 lines, then you can do it
-
in the current session as follows:
CTRL A : <Enter> scrollback 100000<Enter>
-
for all new sessions by adding the following line in your
$HOME/.screenrc
config file:defscrollback 100000
Default runtime
Instead of always having to type --runtime=nvidia
or --gpus=all
, you can simply define
the default runtime to use with your docker commands (source):
- edit the
/etc/docker/daemon.json
file asroot
user - insert the key
default-runtime
with valuenvidia
so that it looks something like this:
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
- save the file
- restart the docker daemon (e.g.,
sudo systemctl restart docker
)
Rebuild
Rebuilding some layers
Since docker hashes the layer instructions rather than the content, it will not detect changes to a file (e.g., modifications in a script) since the last build and will keep using the cached version of that layer.
Assuming the following Dockerfile
snippet, where file1
changed:
RUN ...<layer X>...
COPY file1 ...<layer X+1>...
COPY file2 ...<layer X+2>...
You can inject a dummy ARG variable as follows to rebuild the layers from layer X+1
onwards:
RUN ...<layer X>...
ARG blah=1
COPY file1 ...<layer X+1>...
COPY file2 ...<layer X+2>...
Every time you need rebuild from there on, simply change the value that you are assigning to the variable, to change the hash value.
Of course, once your Dockerfile
has been finalized, you can remove all these unnecessary operations and
build your image properly.
Complete rebuild
Especially during development of a docker image, it can happen that library versions have not been added to the Dockerfile
just yet. Since docker hashes the command-lines, it will not rebuild a layer (and subsequent ones) if the hash does not change. If you do not want to clear your complete system (due to time constraints or capped internet usage), then you can force docker to build the image without using the hashed layers by adding the following flag to your build command:
--no-cache
Thorough clean up
You can stop/remove all containers and images with this one-liner:
docker stop $(docker ps -a -q) && docker rm $(docker ps -a -q) && docker system prune -a
If you need to remove dangling volumes, use this:
docker volume prune
Excluding files/dirs from docker context
By default, docker will use all files and directories within the directory that contains the Dockerfile
to
the daemon to use as context. From this context, the image gets built. Should you have test data/models
in the same directory then this could mean sending quite a lot of unnecessary data to the daemon. To avoid this,
you can take advantage of the .dockerignore
file, which lists files and directories to ignore. The following
example ignores the test
directory and all vim backup files:
test/
*~
For more information, see the official documentation on the dockerignore file.