simple-confusion-matrix library released

The simple-confusion-matrix Python 3 library has been released today:

github.com/waikato-datamining/simple-confusion-matrix

It is a simple library that can generate confusion matrices from CSV files or lists of actual and predicted labels. It can output the generated matrix either in plain text or CSV, as string or to a file.

Rather than just using counts, it can generate also:

  • percentages (all cells sum up to 1)

  • percentages per row (all cells in a row sum up to 1)

The latter is useful when dealing with imbalanced datasets, giving you a good idea of how well each label is being predicted.

PyTorch image classification available

Today, a new library for performing image classification has made its debut:

wai.pytorchimageclass

The library is based on the PyTorch example code for imagenet. For ResNet-based networks, you can finetune pretrained models on your own data rather than just using the imagenet dataset. In addition, you can make predictions (single and batch/continuous), output information on built models and export trained models to TorchScript.

The library is also available via Docker images, one for GPU-based machines and one for CPU-only ones. However, the latter one should only be used for inference and not training, as it is simply too slow.

More information on the library and the Docker images is available from Github:

github.com/waikato-datamining/pytorch/tree/master/image-classification

simple-file-poller library released

The simple-file-poller Python 3 library has been released this week:

github.com/waikato-datamining/simple-file-poller

This library is aimed at Python projects that perform continuous processing of files, e.g., deep learning models that locate objects in images. These projects typically pick up files from one directory, make predictions, write the output in some format to another directory and then either move the input files to the output directory or simply delete them.

Instead of having to write this code for polling and moving over and over again, the simple-file-poller library allows you to plug in your file processing code via a function that you supply to a Poller object. Furthermore, you can also supply a function that can check whether files are valid and can be processed (e.g., image files).

The Poller class supports two polling modes: time-based and watchdog-based. The former waits for a specified number of seconds between polls (if there were no files present). This simple approach can be used when the file processing is not time critical. The latter approach watches the input directory for files being created and then reacts to that immediately. This approach allows for very low latency processing, especially useful for processing pipelines.

Another feature is the ability to write any output to a temporary directory first, before moving it into the output directory. This avoids race conditions with other processes that further process the generated output files, as the files are guaranteed to have been fully written.

The following frameworks make use of the simple-file-poller now (and more will follow):

Keras image segmentation Docker image available

A new Docker image is available for training Keras image segmentation models using a GPU backend. The image is based on TensorFlow 1.14 and Divam Gupta's code, plus additional tools for converting indexed PNGs into RGB ones and continuously processing images with a model.

More information on the Docker image is available from Github:

github.com/waikato-datamining/tensorflow/tree/master/image-segmentation-keras

DeepSpeech Docker image available

A new Docker image is available for training DeepSpeech models using a GPU backend. The image is based on Mozilla's DeepSpeech 0.7.4 one for training models, adding more functionality to it:

  • support for MP3 and OGG files, not just WAV

  • automatic alphabet generation from the transcripts

  • split sound files into chunks based on detected pauses

  • batch process audio files (can be continuous)

  • many Python utilities have been sym-linked

More information on the Docker image is available from Github:

github.com/waikato-datamining/tensorflow/tree/master/deepspeech