Data mining (Posts about library)

video-frame-selector library released

Applied Machine Learning Group, University of Waikato — Tue, 02 Nov 2021 00:55:00 GMT

Release 0.0.8 of the video-frame-selector Python 3 library is now available:

github.com/waikato-datamining/video-frame-selector

It includes some bug-fixes relating to the the Redis support and it now can iterate over images in a directory as well (not just processing video or webcam feed).

tflite model maker tools

Applied Machine Learning Group, University of Waikato — Wed, 06 Oct 2021 03:02:00 GMT

Tensorflow made the tflite model maker available a while go, which (according to the project website) simplifies the process of training a TensorFlow Lite model using custom dataset. It uses transfer learning to reduce the amount of training data required and shorten the training time.

But instead of falling back on scripting Python again, we pulled code from the their example notebooks into a library that offers command-line utilies and published it on PyPI to avoid having to write Python code over and over again for every new problem/dataset. It also makes it easy to create docker images from it, making it highly reusable.

video-frame-selector library released

Applied Machine Learning Group, University of Waikato — Mon, 02 Aug 2021 23:20:00 GMT

The video-frame-selector Python 3 library is now available:

github.com/waikato-datamining/video-frame-selector

With this library you can present frames obtained from video files or a webcam to an image analysis framework, such as detectron2, and react to the generated predictions. For instance, you might want to trawl through a video from a trail camera and generate either JPG images or a shortened video that contains only the frames that show actual animal. Or you might only look for a specific animal, rather than all of them (e.g., only NZ pests such as rats or stoats). Frames that are to be kept can also be cropped to the smallest area that encompasses all the detected bounding boxes (you can enforce a margin around the cropped content and/or a minimum width/height).

simple-confusion-matrix library released

Applied Machine Learning Group, University of Waikato — Fri, 16 Apr 2021 02:29:00 GMT

The simple-confusion-matrix Python 3 library has been released today:

github.com/waikato-datamining/simple-confusion-matrix

It is a simple library that can generate confusion matrices from CSV files or lists of actual and predicted labels. It can output the generated matrix either in plain text or CSV, as string or to a file.

Rather than just using counts, it can generate also:

percentages (all cells sum up to 1)
percentages per row (all cells in a row sum up to 1)

The latter is useful when dealing with imbalanced datasets, giving you a good idea of how well each label is being predicted.

PyTorch image classification available

Applied Machine Learning Group, University of Waikato — Mon, 11 Jan 2021 03:53:00 GMT

Today, a new library for performing image classification has made its debut:

wai.pytorchimageclass

The library is based on the PyTorch example code for imagenet. For ResNet-based networks, you can finetune pretrained models on your own data rather than just using the imagenet dataset. In addition, you can make predictions (single and batch/continuous), output information on built models and export trained models to TorchScript.

The library is also available via Docker images, one for GPU-based machines and one for CPU-only ones. However, the latter one should only be used for inference and not training, as it is simply too slow.

More information on the library and the Docker images is available from Github:

github.com/waikato-datamining/pytorch/tree/master/image-classification

video-frame-processor library released

Applied Machine Learning Group, University of Waikato — Tue, 05 Jan 2021 01:01:00 GMT

The video-frame-processor Python 3 library has been released today:

github.com/waikato-datamining/video-frame-processor

While it is easy with opencv to open files and webcams, there is no point in writing the same code over and over again. This is where the video-frame-processor library comes in, allowing you to concentrate on the processing of the frames from the video source.

simple-file-poller library released

Applied Machine Learning Group, University of Waikato — Sat, 12 Dec 2020 20:43:00 GMT

The simple-file-poller Python 3 library has been released this week:

github.com/waikato-datamining/simple-file-poller

This library is aimed at Python projects that perform continuous processing of files, e.g., deep learning models that locate objects in images. These projects typically pick up files from one directory, make predictions, write the output in some format to another directory and then either move the input files to the output directory or simply delete them.

Instead of having to write this code for polling and moving over and over again, the simple-file-poller library allows you to plug in your file processing code via a function that you supply to a Poller object. Furthermore, you can also supply a function that can check whether files are valid and can be processed (e.g., image files).

The Poller class supports two polling modes: time-based and watchdog-based. The former waits for a specified number of seconds between polls (if there were no files present). This simple approach can be used when the file processing is not time critical. The latter approach watches the input directory for files being created and then reacts to that immediately. This approach allows for very low latency processing, especially useful for processing pipelines.

Another feature is the ability to write any output to a temporary directory first, before moving it into the output directory. This avoids race conditions with other processes that further process the generated output files, as the files are guaranteed to have been fully written.

The following frameworks make use of the simple-file-poller now (and more will follow):