Data mining (Posts about update)

S3000 Pod Support

Applied Machine Learning Group, University of Waikato — Thu, 28 May 2026 01:51:00 GMT

Typically, our commercial framework for laboratories, S3000, combines training of models/cleaners/evaluators/etc and generating predictions on the same server. But that was mostly due to S3000 instances starting out with a small number of models, not because it was a requirement. Especially for mission-critical systems that output predictions 24/7, it makes sense to have a separate grunty server for training the models and then one or more light-weight servers that serve the predictions.

However, copying the versioned directory structures from the training directory across to other servers is not particularly efficient and it is easy to make mistakes. Especially, if one has to update the setups and/or components as well.

In order to speed up deployments of new models or simply make replacing only a few models easier, S3000 introduces the concept of Pods. A Pod is a self-contained zip file that combines all the serialized models like classifier and evaluators plus a JSON configuration file describing the pod. Prediction generators that are Pod-aware will then read these Pod files and their configuration to generate a prediction workflow on-the-fly which will then load the models into internal storage, ready to be used.

Currently, only SIMPLE components are supported and FUSION will be added at a later stage.

S3000 HTML Support

Applied Machine Learning Group, University of Waikato — Tue, 05 May 2026 04:10:00 GMT

During training runs, our commercial framework for laboratories, S3000, can output cross-validation statistics, which include summary statistics, spreadsheets and plots. Up till now, the plots were PNG image files, making it easy to include them in reports or quickly scan the model performances.

A while ago, support for kernel-density estimate (KDE) plots was introduced, which can tell a better story for datasets with a lot of data points. The default scatter plot for actual vs predicted can be replaced using the ConfigTags.props file, e.g., with this KDE setup:

output_model_statistics#chart=adams.gui.visualization.jfreechart.chart.DensityScatterPlot -label-x actual -label-y predicted -mode KDE -generator "adams.gui.visualization.core.BiColorGenerator -first-color #0000ff -second-color #ffc800 -alpha 127"

However, static plots make it easy to spot outliers, but much harder to identify them. ADAMS introduced the generation of self-contained HTML plots in the WEKA Investigator, which can now be used in S3000 as well when outputting model statistics. The following ConfigTags.props entry will

output_model_statistics#html=adams.flow.transformer.actualvspredictedprocessor.ClassifierErrorsKernelDensityEstimate -circle-size 10

S3000 Customization

Applied Machine Learning Group, University of Waikato — Mon, 06 Oct 2025 03:43:00 GMT

Out of the box, our S3000 software comes with predefined workflow generator plugins that can be configured to suit a customer's needs. However, sometimes it is necessary to further refine that configuration, e.g., when generating statistics during training runs. In order not to overload the generator plugins with even more options, or create customized generators, the notion of config tags has been introduced. Generators that support such customization will state so in their help screen and list the supported tags. These tags can then be customized in the ConfigTags.props properties file.

S3000 REST webservice support

Applied Machine Learning Group, University of Waikato — Wed, 02 Apr 2025 21:35:00 GMT

While our commercial framework for laboratories, S3000, had support for making predictions via webservices for a long time, that was limited to asynchronous ones: a webservice endpoint receives data coming in and, once the predictions have been generated, the results get forwarded to another webservice.

With recent changes to the codebase, it is now possible to offer synchronous REST webservices as well. In order to reduce latency as much as possible, provenance logging under the hood has been modified to have a much higher throughput that no longer impacts the speed of the predictions.

Thanks to the plugin architecture of S3000, customer-specific webservices can be implemented and deployed with minimal effort.

S3000 Python support

Applied Machine Learning Group, University of Waikato — Tue, 04 Mar 2025 04:07:00 GMT

Our commercial offering for laboratories, S3000, now has official support for custom Python models. Being a Java application, S3000 historically relied on Weka for its modeling. However, with the integration of Jep (Java Embedded Python) in ADAMS, it is now possible to use scikit-learn or deep learning libraries for training models and making predictions.

In order to make this work, ADAMS needs to launch from the context of an activated virtual Python environment that contains all the required Python libraries and then use the JepRegressor Weka meta-classifier to define the relevant code for training and making predictions. Due to Java serialization not being applicable to Python objects, these scripts must implement the relevant serialization/deserialization themselves, e.g., via pickle.

Redis-related Docker image updates

Applied Machine Learning Group, University of Waikato — Tue, 08 Aug 2023 23:41:00 GMT

The redis-docker-harness Python library, which is used by a lot of our Docker images, has received a number of updates (at time of writing, the version of the library in use is 0.0.4):

ability to specify a password for the Redis server
specify the timeout parameter for the the Redis client, with larger timeouts resulting in lower CPU load (the default is now 0.01 instead of 0.001)

Unfortunately, this required re-releasing the most recent images of the following frameworks:

detectron2
mmdetection
mmsegmentation
yolov5
yolov7
Segment Anything (SAM)
DEXTR

The images kept their version number, you just need to pull them again, or use --pull ALWAYS in conjunction with docker run.