Changelog

Project and dataset secrets now support an allowlist for endpoint restriction.

Organisations can now use Okta OIDC for authentication.

Pairwise video experiments also now have a sequential video player as an option.

This release brings new session and rater health scores.

We've also improved the robustness of both AI raters and pairwise video experiments.

You can now set up project-level secrets.

We've also made it possible to filter plots by parameters, and embedded experiments can now store custom metadata on slates.

Audio surveys now support a new "highlight transcript" question type, allowing raters to highlight words in a transcript. Survey experiments also gain a new "Percent chosen" metric for radio buttons and checkboxes, with new plots of these metrics on the results pages.

Custom strategies can now also be fetched via the API.

This release comes with a variety of improvements.

• You can now search by rater ID in the session browser
• Survey experiments can now be configured via JSON config files in the dataset
• We've also added billing management for enterprises, and Portuguese and Italian translations.
• We've fixed video controls on introduction pages for experiments with hero videos.

Jobs now have a new "pending" status, and will automatically abort when an experiment is deleted. We've also added automatic checks for fragmentation of externally hosted mp4s, and disabled the ability to force-launch jobs via the API.

This release also brings rater performance leaderboards, and a handful of other API improvements.

We've updated API keys to be associated with users

You can now set up project webhooks for dataset status changes, job completion, and session completion, with webhook and secret management available in both the API and admin.

We've also improved the reliability of embedded experiments including clearer API error reporting, wider time estimate support, automatic retrying of failed media parsing, and config files now filtered out of uniform strategy stimuli.

Hero videos can now be set to take over the full screen. They are also now configurable directly via the UI.

Experiments can now include a custom hero image or video via hero_media_url, along with an optional hero_caption. Experiment API responses also now return hero_media_url in all cases, with a null value when not set.

We've also increased the character limit for experiment introduction text.

Pairwise video experiments with a discrete or continuous response type now support a "tie" option.

A few more updates in this release

• It is now possible to attach custom metadata to slates.
• We've also added Japanese and Korean translations, and leaderboards are now available for ACR type experiments.
• Experiments with a single condition also now support a uniform sampling strategy.

This release comes with a variety of improvements.

• ACR type experiments now support Elo scores.
• Aborted sessions are now automatically refunded, with no manual intervention needed.
• The download button now returns JSON consistent with our API, replacing the previous CSV format.

We've introduced a method that reinterprets rankings as pairwise comparisons, improving the efficiency of the Plackett-Luce metric.

We've also replaced rating buttons with sliders in pairwise video experiments, and ACR image experiments now support continuous scores and sliders too.

Also fixed a hover state issue in bar plots.

Added a "scale to fit" option for ACR image experiments.

It is now possible to use self-hosted datasets by providing a list of URLs, instead of uploading media files directly to us.

We also improved our API, and it is now possible to launch jobs where previously it was only possible to configure drafts via the API.

We released new types of experiments that allow the configuration of arbitrary surveys below images, audio, or video.

We improved our support for datasets with very large numbers of conditions. This is useful, for example, when you want to collect labels for training and need to label a large number of audio, images, or videos that are not AI-generated.

We added the ability to add configurable plots to rubrics and leaderboards.

It is now also possible to create draft experiments and jobs via our API.

Today, we are opening up Mabyduck to everyone.

We made small tweaks to our design and changes to our backend to prepare for a public launch.

We added optional confidence regions to line graph visualizations of your results.

This release also adds support for references in ACR audio experiments.

This release comes with a variety of improvements.

• We refactored the way our rater pools work, paving the way for us to offer you highly customized rater pools.
• We updated the UI of the pairwise video experiment to match the recently updated UI of pairwise image experiments.
• We added support for 4 more languages, namely Spanish, Polish, Chinese, and Vietnamese.

We improved the handling of very large dataset uploads through the browser. If a dataset upload is interrupted for any reason, it is now possible to resume uploads.

This version also adds a new Markdown input field for writing introductions.

Pairwise image experiments now support references. We also introduced new configuration options for the MUSHRA experiment.

We have implemented our own pre-screening protocols. This allows us to provide you with a higher quality of raters whose ability and hardware enable them to detect fine differences between stimuli.

It is now possible to launch experiments to crowd-sourced raters through our platform.

Release 0.1.4 is packed with new features:

• A new API for fetching results programmatically.
• A new absolute category rating (ACR) experiment for audio stimuli.
• Added configuration options for pairwise audio experiments, such as the option to vote "Tie" or checking if audio has been played for a given duration.

We addressed some minor bugs in the MUSHRA experiment.

Datasets now support config files. These can be used to change the interface for each slate. For example, to display text prompts next to stimuli.