Changelog

Survey experiments involving videos now have additional configuration options for controlling the size of videos, while ACR video experiments now also support references.

We extended our support for markers, allowing you to highlight time ranges in audio and videos for raters to focus on.

You can now also adjust the compensation paid to raters for each job, allowing for faster completion times of high-priority jobs.

Radar charts are now available in the results dashboard for experiments evaluating content along multiple dimensions.

We also improved support for long videos by adding time indicators. For example, the UI of video experiments will now show the remaining watch time for experiments with "minimum play durations" exceeding 3 seconds.

We've added configuration options to give you more control over the size of videos in pairwise video experiments. For example, references can now be scaled down relative to the other videos.

Experiments can now be tagged, which makes it easier to group and organize them.

Custom rater pools now indicate how many raters are contained in them.

Finally, we introduced organization-level API keys and a new endpoint to fetch billing events of organization billing accounts.

Our API now lets you retrieve a project's billing events.

We also added a new "Top-1" sampling strategy which automatically focuses resources to identify the best-performing method.

SSO via Okta is now available for our enterprise customers.

We introduced session health scores, which indicate if raters displayed any unexpected behavior.

We also added support for project-level spending limits for organizations.

We improved the processing speed for large datasets of externally hosted files.

We added support for studies in Arabic.

Device requirements checked at the beginning of a session now take into account if datasets contain any 10-bit videos.

We updated our workflows API to be able to handle large-scale custom strategies.

This release contained several small improvements:

• We improved UI performance for very large datasets.
• The iframe size of embedded experiments can now be controlled dynamically.
• We added support for sliders in surveys.

We added support for explicit golden datasets and golden questions mixed into a session.

Project and dataset secrets now support an allowlist to restrict the endpoints with which secrets can be shared.

Pairwise video experiments now also have sequential video playback as an option.

We improved the robustness of our AI rater pools.

This release brings updates to our session and rater health scores which we use to evaluate the performance of raters.

You can now set up project-wide secrets which can be used across embedded experiments.

We've also made it possible to filter plots by parameters, and embedded experiments can now store custom metadata on slates.

Audio surveys now support a new "highlight transcript" question type, allowing raters to highlight words in a transcript. Survey experiments also gain a new "Percent chosen" metric for radio buttons and checkboxes, with new plots of these metrics on the results pages.

This release comes with a variety of improvements.

• Survey experiments can now be configured via JSON config files contained in a dataset. This makes it possible to use different questions on different slates.
• We added support for Portuguese and Italian.

We added automatic checks of video encoding settings for self-hosted videos. For example, to check if videos have been fragmented.

We updated API keys to be associated with users.

You can now set up project webhooks for dataset status changes, job completion, and session completion.

Hero videos included in introductions can now be configured to take over the full screen. They are also now configurable directly via the UI, not just via the API.

Experiment introductions can now include a custom hero image, configurable via the API.

This release contains a couple of new features:

• We added support for Japanese and Korean.
• Leaderboards are now available for ACR type experiments.
• Experiments with a single condition also now support a new uniform sampling strategy.

This release comes with a variety of improvements.

• ACR type experiments now support Elo scores.
• Aborted sessions are now automatically refunded, with no manual intervention needed.
• The download button now returns JSON consistent with our API, replacing the previous CSV format.

We introduced Elo scores for ACR experiments. These scores are based on a Plackett-Luce model which ignores the absolute scores but only considers how an individual rater would rank conditions.

Added a "scale to fit" option for ACR image experiments.

It is now possible to use self-hosted datasets by providing a list of URLs, instead of uploading media files directly to us.

We also improved our API, and it is now possible to launch jobs where previously it was only possible to configure drafts via the API.

We released new types of experiments that allow the configuration of arbitrary surveys below images, audio, or video.

We improved our support for datasets with very large numbers of conditions. This is useful, for example, when you want to collect labels for training and need to label a large number of audio, images, or videos that are not AI-generated.

We added the ability to add configurable plots to rubrics and leaderboards.

It is now also possible to create draft experiments and jobs via our API.

Today, we are opening up Mabyduck to everyone.

We made small tweaks to our design and changes to our backend to prepare for a public launch.

We added optional confidence regions to line graph visualizations of your results.

This release also adds support for references in ACR audio experiments.

This release comes with a variety of improvements.

• We refactored the way our rater pools work, paving the way for us to offer you highly customized rater pools.
• We updated the UI of the pairwise video experiment to match the recently updated UI of pairwise image experiments.
• We added support for 4 more languages, namely Spanish, Polish, Chinese, and Vietnamese.

We improved the handling of very large dataset uploads through the browser. If a dataset upload is interrupted for any reason, it is now possible to resume uploads.

This version also adds a new Markdown input field for writing introductions.

Pairwise image experiments now support references. We also introduced new configuration options for the MUSHRA experiment.

We have implemented our own pre-screening protocols. This allows us to provide you with a higher quality of raters whose ability and hardware enable them to detect fine differences between stimuli.

It is now possible to launch experiments to crowd-sourced raters through our platform.

Release 0.1.4 is packed with new features:

• A new API for fetching results programmatically.
• A new absolute category rating (ACR) experiment for audio stimuli.
• Added configuration options for pairwise audio experiments, such as the option to vote "Tie" or checking if audio has been played for a given duration.

We addressed some minor bugs in the MUSHRA experiment.

Datasets now support config files. These can be used to change the interface for each slate. For example, to display text prompts next to stimuli.