Changelog

This week's release contained many small improvements. Here are some of them:

Uniform and exhaustive sampling strategies now support crossed designs, useful for when you need each rater to see every data point.

Our parallel video player now no longer requires fragmented videos and has support for edit lists.

We made some performance improvements, allowing us to support many more studies in parallel.

It is now possible to record sessions of embedded experiments. This can be used, for example, to debug potential technical issues.

Clicking on a rater pool now exposes our proprietary rater health scores, and gives you more insight into the performance of your raters.

We are also trialing bring-your-own rater pools, allowing you to invite raters to your studies via email.

A new rater pool API allows you to create and manage rater pools programmatically.

Finally, we added support for 96 kHz AAC audio to our parallel video player.

You can now create reusable experiment templates via our app and API. This makes it easier to apply the same experiment configuration across different datasets, and to share it with other users of your project.

We have been experimenting with AI raters powered by multi-modal LLMs. They are now widely available on our platform.

We updated our pricing tiers and launched a new landing page.

This release also included a number of small bug fixes.

It is now possible to configure pre- and post-session surveys in experiments. This allows you to ask raters questions about their overall experience, for example.

Fetching slates or ratings now supports pagination, making it easier to retrieve the results of large-scale experiments.

We added the option to exclude raters from jobs who participated in recent jobs.

We also added more demographic filtering options.

Survey experiments involving videos now have additional configuration options for controlling the size of videos, while ACR video experiments now also support references.

We extended our support for markers, allowing you to highlight time ranges in audio and videos for raters to focus on.

You can now also adjust the compensation paid to raters for each job, allowing for faster completion times of high-priority jobs.

We added a new "exhaustive" sampling strategy for pairwise experiments. This strategy will sample every possible pair of a dataset in pseudorandom order.

We also replaced our documentation backend, which will make it easier for us to keep documentation up-to-date and in sync with features in our app and API.

Radar charts are now available in the results dashboard for experiments evaluating content along multiple dimensions.

We also improved support for long videos by adding time indicators. For example, the UI of video experiments will now show the remaining watch time for experiments with "minimum play durations" exceeding 3 seconds.

We've added configuration options to give you more control over the size of videos in pairwise video experiments. For example, references can now be scaled down relative to the other videos.

Experiments can now be tagged, which makes it easier to group and organize them.

Custom rater pools now indicate how many raters are contained in them.

Finally, we introduced organization-level API keys and a new endpoint to fetch billing events of organization billing accounts.

We added self-managed custom rater pools, giving you full control over the demographics of a rater pool.

We also made search bars all across the app a lot more powerful.

Our API now lets you retrieve a project's billing events.

We also added a new "Top-1" sampling strategy which automatically focuses resources to identify the best-performing method.

SSO via Okta is now available for our enterprise customers.

We introduced session health scores, which indicate if raters displayed any unexpected behavior.

We also added support for project-level spending limits for organizations.

We improved the processing speed for large datasets of externally hosted files.

We added support for studies in Arabic.

Device requirements checked at the beginning of a session now take into account if datasets contain any 10-bit videos.

We updated our workflows API to be able to handle large-scale custom strategies.

Pairwise audio experiments now support evaluation of audio along multiple criteria, matching our pairwise video experiments.

This release contained several small improvements:

• We improved UI performance for very large datasets.
• The iframe size of embedded experiments can now be controlled dynamically.
• We added support for sliders in surveys.

We added support for explicit golden datasets and golden questions mixed into a session.

Project and dataset secrets now support an allowlist to restrict the endpoints with which secrets can be shared.

This release comes with a variety of new features.

• We added a new development mode which allows live editing of embedded experiments.
• The review mode of sessions now displays meta information.
• The Workflows API now supports config files.
• We improved the performance of our database to better handle large-scale experiments.
• Self-hosted datasets can now also be specified as a CSV file uploaded to our platform.

Pairwise video experiments now also have sequential video playback as an option.

We improved the robustness of our AI rater pools.

This release brings updates to our session and rater health scores which we use to evaluate the performance of raters.

You can now set up project-wide secrets which can be used across embedded experiments.

We've also made it possible to filter plots by parameters, and embedded experiments can now store custom metadata on slates.

Audio surveys now support a new "highlight transcript" question type, allowing raters to highlight words in a transcript. Survey experiments also gain a new "Percent chosen" metric for radio buttons and checkboxes, with new plots of these metrics on the results pages.

We improved the calibration of Elo score error bars, which also improves our active sampling strategies.

This release comes with a variety of improvements.

• Survey experiments can now be configured via JSON config files contained in a dataset. This makes it possible to use different questions on different slates.
• We added support for Portuguese and Italian.

We added automatic checks of video encoding settings for self-hosted videos. For example, to check if videos have been fragmented.

We updated API keys to be associated with users.

You can now set up project webhooks for dataset status changes, job completion, and session completion.

Hero videos included in introductions can now be configured to take over the full screen. They are also now configurable directly via the UI, not just via the API.

Pairwise video experiments can now include arbitrary checkboxes.

We've also added Elo support for MUSHRA experiments.

Experiment introductions can now include a custom hero image, configurable via the API.

This release contains a couple of new features:

• We added support for Japanese and Korean.
• Leaderboards are now available for ACR type experiments.
• Experiments with a single condition also now support a new uniform sampling strategy.

This release comes with a variety of improvements.

• ACR type experiments now support Elo scores.
• Aborted sessions are now automatically refunded, with no manual intervention needed.
• The download button now returns JSON consistent with our API, replacing the previous CSV format.

We introduced Elo scores for ACR experiments. These scores are based on a Plackett-Luce model which ignores the absolute scores but only considers how an individual rater would rank conditions.

We added new response types and support for multiple dimensions in pairwise video experiments.

It is now possible to create datasets, experiments, and jobs in a single API request via our new workflows API. This endpoint also supports specifying a fully custom strategy.

This release also added custom S3 storage support for our enterprise customers.

Added a "scale to fit" option for ACR image experiments.

It is now possible to use self-hosted datasets by providing a list of URLs, instead of uploading media files directly to us.

We also improved our API, and it is now possible to launch jobs where previously it was only possible to configure drafts via the API.

Our embedded experiments are now widely available. This type of experiment uses JavaScript to include arbitrary content in experiments, and is ideal for running interactive studies.

We released new types of experiments that allow the configuration of arbitrary surveys below images, audio, or video.

We improved our support for datasets with very large numbers of conditions. This is useful, for example, when you want to collect labels for training and need to label a large number of audio, images, or videos that are not AI-generated.

Selection strategies have received more configuration options. For example, it is now possible to evaluate only a subset of a dataset. It is also possible to always include one method in pairwise comparisons against other methods.

This release also makes it possible to scale (instead of cropping) images in pairwise image experiments.

We added the ability to add configurable plots to rubrics and leaderboards.

It is now also possible to create draft experiments and jobs via our API.

Today, we are opening up Mabyduck to everyone.

We made small tweaks to our design and changes to our backend to prepare for a public launch.

This release contained several improvements:

• An updated session browser, which makes it easier to see demographic and other meta information at a glance.
• We added the option to filter leaderboards to include or exclude certain conditions.
• We now automatically fragment mp4 files uploaded to our platform. Our pairwise video experiments previously required you to upload already fragmented videos.

We added optional confidence regions to line graph visualizations of your results.

This release also adds support for references in ACR audio experiments.

This release comes with a variety of improvements.

• We refactored the way our rater pools work, paving the way for us to offer you highly customized rater pools.
• We updated the UI of the pairwise video experiment to match the recently updated UI of pairwise image experiments.
• We added support for 4 more languages, namely Spanish, Polish, Chinese, and Vietnamese.

We improved the handling of very large dataset uploads through the browser. If a dataset upload is interrupted for any reason, it is now possible to resume uploads.

This version also adds a new Markdown input field for writing introductions.

We introduced the ability for raters to leave feedback on individual slates and alert us to any potential issues with an experiment.

This version also updated the leaderboards' design.

We internationalized our experiments. In addition to English, we now support French and German.

Additionally, different experiments can now use different config files. This allows you to upload a single dataset with multiple config files for different experiments.

Pairwise image experiments now support references. We also introduced new configuration options for the MUSHRA experiment.

We introduced a new pairwise image experiment. We also added a way to preview images in datasets.

We have implemented our own pre-screening protocols. This allows us to provide you with a higher quality of raters whose ability and hardware enable them to detect fine differences between stimuli.

It is now possible to launch experiments to crowd-sourced raters through our platform.

We added configuration options to change how waveforms are rendered in MUSHRA experiments. In particular, it is now possible to only render the waveform of the reference so that raters can not draw conclusions based on the waveform.