Experiments and variants

Creating an Experiment

To create a new experiment:

1. Navigate to the Hyppot admin panel at APP_URL/hyppot/panel
2. Enter a descriptive name for your experiment
3. Click "Add Experiment"
4. Define the experiment, its conversions, variants and audience, and then save it.
5. Click on the Activate button to start it

Configuring Tracked Conversions

Conversions you want to track within an experiment are defined within "Tracked conversions" section of experiment definition. Please note that before you can add the conversion there, you need to define it in the "Conversion Types Definition" UI:

Each conversion type can have required metrics value that act as a filter for the conversions. Only the conversions with metrics value equal to the ones defined for the experiment will be considered valid for the experiment. You can use them if you want to track only selected conversions, for example - click on the specific button (like "add to cart").

Configuring Variants

A variant represents a specific version of your feature that you want to test. Each variant has:

• A unique identifier,
• An audience configuration,
• Values for all variables defined in the experiment.

To add a variant:

1. Click "Add Variant"
2. Enter a variant identifier
3. Set values for all experiment variables
4. Configure the audience for this variant

Audience Configuration

Each variant's audience can be configured in three ways:

1. Percentage - What percentage of users should see this variant

   • The sum of percentages across all variants must not exceed 100%
   • A variant with 0% will not be shown to any users

2. Allow List - Specific users who should always see this variant

   • Users in the allow list will see this variant regardless of the percentage setting
   • Useful for testing with specific users or internal teams

3. Deny List - Specific users who should never see this variant

   • Users in the deny list will not see this variant
   • Useful for excluding certain users from the experiment

Audience Assignment

Hyppot uses a deterministic algorithm to assign users to variants, ensuring consistent assignment across sessions. User assignment is stable, regardless if you use .NET Backend integration or Javascript SDK. Thanks to this, you can also combine approaches to build more complex experiments.

Variant variables

You can define variables of the three types:

• String (C#: string, TypeScript: string),
• Number (C#: double, TypeScript: number),
• Boolean (C#: bool, TypeScript: boolean).

To see how can you access variant variables, see the docs for application integration.

Tracking Experiment Data

In Hyppot, there are two main events that can be tracked: Impression and Conversion. Impression is the act of presenting the experiment variant to the user, while Conversion is every important event in your app that you'd like to measure within your experiments. To see how to track events in your app, check application integration.

Impressions

An impression occurs when a user is shown a variant of your experiment. When you use npm package for Javascript, impressions are tracked automatically.

Impressions are deduplicated using either the deduplication key or by user ID within the impression length window (by default, 24 hours). This prevents the same user from being counted multiple times in a short period — for example, when they refresh the page. If you provide a deduplication key, it overrides the impression window check - multiple impressions will be registered if more than one request is sent for the same user with different deduplication keys, even if within the impression length window.

Tracking Conversions

Conversions represent user actions that indicate success for your experiment.

When tracking conversions, you don't need to specify which experiment they belong to. Simply provide the conversion's details (such as type, user ID, timestamp, and any associated metrics). Hyppot automatically links a tracked conversion to relevant experiments and counts it if the following conditions are met for an experiment:

• The experiment is active.
• The impression event must have occurred (i.e., its timestamp is earlier) before the conversion event.
• The conversion occurred within this experiment's configured conversion window (relative to the impression time).
• If the experiment's definition for this conversion type specifies required metrics, the tracked conversion's metrics must match all of them. This allows you to track only selected conversions. For example, for an experiment related to the shopping cart, you might want to count only conversions where a specific "add to cart" button was clicked. Similarly, for a newsletter experiment, you would only track clicks on newsletter-specific buttons.

Conversion window lasts 24 hours by default. This means that the conversion will be associated with the experiment if it happens up to 24 hours after the impression. You can configure this time window during the application setup.

deduplicationKey is an optional parameter that can be used to identify the impression/conversion so you don't accidentally track them twice (for example when retrying after transient error).

Multiple tracked impressions for the same experiment and user will be treated as one if happened within 24 hour time window from the first occurrence (unless you provide different deduplication keys). This can also be configured during the app setup.

The order of tracking events does not matter. You can track conversion even before impression, as long as you provide correct timestamps of when the events ocurred.

Working with Metrics

Hyppot supports tracking metrics for each conversion. Metrics provide additional data points about conversions, allowing for more detailed analysis.

Metrics use the same VariableInstance type as experiment variables:

• They support the same types: String, Number (double), and Boolean
• They're created using the same VariableInstance class and value types
• They provide additional context for each conversion

Metrics can be used in two main ways:

1. As conversion filters to only count specific conversions (as shown in "Configuring Tracked Conversions")
2. As additional data points for analysis in your experiment results.

Viewing Results

After your experiment has been running for some time, you can view the results in the Hyppot admin panel:

1. Navigate to the experiment details
2. Expand the selected experiment
3. Click on the "Statistics" tab
4. View conversion rates and other metrics for each variant