Video Quality Experts Group (VQEG)

No Reference Metrics (NORM)


The NORM (No Reference Metrics) group is an open collaborative for developing No-Reference metrics and methods for monitoring use case specific visual service quality.

Working Methods:


The goal is to predict the overall quality (Mean Opinion Score, MOS) plus key indicators describing visual impairments, and relate these to root cause for each use case. The MOS for the NORM project is intended to accurately predict the subjective user perception of video quality for the use case (e.g. Does the “HD” video provided by a professional OTT VoD service look like it should on a TV, tablet, and smartphone? Or does the drone footage from a Search and Rescue look good enough to find an adult or child in the mountains?). Since subjective video quality is highly use case dependent, the methods and metrics used for one use case may be different than other use cases. The project’s goal does not exclude new algorithmic approaches, but intends to leverage existing algorithms and define methods that leverage machine learning to train them for each use case.

The NORM group is a complementary, industry-driven alternative of QoE (Quality of Experience) to measure automatically the visual quality by using perceived indicators. The perceived indicators should have a robust prediction performance with a minimum of operational restriction. The algorithms and training methods will include the ability to detect visual impairments caused by cameras, encoding, decoding, scaling, and transmission while excluding false positives from artistic intent. The algorithms and training methods will detect quality improvements from composition, subject matter, aesthetics, and artistic intent. Targeted use cases include Video on Demand (VoD), live broadcast services, social media, first responder video, medical, and AI vision systems (autonomous vehicles). Measuring the visual quality of these use cases to provide a Mean Opinion Score (MOS) may be independent from the key indicator measurements and the root cause correlation, which may include parametric (bitstream) measurement.

The ability to use the no-reference model defined by NORM is dependent on the availability of decoded video at the point of measurement (camera, contribution, encoding, network, end-user device). NORM requires the access to decoded video in order to determine the visual quality and key indicators. Access to the bitstream to aid in root cause is beneficial, but not required.

NORM is an open collaborative group that intends to make all of its work public, royalty free. Where applicable, NORM will provide metric code in OpenSource.

The NORM activities are:

Information and Tools

Uses Cases

Professionally Produced Entertainment Use Cases

 User Generated Content Use Cases

Industrial and Application Specific Use Cases

Acceptable No Reference Metric Constraints—Where Do We Start?

Desirable No Reference Metric Traits—What Is Our Design Goal?

No Reference Metric Resources

See here for potential resources for developing no reference metrics.

VQEG is Co-Chaired by: Margaret Pinson, NTIA/ITS and Kjell Brunnström, RISE Research Institute of Sweden AB
The VQEG website is hosted by ITS