Video Quality Experts Group (VQEG)

Audiovisual HD Quality (AVHD)

The AVHD group investigates improved subjective and objective methods for analyzing commonly available video systems. AVHD examines both video-only and audiovisual quality. Currently and prior work includes:

Conference Calls

May26, 17 CET: measuring QoE for live streaming applications

Click here to join the meeting Or call in (audio only) +1 719-283-1497   (United States) or Find a local number Phone Conference ID: 601 521 941#

#1) QoE Metrics for Live Video Streaming Applications (Live QoE)

The success of a live multimedia streaming session is defined by the experience of a participating audience. Both the content communicated by the media and the quality at which it is delivered matter - for the same content, the quality delivered to the viewer is a differentiating factor. Live media streaming systems undertake a lot of investment and operate under very tight service availability and latency constraints to support multimedia sessions for their audience. Both to measure the return on investment and to make sound investment decisions, it is paramount that we be able to measure the media quality offered by these systems.”.

The detail project proposal is available here.

Questions related to this sub-project should be sent to Shahid Satti and Rohit Puri.

#2) Advanced Subjective Methods (AVHD-SUB) Project

The AVHD group investigates improved audiovisual subjective quality testing methods. This effort may lead to a revision of ITU-T Rec. P.911. Presentations on this topic are encouraged at all VQEG meetings.

Novel Scene Experiment Design Validation 

The AVHD group has begun investigation into alternative experiment designs for subjective tests. This goal is to validate subjective testing of long video sequences that are only viewed once by each subject. Adaptive streaming requires the use of long video sequences (e.g., 5 min). This prohibits the traditional full matrix design where each source scene is viewed 10 to 25 times.

For more information, see:

Lucjan Janowski; Ludovic Malfait; Margaret H. Pinson, "Evaluating experiment design with unrepeated scenes for video quality subjective assessment," Quality and User Experience, 4, 2 (2019)

To get involved, contact Lucjan Janowski and Margaret Pinson.

Subjective Test Environment (Finished)

The AVHD group conducted a joint investigation into the impact of environment on mean opinion scores (MOS). The results of that study were published in the following paper:

"The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study," IEEE Journal of Selected Topics in Signal Processing, Vol. 6, No. 6, October 2012, pp. 640-651.

Margaret H. Pinson, Lucjan Janowski, Romuald Pépion, Quan Huynh-Thu, Christian Schmidmer, Phillip Corriveau, Audrey Younkin, Patrick Le Callet, Marcus Barkowsky, and William Ingram

#3) Audiovisual Model Validation (AVHD-AS)

AVHD-AS / P.NATS Phase 2 Project, March 2017 to January 2020 (Finished)

AVHD-AS/P.NATS Phase 2 was a joint project of VQEG and ITU-T Study Group 12 Question 14. The goal was to develop a multitude of objective models, varying in terms of complexity / type of input / use-cases for the assessment of video quality in adaptive bitrate streaming services over reliable transport up to 4K.

The project plan specified (step 1) the model validation process and (step 2) a procedure to merge high performing models, to create a single model in each category. In the bitstream model category, high performing models were identified but disagreements in step 2 became an obstacle. High performing models in other categories led to the approval of:

The "AVHD-AS / P.NATS Phase II Final Report" is available here.

For a description of the AVHD-AS / P.NATS Phase II validation test, see:

A. Raake et al., "Multi-Model Standard for Bitstream-, Pixel-Based and Hybrid Video Quality Assessment of UHD/4K: ITU-T P.1204," in IEEE Access, vol. 8, pp. 193020-193049, 2020, doi: 10.1109/ACCESS.2020.3032080.

TU Ilmenau created and published a database during this effort. This is not part of the training and validation set. The other datasets are not currently available.

Question relating to this project should be sent to Shahid Mahmood Satti, Silvio Borer, Alexander Raake, and Jörgen Gustafsson.

#4) Audiovisual Quality Integration (AVHD-AV) Project

 Audiovisual Combining Function (Finished)

The goal of this project is to compare different methods of combining an audio-only MOS and a video-only MOS to an audio-visual MOS. Several proposals have been presented in the past and were collected and validated by Margaret Pinson (NTIA/ITS) and one particular method seemed most suitable:

audio-only MOS × video-only MOS = audio-visual MOS

The results of that study were published in the following paper:

Margaret H. Pinson, William J. Ingram, and Arthur A. Webster, "Audiovisual Quality Components: An Analysis," IEEE Signal Processing Magazine, vol.28, no.6, pp.60-67, Nov. 2011. 

The original plan was to conduct a competitive benchmark, but at the moment this seems obsolete and the project is dormant.

Questions should be addressed to the AVHD chairs: Shahid Satti, Quan Huynh-Thu, and Silvio Borer.

VQEG is Co-Chaired by: Margaret Pinson, NTIA/ITS and Kjell Brunnström, RISE Research Institute of Sweden AB
The VQEG website is hosted by ITS