www.its.bldrdoc.gov - >> Resources >> Video Quality Research >> Documents Dynamic

Video Quality Documents

   
   

Lucjan Janowski; Margaret H. Pinson, "Subject Bias: Introducing a Theoretical User Model," Fifth International Workshop on Quality of Multimedia Experience (QoMEX 2014), Singapore, 18-20 September 2014

We propose a model for rating behavior based on subject bias and subject error. Evidence for subject bias can be found in freely available subjective experiments. When subject bias is removed from ratings, the sensitivity of statistical comparisons b...

   

Margaret H. Pinson; Lucjan Janowski, AGH/NTIA: A Video Quality Subjective Test with Repeated Sequences, NTIA Technical Memo TM-14-505, June 2014

This report provides full technical details for the video quality subjective test AGH/NTIA. Analyses of this dataset appear in separate publications. The purpose of this document is to provide design details that are beyond the scope of a conference ...

   

Margaret Pinson; Naeem Ram, VQEG eLetter: Volume 1, Issue 1, March 2014

The VQEG eletter publishes up-to-date technical advances on video quality related topics. The VQEG eLetter also makes available limited distribution publications that contain valuable technical information, such as out-of-print ATIS contributions. Th...

   

Margaret H. Pinson; Marc Sullivan; Andrew Catellier, "A new method for immersive audiovisual subjective testing," Eighth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM 2014), Chandler, AZ, January 30-31, 2014

An immersive subjective test method is proposed in which subjects view each source stimulus only once. In order to encourage a subject’s engagement with test content, longer stimuli are used. Distractor questions are used in addition to the tradition...

   

Andrew A. Catellier; Luke Connors, Web-Enabled Subjective Test (WEST) Research Tools Manual, NTIA Handbook HB-14-501, January 2014

The Web-Enabled Subjective Test software allows researchers to conduct subjective tests on multiple devices with aggregated data collection and reporting. The system leverages modern web technologies and infrastructure to deliver test stimuli to typi...

   

M. H. Pinson; N. Staelens; A. Webster, "The history of video quality model validation," 2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP), pp.458,463, Sept. 30 2013-Oct. 2 2013 doi: 10.1109/MMSP.2013.6659332

This paper describes objective video quality validation efforts conducted in the past two decades. Validation efforts to be examined include a validation test performed by the T1A1 committee in the early 1990's; five rounds of validation testing perf...

   

Margaret H Pinson; Marcus Barkowsky; Patrick Le Callet, "Selecting scenes for 2D and 3D subjective video quality tests," EURASIP Journal on Image and Video Processing 2013, 2013:50 doi: 10.1186/1687-5281-2013-50

This paper presents recommended techniques for choosing video sequences for subjective experiments. Subjective video quality assessment is a well-understood field, yet scene selection is often driven by convenience or content availability. Three-dime...

   

M. H. Pinson; C. Schmidmer; L. Janowski; R. Pépion; Q. Huynh-Thu; P. Corriveau; A. Younkin; P. LeCallet; M. Barkowsky; W. Ingram, "Subjective and Objective Evaluation of an Audiovisual Subjective Dataset for Research and Development," Fifth International Workshop on Quality of Multimedia Experience (QoMEX 2013), Klagenfurt am Wörthersee, Austria, July 3-5, 2013

In 2011, the Video Quality Experts Group (VQEG) ran subjects through the same audiovisual subjective test at six different international laboratories. That small dataset is now publically available for research and development purposes.

   

Margaret H. Pinson, "The Consumer Digital Video Library [Best of the Web]," IEEE Signal Processing Magazine, vol. 30, no. 4, pp. 172,174, July 2013 doi: 10.1109/MSP.2013.2258265

The difficulty of finding and getting rights to use the high quality video sequences needed for video quality research has hampered research in this area for many years. This article describes the Consumer Digital Video Library (CDVL) website (www.cd...

   

Margaret H. Pinson; Stephen Wolf, Fast Low Bandwidth Model: A Reduced Reference Video Quality Metric, NTIA Technical Memo TM-13-497, June 2013

This memorandum describes the Fast Low Bandwidth model. This summary is intended to help the reader understand the model from an algorithmic standpoint. Some knowledge of prior NTIA objective video quality models is necessary for the understanding of...

   

Joel Dumke, "Visual acuity and task-based video quality in public safety applications," Proc. of SPIE-Image Quality and System Performance X, Electronic Imaging, SPIE Vol. 8653, 865306, Burlingame, CA, 5-7 February 2013 doi: 10.1117/12.2004882

This paper explores the utility of visual acuity as a video quality metric for public safety applications. An experiment has been conducted to track the relationship between visual acuity and the ability to perform a forced-choice object recognition ...

   

Margaret H. Pinson; Karen Sue Boyd; Jessica Hooker; Kristina Muntean, "How To Choose Video Sequences For Video Quality Assessment," Proceedings of the Seventh International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM-2013), Scottsdale, AZ, January 30-February 1, 2013

This paper presents recommended techniques for choosing video sequences for subjective experiments. Subjective video quality assessment is a well understood field, yet scene selection is often limited by content availability. The Consumer Digital Vid...

   

Margaret H. Pinson; Lucjan Janowski; Romuald Pépion; Quan Huynh-Thu; Christian Schmidmer; Phillip Corriveau; Audrey Younkin; Patrick Le Callet; Marcus Barkowsky; William Ingram, "The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study," IEEE Journal of Selected Topics in Signal Processing, Vol. 6, No. 6, October 2012, pp. 640–651

Traditionally, audio quality and video quality are evaluated separately in subjective tests. Best practices within the quality assessment community were developed before many modern mobile audiovisual devices and services came into use, such as inter...

   

Andrew Catellier; Margaret Pinson; William Ingram; Arthur Webster, "Impact of Mobile Devices and Usage Location on Perceived Multimedia Quality," Proceedings of the 4th International Workshop on Quality of Multimedia Experience (QoMEX), pp. 39-44, Yarra Valley, Australia, July 5-7, 2012.

We explore the quality impact when audiovisual content is delivered to different mobile devices. Subjects were shown the same sequences on five different mobile devices and a broadcast quality television. Factors influencing quality ratings include v...

   

Margaret H. Pinson; William Ingram; Arthur Webster, "Audiovisual Quality Components: An Analysis," IEEE Signal Processing Magazine, vol.28, no.6, pp.60-67, Nov. 2011 doi: 10.1109/MSP.2011.942470

The perceived quality of an audiovisual sequence is heavily influenced by both the quality of the audio and the quality of the video. The question then arises as to the relative importance of each factor and whether a regression model predicting audi...

   

Margaret H. Pinson; Stephen Wolf, Batch Video Quality Metric (BVQM) User's Manual, NTIA Handbook HB-11-441d, September 2011

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Windows 7® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as ou...

   

Stephen Wolf; Margaret H. Pinson, Video Quality Model for Variable Frame Delay (VQM_VFD), NTIA Technical Memo TM-11-482, September 2011

Time varying delays of the output (or processed) video frames with respect to the input (i.e., the original or reference) video frames present significant challenges for Full Reference (FR) video quality measurement systems. Time alignment errors bet...

   

Gregory Cermak; Margaret H. Pinson; Stephen Wolf, "The Relationship Among Video Quality, Screen Resolution, and Bit Rate," IEEE Transactions on Broadcasting, vol.57, no.2, pp.258-262, June 2011 doi: 10.1109/TBC.2011.2121650

How much bandwidth is required for good quality video for a given screen resolution? Data acquired during two Video Quality Experts Group (VQEG) projects allow at least a partial answer to this question. This international subjective testing produced...

   

Stephen Wolf, Variable Frame Delay (VFD) Parameters for Video Quality Measurements, NTIA Technical Memo TM-11-475, April 2011

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol—IP), and a video decoder can produce pauses in the video presentation, after which the video may continue with or without skippin...

   

Margaret H. Pinson; Arthur Webster; William Ingram, Preliminary Investigation into the Impact of Audiovisual Synchronization of Impaired Audiovisual Sequences, NTIA Technical Memo TM-11-474, March 2011

The quality perception of an audiovisual sequence is heavily influenced by the quality of the audio, the quality of the video, and the audiovisual time synchronization. The questions then arise: what is the relative importance of each factor, and can...

   

Margaret H. Pinson; Stephen Wolf; Gregory Cermak, "HDTV Subjective Quality of H.264 vs. MPEG-2, with and without Packet Loss," IEEE Transactions onBroadcasting, vol.56, no.1, pp.86-91, March 2010 doi: 10.1109/TBC.2009.2034511

The intent of H.264 (MPEG-4 Part 10) was to achieve equivalent quality to previous standards (e.g., MPEG-2) at no more than half the bit-rate. H.264 is commonly felt to have achieved this objective. This document presents results of an HDTV subjectiv...

   

Marcus Barkowsky; Margaret Pinson; Romuald Pépion; Patrick Le Callet, "Analysis of Freely Available Subjective Dataset for HDTV including Coding and Transmission Distortions," Fifth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM–10), Scottsdale, Arizona, January 13–15, 2010

We present the design, preparation, and analysis of a subjective experiment on typical HDTV sequences and scenarios. This experiment follows the guidelines of ITU and VQEG in order to obtain reproducible results. The careful selection of content and ...

   

Margaret H. Pinson; Stephen Wolf; Neha Tripathi; Chin Koh, "The Consumer Digital Video Library," Fifth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM–10), Scottsdale, Arizona, January 13–15, 2010

The shortage of high quality video sequences has hampered video quality research for 20 years. Every Video Quality Experts Group (VQEG) test has encountered difficulty in obtaining quality source content. For some validation tests, this caused a mult...

   

Stephen Wolf, A Full Reference (FR) Method Using Causality Processing for Estimating Variable Video Delays, NTIA Technical Memo TM-10-463, October 2009

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation, after which the video may continue with or without skipp...

   

K. Brunnstrom; D. Hands; F. Speranza; A. Webster, "VQEG validation and ITU standardization of objective perceptual video quality metrics [Standards in a Nutshell]," IEEE Signal Processing Magazine, vol.26, no.3, pp. 96-101, May 2009 doi: 10.1109/MSP.2009.932162

For industry, the need to access accurate and reliable objective video metrics has become more pressing with the advent of new video applications and services such as mobile broadcasting, Internet video, and Internet Protocol television (IPTV). Indus...

   

Stephen Wolf; Margaret Pinson, "Reference Algorithm for Computing Peak Signal to Noise Ratio (PSNR) of a Video Sequence with a Constant Delay," ITU-T Contribution COM9-C6-E, Geneva, February 2-6, 2009.

Peak Signal to Noise Ratio (PSNR) has been used as a benchmark to evaluate new objective perceptual video quality metrics. For example, PSNR has been used as a benchmark for both the Multimedia (MM) and Reduced Reference Television (RRTV) test progra...

   

Stephen Wolf; Margaret Pinson, "Fast Low Bandwidth Video Quality Model (VQM) Description and Reference Code," ITU-T Contribution COM9-C5-E, Geneva, February 2-6, 2009

Study Group 9 has been working on finalizing Draft New Recommendation J.redref which specifies Reduced Reference (RR) methods for performing video quality measurements of digital television systems. The U.S. National Telecommunications and Informatio...

   

Stephen Wolf, "A No Reference (NR) and Reduced Reference (RR) Metric for Detecting Dropped Video Frames," Fourth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM-09), Scottsdale, Arizona, January 15-16, 2009

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation that result from dropped or repeated video frames. For ex...

   

Margaret H. Pinson; Stephen Wolf, In-Service Video Quality Metric (IVQM) User’s Manual, NTIA Handbook HB-09-434b, January 2009

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Red Hat Linux® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., a...

   

Margaret H. Pinson; Stephen Wolf, Batch Video Quality Metric (BVQM) User’s Manual, NTIA Handbook HB-09-441c, January 2009

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Vista® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as output...

   

Margaret H. Pinson; Stephen Wolf, Techniques for Evaluating Objective Video Quality Models Using Overlapping Subjective Data Sets, NTIA Technical Report TR-09-457, November 2008

This report presents techniques for evaluating objective video quality models using overlapping subjective data sets. The techniques are demonstrated using data from the Video Quality Experts Group (VQEG) Multi-Media (MM) Phase I experiments. These r...

   

Stephen Wolf, A No Reference (NR) and Reduced Reference (RR) Metric for Detecting Dropped Video Frames, NTIA Technical Memo TM-09-456, October 2008

Digital video transmission systems consisting of a video encoder, a digital transmission method (e.g., Internet Protocol – IP), and a video decoder can produce pauses in the video presentation that result from dropped or repeated video frames. For ex...

   

Margaret H. Pinson; Stephen Wolf, Batch Video Quality Metric (BVQM) User’s Manual, NTIA Handbook HB-08-441b, November 2007

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® or Red Hat Linux® operating systems. BVQM performs objective automated quality assessments of processed video clip batches (i.e., a...

   

Margaret H. Pinson; Stephen Wolf, Reduced Reference Video Calibration Algorithms, NTIA Technical Report TR-08-433b, November 2007

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

   

Margaret H. Pinson; Stephen Wolf; Robert B. Stafford, "Video Performance Requirements for Tactical Video Applications," 2007 IEEE Conference on Technologies for Homeland Security, pp.85-90, 16-17 May 2007 doi: 10.1109/THS.2007.370025

The Public Safety Statement of Requirements (PS-SoR) for Communications and Interoperability focuses on the needs of first responders to communicate and share information as authorized, when it is needed, where it is needed, and in a mode or form tha...

   

Stephen Wolf; Margaret H. Pinson, "Application of the NTIA General Video Quality Metric (VQM) to HDTV Quality Monitoring," Third International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM-07), Scottsdale, Arizona, January 25-26, 2007

This paper summarizes results from an experiment whose goal was to assess whether the NTIA General Video Quality Metric (VQM) is an acceptable objective metric for measuring High Definition TV (HDTV) video quality. The HDTV subjective test that was p...

   

Margaret H. Pinson; Stephen Wolf, In-Service Video Quality Metric (IVQM) User’s Manual, NTIA Handbook HB-06-434a, July 2006

The purpose of this handbook is to provide a user’s manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicatin...

   

Margaret H. Pinson; Stephen Wolf, Reduced Reference Video Calibration Algorithms, NTIA Technical Report TR-06-433a, July 2006

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

   

Margaret H. Pinson; Stephen Wolf, In-Service Video Quality Metric (IVQM) User’s Manual, NTIA Handbook HB-06-434, December 2005

The purpose of this handbook is to provide a user's manual for the in-service video quality metric (IVQM) tool. IVQM performs automated processing of live video signals. This program runs under the Windows XP® operating system on two PCs communicatin...

   

Mark A. McFarland; Margaret H. Pinson; Stephen Wolf, Batch Video Quality Metric (BVQM) User’s Manual, NTIA Handbook HB-06-441a, December 2005

This handbook provides a user’s manual for the batch video quality metric (BVQM) tool. BVQM runs under the Windows XP® operating system. BVQM performs objective automated quality assessments of processed video clip batches (i.e., as output by a video...

   

Margaret H. Pinson; Stephen Wolf, Reduced Reference Video Calibration Algorithms, NTIA Technical Report TR-06-433, October 2005

This report describes four Reduced Reference (RR) video calibration algorithms of low computational complexity. RR methods are useful for performing end-to-end in-service video quality measurements since these methods utilize a low bandwidth network ...

   

Margaret Pinson; Stephen Wolf, "Low Bandwidth Reduced Reference Video Quality Monitoring System," First International Workshop on Video Processing and Quality Metrics for Consumer Electronics, Scottsdale, Arizona, January 23-25, 2005.

This paper presents a new reduced reference (RR) video quality monitoring system that utilizes less than 10 kbits/s of reference information from the source video stream. This new video quality monitoring system utilizes feature extraction techniques...

   

Margaret H. Pinson; Stephen Wolf, Video scaling estimation technique, NTIA Technical Memo TM-05-417, January 2005

Digital video compression algorithms are being deployed that spatially stretch or shrink the video picture. Although small changes in spatial scaling are not usually noticeable to viewers, objective video quality measurement systems may be adversely ...

   

Margaret Pinson; Stephen Wolf, "A new standardized method for objectively measuring video quality," IEEE Transactions on Broadcasting, vol.50, no.3, pp. 312-322, Sept. 2004 doi: 10.1109/TBC.2004.834028

The National Telecommunications and Information Administration (NTIA) General Model for estimating video quality and its associated calibration techniques were independently evaluated by the Video Quality Experts Group (VQEG) in their Phase II Full R...

   

Margaret H. Pinson; Stephen Wolf, The impact of monitor resolution and type on subjective video quality testing, NTIA Technical Memo TM-04-412, March 2004

This memorandum compares subjective video quality test results from a professional cathode ray tube (CRT) television monitor with that of a consumer liquid crystal display (LCD) video phone monitor. The CRT monitor supported the full ITU–R Recommenda...

   

Michael H. Brill; Jeffrey Lubin; Pierre Costa; Stephen Wolf; John Pearson, "Accuracy and cross-calibration of video quality metrics: new methods from ATIS/T1A1," Signal Processing: Image Communication, Vol. 19, Feb. 2004, pp. 101-107

Video quality metrics (VQMs) have often been evaluated and compared using simple measures of correlation to subjective mean opinion scores from panels of observers. However, this approach does not fully take into account the variability implicit in t...

   

Stephen Wolf, Color correction matrix for digital still and video imaging systems, NTIA Technical Memo TM-04-406, December 2003

This document discusses a method for correcting inaccurate color output by digital still and video imaging systems. The method uses a known reference image together with a least–squares algorithm to estimate the optimal color channel mixing matrix th...

   

Margaret Pinson; Stephen Wolf, "Comparing Subjective Video Quality Testing Methodologies," SPIE Video Communications and Image Processing Conference, Lugano, Switzerland, July 2003.

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. Some of these test methods are double stimulus where viewers rate the qu...

   

Margaret Pinson; Stephen Wolf, "An Objective Method for Combining Multiple Subjective Data Sets," SPIE Video Communications and Image Processing Conference, Lugano, Switzerland, July 2003.

International recommendations for subjective video quality assessment (e.g., ITU-R BT.500-11) include specifications for how to perform many different types of subjective tests. In addition to displaying the video sequences in different ways, subject...

   

Stephen D. Voran, An iterated nested least-squares algorithm for fitting multiple data sets, NTIA Technical Memo TM-03-397, October 2002

A multiple data set fitting problem often arises in conjunction with the development of objective estimators of perceived audio or video quality. In such development work, we often seek the best linear relationship between a set of objective audio or...

   

Stephen Wolf; Margaret Pinson, Video Quality Measurement Techniques, NTIA Technical Report TR-02-392, June 2002

Objective metrics for measuring digital video performance are required by Government and industry for specification of system performance requirements, comparison of competing service offerings, service level agreements, network maintenance, and opti...

   

Margaret Pinson; Stephen Wolf, Video Quality Measurement User’s Manual, NTIA Handbook HB-02-01, February 2002

The purpose of this handbook is to provide a user’s manual for the video quality metric (VQM) tool. The VQM software tool performs automated batch processing of video files. Program VQM runs under the UNIX operating system and uses a control file to ...

   

Wael Ashmawi; Roch Guerin; Stephen Wolf; Margaret Pinson, "On the Impact of Policing and Rate Guarantees in Diff-Serv Networks: A Video Streaming Application Perspective," Association for Computing Machinery, Special Interest Group on Data Communications (SIGCOMM 2001), pp. 83-95, August 27-31, 2001, San Diego, CA

Over the past few years, there have been a number of proposals aimed at introducing different levels of service in the Internet. One of the more recent proposals is the Differentiated Services (Diff-Serv) architecture, and in this paper we explore ho...

   

Stephen Wolf; Margaret H. Pinson, "The Relationship Between Performance and Spatial-Temporal Region Size for Reduced-Reference, In-Service Video Quality Monitoring Systems," Proc. of 5th World Multiconference on Systemics, Cybernetics and Informatics (ISAS-SCI 2001), Orlando, FL, July 22-25, 2001.

This paper presents objective-to-subjective correlation results for a reduced-reference, in-service, video quality monitoring system. This reduced-reference system utilizes quality parameters that are computed by comparing features extracted from spa...

   

Stephen Wolf; Margaret H. Pinson, "Spatial-Temporal Distortion Metrics for In-Service Quality Monitoring of Any Digital Video System," SPIE International Symposium on Voice, Video, and Data Communications, Boston, MA, September 11-22, 1999.

Many organizations have focused on developing digital video quality metrics which produce results that accurately emulate subjective responses. However, to be widely applicable a metric must also work over a wide range of quality, and be useful for i...

   

Maragaret H. Pinson; Stephen Wolf, "Medium Bandwidth Techniques for Estimating Temporal Delays Between Input and Output Video Sequences," ANSI T1A1 Contribution T1A1.5/99-205, May 1999

This contribution presents new techniques for estimating temporal delays between input and output video streams from video teleconferencing systems. The discussion in this contribution considers "variable alignment," defined as an alignment process t...

   

Margaret H. Pinson; Stephen Wolf, "Low Bandwidth Techniques for Estimating Temporal Delays Between Input and Output Video Sequences," ANSI T1A1 Contribution T1A1.5/99-204, May 1999.

This contribution presents a new technique for estimating temporal delays between input and output video streams from video teleconferencing systems. The discussion in this contribution is limited to "constant alignment," defined as an alignment proc...

   

Stephen Wolf; Margaret H. Pinson, "In-Service Performance Metrics for MPEG-2 Video Systems," Proc. Made to Measure 98—Measurement Techniques of the Digital Age Technical Seminar, Montreux, Switzerland, November 12-13, 1998, International Academy of Broadcasting (IAB), ITU and Technical University of Braunschweig.

With the advent of new digital video systems that utilize compression to achieve a savings in transmission or storage bandwidth, the quality of the received output video can be dependent not only upon the inherent spatial and temporal information con...

   

Charles Fenimore; John Libert; Stephen Wolf, "Perceptual Effects of Noise in Digital Video Compression," 140th SMPTE Technical Conference, Pasadena, CA, October 28-31, 1998.

We present results of subjective viewer assessment of video quality of MPEG-2 compressed video containing wide-band Gaussian noise. The video test sequences consisted of seven test clips (both classical and new materials) to which noise with a peak-s...

   

Coleen Jones; D. J. Atkinson, "Development of Opinion-Based Audiovisual Quality Models for Desktop Video-Teleconferencing," 1998 Sixth International Workshop on Quality of Service (IWQoS 98), pp.196-203, Napa, California, May 18-20, 1998. doi: 10.1109/IWQOS.1998.675239

This paper discusses the analysis of an audiovisual desktop video-teleconferencing subjective experiment conducted at the Institute for Telecommunication Sciences. Objective models of the individual audio and video quality are presented. Also discuss...

   

Stephen Wolf; Margaret H. Pinson; Arthur A. Webster; Gregory W. Cermak; E. Paterson Tweedy, "Validating Objective Measures of MPEG Video Quality," SMPTE Journal, vol. 107, no. 4, pp. 226-235, New York City, April 1998

In 1996, the American National Standards Institute (ANSI) adopted ANSI T1.801.03, which presents a number of new objective video quality metrics for quantifying the effects of digital compression and transmission impairments. The measurements in ANSI...

   

Stephen Wolf, "Measuring the End-to-End Performance of Digital Video Systems," IEEE Transactions on Broadcasting, Vol. 43, No. 3, pp. 320-328, September 1997. doi: 10.1109/11.632940

Significant research and development efforts by industry and government laboratories were brought to fruition in 1996 with the approval of American National Standard (ANSI) T1.801.03 entitled "American National Standard for Telecommunications - Digit...

   

Stephen Wolf; Margaret Pinson; Arthur Webster; Greg Cermak; E. Paterson Tweedy, "Objective and Subjective Measures of MPEG Video Quality," ANSI T1A1 Contribution T1A1.5/96-121, October 28, 1996.

Presents an in-depth analysis and discussion of the results from applying the ANSI T1.801.03-1996 objective video quality metrics to subjectively rated MPEG-1 and MPEG-2 video test scenes. The objective metrics presented in ANSI T1.801.03-1996 (Ameri...

   

Presents a computerized search method for determining the gain, level offset, active video shift, and video delay of a digital video transmission channel. The method uses digitized input and output NTSC video fields that have been time tagged with SM...

   

D. J. Atkinson, Exploring B-ISDN performance: Selected experiments and results, NTIA Technical Report TR-96-329, April 1996

This report describes experiments conducted to explore the user-information transfer performance of the broadband integrated services digital network (B-ISDN), the emerging infrastructure for the global information age. These performance experiments ...

   

Stephen Wolf, "Tape Duplication and Traceability Process for ANSI T1.801.02-1996 (Terms and Definitions)," ANSI T1A1 Contribution T1A1.5/95-158, December 14, 1995.

Recommends a process to be used by ANSI for duplication and traceability of video tapes for ANSI T1.801.02-1996 (American National Standard for Telecommunications - Digital Transport of Video Teleconferencing/Video Telephony Signals - Performance Ter...

   

Stephen Wolf, "Tape Duplication and Traceability Process for ANSI T1.801.01-1996 (Test Scenes)," ANSI T1A1 Contribution T1A1.5/95-157, December 14, 1995.

Recommends a process to be used by ANSI for duplication and traceability of video tapes for ANSI T1.801.01-1996 (American National Standard for Telecommunications - Digital Transport of Video Teleconferencing/Video Telephony Signals - Video Test Scen...

   

Stephen Wolf, "Field Vs. Frame Calculations for Visual Channel Delay," ANSI T1A1 Contribution T1A1.5/95-152, December 11, 1995.

This contribution recommends that calculations of visual channel delay be based on fields rather than frames for an interlaced video system such as NTSC. Using fields rather than frames will double the resolution of the measurement without significan...

   

Stephen Wolf, "An Update on ITS Video Quality Parameters," ANSI T1A1 Contribution T1A1.5/92-135, July 13, 1995.

Presents a summary of efforts to make the objective video quality metrics more efficient in terms of computation and transmission bandwidth. Objective to subjective correlation results are presented for the subjectively rated data set given in the 19...

   

Stephen Wolf, "An Analysis Technique for Detecting Temporal Edge Noise," ANSI T1A1 Contribution T1A1.5/95-105, January 9, 1995.

Presents an objective analysis technique that can be used for detecting temporal edge noise in an output video sequence that has been digitally compressed. Temporal edge noise is defined in ANSI T1.801.02-1996 as "A form of edge busyness characterize...

   

Dwight Melcher; Stephen Wolf, "Objective Measures for Detecting Digital Tiling," ANSI T1A1 Contribution T1A1.5/95-104, January 9, 1995.

Describes an algorithm for (1) quantifying the spatial gradients or edges in an image as a function of angle or orientation, (2) extracting low bandwidth features from these input and corresponding output spatial gradient images, and (3) deriving obj...

   

Stephen Wolf, "An In-depth Analysis of the P6 Lost Motion Energy Parameter," ANSI T1A1 Contribution T1A1.5/95-103, January 9, 1995.

Provides an in-depth analysis of the behavior of the P6 video performance parameter, a parameter the measures the motion energy that is lost when a video scene is transmitted through a digital video compression system. Objective to subjective correla...

   

This contribution discusses the advantages and disadvantages of two methods for obtaining a composite rating of Hypothetical Reference Circuit (HRC) performance for an ensemble of source material. The term HRC refers to a specific realization of a vi...

   

Arthur Webster, "Two Criteria for Video Test Scene Selection," ITU-T (Telecommunications Standardization Sector), Temporary Document 35-E, Working Party 2, Study Group 12, Question 22, Geneva, December 2-5, 1994.

Describes two objective measurements that can be used to quantify the spatial and temporal information content of a video test scene. Test scene coding difficulty (and hence video quality) often depend on the spatial and temporal information complexi...

   

Coleen Jones; Ned Crow; Stephen Wolf; Arthur Webster, "Analysis of T1A1.5 Subjective and Objective Test Data," ANSI T1A1 Contribution T1A1.5/94-152, October 3, 1994.

Presents detailed intra-laboratory (within laboratory) and inter-laboratory (between laboratories) analysis of variance (ANOVA) results for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 ...

   

Coleen Jones; Ned Crow, "Status of NTIA's Analysis of Subjective Data," ANSI T1A1 Contribution T1A1.5/94-133, July 18, 1994.

Presents results from the analysis of variance (ANOVA) of the subjective opinion scores for the T1A1 subjective video experiment that contained 625 mean opinion scores (25 test scenes passed through 25 different video transmission systems that ranged...

   

Coleen Jones, "The Correlation of Traditional Bandwidth and Signal-to-Noise Ratio Parameters to Subjective Data," ANSI T1A1 Contribution T1A1.5/94-132, July 18, 1994.

This contribution presents objective to subjective correlation results for the traditional analog measurements of bandwidth (i.e., frequency response) and signal-to-noise ratio (SNR) for the T1A1 subjective video experiment that contained 625 mean op...

   

Edwin L. Crow, "Methods for Analysis of Inter-laboratory Video Performance Standard Subjective Test Data," ANSI T1A1 Contribution T1A1.5/94-128, March 28, 1994.

Presents in detail the ANOVA methods for analysis of the subjective T1A1 video test data, especially with regard to any systematic effect a laboratory may have on the Mean Opinion Score (MOS) of any given Hypothetical Reference Circuit (HRC). The ter...

   

Stephen Wolf; Margaret Pinson, "Corrections and Extensions to T1A1.5/93-152," ANSI T1A1 Contribution T1A1.5/94-110, January 17, 1994.

Contribution T1A1.5/93-152 summarized the methods of measurement for objective video quality parameters based on the Sobel-filtered image and the motion difference image that were submitted prior to conducting the T1A1 subjective experiment (this exp...

   

Arthur Webster; Coleen Jones, "VTC Hypothetical Reference Circuit Signal to Noise Ratio Measurements," ANSI T1A1 Contribution T1A1.5/94-103, January 17, 1994.

Discusses the method of measurement used to obtain the signal to noise ratio (SNR) values for each of the twenty-five video systems in the T1A1 subjective video experiment. The 25 video systems ranged in bit rate from 64 kb/sec to 45 Mb/sec. The sign...

   

Coleen Jones, "VTC Hypothetical Reference Circuit Bandwidth Measurements," ANSI T1A1 Contribution T1A1.5 / 94-102, January 17, 1994

Discusses the method of measurement used to obtain the frequency response or bandwidth values for each of the twenty-five video systems in the T1A1 subjective video experiment. The 25 video systems ranged in bit rate from 64 kb/sec to 45 Mb/sec. The ...

   

Neal Seitz; Stephen Wolf; Stephen Voran; Randy Bloomfield, "User-Oriented Measures of Telecommunication Quality," IEEE Communications Magazine, vol.32, no.1, pp.56-66, Jan. 1994 doi: 10.1109/35.249792

Discusses the standardization of user-oriented, technology-independent measures of telecommunication service quality, including voice, video, and data performance standards. Standards committee work has progressed in three broad phases. In the first ...

   

Arthur Webster, "Methods of Measurement for Two Objective Video Quality Parameters Based on the Fourier Transform," ANSI T1A1 Contribution T1A1.5/93-153, November 8, 1993.

Summarizes detailed methods of measurement for two objective video quality parameters based on the Fourier transform image. This contribution was submitted to ANSI T1A1 prior to conducting the T1A1 subjective experiment (this experiment collected 625...

   

Stephen Wolf; Margaret Pinson; Coleen Jones; Arthur Webster, "A Summary of Methods of Measurement for Objective Video Quality Parameters Based on the Sobel Filtered Image and the Motion Difference Image," ANSI T1A1 Contribution T1A1.5/93-152, November 8, 1993.

Summarizes detailed methods of measurement for objective video quality parameters based on the Sobel-filtered image and the motion difference image. This contribution was submitted to ANSI T1A1 prior to conducting the T1A1 subjective experiment (this...

   

Coleen Jones, "Real-Time Measurement System for ITS Video Quality Parameters," ANSI T1A1 Contribution T1A1.5/93-105, August 9, 1993.

Discusses a real-time personal computer-based system for characterizing spatial (e.g., blurring) and temporal (e.g., jerky motion) distortions in a compressed digital video system. Comparisons with a high quality non real-time laboratory version of t...

   

Stephen Voran; Stephen Wolf, "An Objective Technique for Assessing Video Impairments," IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, vol.1, no., pp.161-165 vol.1, 19-21 May 1993 doi: 10.1109/PACRIM.1993.407197

Measurements that quantify perceptual video attributes in both the spatial and temporal domains are extracted from the video that has undergone digital coding and compression. These measurements are then used to compute a single score that quantifies...

   

Arthur A. Webster; Coleen T. Jones; Margaret H. Pinson; Stephen D. Voran; Stephen Wolf, "An objective video quality assessment system based on human perception," SPIE: Human vision, visual processing and digital display IV, 1-4 February 1993, San Jose California

The Institute for Telecommunication Sciences (ITS) has developed an objective video quality assessment system that emulates human perception. The system returns results that agree closely with quality judgements made by a large panel of viewers. Such...

   

Coleen Jones; Stephen Wolf; Margaret Pinson, "Preliminary Results of One-Way Video Delay Measurement Algorithms," ANSI T1A1 Contribution T1A1.5/92-139, July 13, 1992.

Out-of-service measurements can be used to obtain a baseline video system delay. However, in many digital video systems, video delay is dynamic and dependent upon the scene content. This contribution describes an objective measure of video delay that...

   

Stephen Wolf, "An Automated Technique for Measuring Transmitted Frame Rate (TFR) and Average Frame Rate (AFR)," ANSI T1A1 Contribution T1A1.5/92-138, July 13, 1992.

The motion difference images of the original and degraded video sequences is used to dynamically measure the Transmitted Frame Rate (TFR) and the Average Frame Rate (AFR) of a digital video system codec (coder-decoder). The TFR measurement algorithm ...

   

Stephen Voran, "The Effect of Multiple Scenes on Objective Video Quality Assessment," ANSI T1A1 Contribution T1A1.5/92-136, July 13, 1992.

Presents objective to subjective correlation results when the scores from multiple video test scenes are averaged. It is shown that the prediction errors of the objective video quality model are reduced, yielding objective measurement results that mo...

   

Stephen Voran; Stephen Wolf, "The Development and Evaluation of an Objective Video Quality Assessment System that Emulates Human Viewing Panels," International Broadcasting Convention (IBC), pp.504-508, 3-7 Jul 1992.

Discusses the approach used and the research conducted to develop an objective video quality assessment system that emulates human perception. The system returns results that agree closely with quality judgments made by a large panel of viewers for t...

   

Stephen Voran; Stephen Wolf, "Objective Measures of Video Impairment: Analysis of 128 Scenes," ATIS T1 Contribution T1A1.5/92-112, March 25, 1992

ATIS Committee T1 contribution to standards project “Analog Interface Performance Specifications for Digital Video Teleconferencing/Video Telephony Service and DS3 Television”

   

Arthur Webster; Stephen Wolf, "Spatial and Temporal Information Measures for Video Quality," ANSI T1Q1 Contribution T1Q1.5/92-113, January 22, 1992.

Presents an overview of metrics that can be used to quantify the amount of spatial and temporal information in a video sequence. By applying the metrics to the input and output video, the amount of spatial and temporal information that is lost by the...

   

Stephen Voran, "The development of objective video quality measures that emulate human perception," Global Telecommunications Conference, 1991 (GLOBECOM '91. Countdown to the New Millennium. Featuring a Mini-Theme on: Personal Communications Services), pp.1776-1781 vol.3, 2-5 Dec 1991

Discusses research efforts to derive objective measures of video quality that emulate human perception. The derivation of these metrics involves the following steps: (1) a set of test scenes in selected and distorted, (2) a set of candidate objective...

   

Presents preliminary results on the objective to subjective quality correlation for a set of 5 test scenes passed through three systems (NTSC, VHS, and DS1 MPEG coding). One objective metric that correlates well with subjective quality quantifies the...

   

Stephen Voran; Stephen Wolf, "Motion-Still Segmentation Algorithm for VTC/VT Objective Quality Assessment," ANSI T1Q1 Contribution T1Q1.5/91-110, January 22, 1991.

The ability of the human eye to resolve detail in a video scene is related to how much motion is present at the point of focus and whether or not the eye can track the motion. Thus, stationary portions of the video scene can be resolved in great deta...

   

Stephen Wolf, Features for automated quality assessment of digitally transmitted video, NTIA Technical Report TR-90-264, June 1990

This report describes an automated method of video quality assessment based on extraction and classification of features from sampled input and output video. The first subsystem of the automated video quality measurement system is the feature extract...