bp streaming

bp streaming - Transcript

best practices

going witH tHe flow best practices for mastering multimedia streaming
fall 2008

Are you sure that what you see is what they get
With widespread distribution of broadband Internet access multimedia has become a dynamic part of the Web experience According to Pew Internet Research more than 50 of Internet users have watched a video online a number that jumps to 75 when you look at the key media market of people under thirty In Stat a leading communications research firm recently predicted that the online video market will grow to over 4 5 billion by 2012 Online video and audio can open new streams of revenue or new ways of connecting with customers and other constituents But delivering multimedia online represents an investment that goes beyond initial production costs It s not enough to make a video you need to be sure that the content you created is the content your visitors actually see without delays or disruptions that disappoint And undermine the value of your investment Protecting the quality of multimedia customer experiences is what this white paper is all about We ll discuss the importance of identifying your most significant multimedia streams and quantifying their performance We ll examine options for making improvements both in house and through third party vendors Finally we ll look at ways you can establish performance monitoring strategies that verify the quality of your multimedia experience and validate the investments you make to deliver it

best practices

going with the flow best practices for mastering multimedia streaming

i mastery begins witH measurement
When it comes to multimedia most of the business attention focuses on production creating the content to be offered online Yet operations groups play the crucial role of connecting site visitors or customers to the content you have produced In addition to creating the original content business management must be attentive to three key areas identifying important streams establishing performance targets and comparing your performance to your competitors But IT and operations have equally important roles to fulfill defining the right technical delivery methods quantifying performance targets and ensuring adequate capacity

Identifying high value streams
All Web elements are not equal Your attention should be concentrated on those content streams that matter such as those that Drive revenue as in a pay per view model of business Fulfill contractual obligations like training seminars delivered as part of a service package Represent risk such as areas in which failure to delivery or to deliver with a minimum of errors could expose your company to lost revenues lost contracts or lost customers clients As management focuses on the most critical streams operations must examine the means for delivering them What s available in house Are additional investments required Does the organization need to consider third party vendor relationships to meet current technical demands or to expand capacity

Quantify your performance targets
You cannot reach what you do not aim for As a matter of policy your multimedia performance levels must be measured against an acceptable objective benchmark Establishing the benchmark should be a joint effort between management and operations with the former contributing business goals as the latter confirms realistic expectations for achievement and is responsible for the method of monitoring In our experience the minimum standard for delivery is 97 error free multimedia performance Anything less represents a risk of angry or lost customers Areas of concern include Up time the percentage of time your multimedia content is actually available for viewing Start time a measure of the amount of time it takes for your content to launch Rebuffer ratio the percentage of time lost to delays as customers wait for content to buffer and or rebuffer

Compare your measurements
Your customers expectations are also formed by their experiences at other online sites In addition to capturing your own performance measurements you need access to comparable measurements for similar content both those of your competitors and of industry leaders such as YouTube that set the standard for Web content delivery These comparative indications give you precious insight on where you stand with your Web users as an acceptable content provider as a champion who beats the averages or as an under performer who risks increased dissatisfaction and decreased revenues

2

Gomez Inc

10 Maguire Road Lexington MA 02421 www gomez com T 781 778 2700 F 781 778 2799

best practices

going with the flow best practices for mastering multimedia streaming

ii managing your tecHnology investments
Measuring Availability
Metric to Watch Availability Streams to Test Test important streams across multiple file formats and bitrates Frequency Test using a higher frequency every 15 minutes from each geography for a shorter test period 15 seconds Testing Locations Monitor from geographies that are important to you and or where you have infrastructure For instance if you have delivery infrastructure in North America and in Europe set up testing to access each location Review Period Review availability data as needed and report on availability metrics using a minimum of 24 hours worth of data

As your commitment to multimedia grows so too will your investment in technologies in house or with vendors to help you deliver your content effectively and consistently Without careful planning these investments can spiral out of control a hodge podge of ad hoc efforts engineered as short term solutions The alternative is a rational process that meets growing needs while validating the merits of each investment By taking the following steps you create a system of accountability as you improve your technology infrastructure

1 Establish a baseline
It s one of the cardinal rules of business never make changes in a vacuum Before you make any improvements take a snapshot of current streaming performance availability start time rebuffering ratios and error rates These measurements will give you a baseline against which you can compare future efforts and evaluate their worth

2 Create a testing methodology
Random measurements are useless You need a routine process that creates standard figures which can be meaningfully compared across time and across geographical regions Too much sampling is expensive and intrusive too little fails to provide insight In our experience we have found an optimal balance that works for most multimedia providers 60 second samples captured once per hour from each important location you want to serve Europe Asia US West Coast etc Review Gomez s testing recommendations in this section for best practices

3 Look for the differentials
One of your goals is consistency That s why it s important to look for the differentials the amount by which performance differs in different regions at different times of day or even in different seasons These differentials can help you identify the areas most in need of improvement After you make a technology upgrade or launch a new encoding method the differential between previous performance and post upgrade performance can reveal the relative value of your investment and whether or not it meets your expectations

4 Fix what you can then watch the numbers
With your testing methodology in place you can identify the areas that need work Start with the low hanging fruit the fixes you can make in house for short money Then compare the results to your baseline Look to answer questions like Does changing the file size accelerate start up times When you change the bit rate how does it affect rebuffering ratios and the number of errors Can you see improvement in your entire service area or are there variations in performance This act and measure approach can validate the approaches that work and help you precisely target performance areas that need more effort

3

best practices

going with the flow best practices for mastering multimedia streaming

5 Put vendor hardware to the test
Measuring Performance Quality
Metric to Watch Start up Time Rebuffer Ratio Streams to Test Test important streams across multiple file formats and bitrates Frequency Test every 60 minutes from each node and test each stream for 30 or 60 seconds depending on stream length Testing Locations Monitor from top geographies using a minimum of 4 locations Review Period Review start up time and rebuffer ratio on a daily basis and report on these metrics using 2 3 days worth of data

There s no shortage of vendors willing to sell you hardware to help you master increasing multimedia loads But can they fulfill their promises Before you commit put their hardware to the same tests that you ve applied to your current systems Can you see performance improvement And if so is it enough to justify the purchasing expense

6 Recognize your limits
Unless your business actually specializes in multimedia services or makes multimedia the very core of its business model you ll experience the law of diminishing returns the investment required to meet your multimedia delivery needs will be out of proportion to the business results you can expect In part this reflects the complexity of the problem a media universe with multiple media formats multiple video sizes PC browser versus PDA multiple viewer types and other variables too numerous to control For most online enterprises the investment in hardware and resources needed to meet all these media variables outstrips potential ROI But these same investments make sense to those who specialize in streaming services for multiple clients When your technology needs exceed your capabilities yet demand more than you re willing to spend on infrastructure it may be time to work with outside third party vendors

iii working more effectively witH tHird parties
Comparing CDNs
Metric to Watch Availability Start up Time Rebuffer Ratio Streams to Test Test important streams across multiple file formats and bitrates Set up baseline comparisons between the original delivery infrastructure and the CDN s Frequency Test every 30 minutes from each node and test each stream for 30 or 60 seconds depending on stream length Frequently expanding testing duration to 2 3 minutes is helpful as it may show additional issues Testing Locations Monitor from top geographies using 4 6 locations Review Period Review availability start up time and rebuffer ratio on a daily basis and report on these metrics using 10 14 days worth of data
4
Gomez Inc 10 Maguire Road Lexington MA 02421 www gomez com T 781 778 2700 F 781 778 2799

Third party vendors known as content delivery networks or CDNs can distribute your multimedia content for you or supplement your infrastructure to help you reach far flung regions in your overall viewer area Many CDNs have the technology you need But to justify your relationship with them you need to evaluate their performance to be sure they can solve your specific problem and to validate your investment Make meaningful comparisons Just as you would when you make infrastructure upgrades you need to take measurements that allow you compare performance levels including Against your baseline Test the CDN before making a commitment How do the performance figures compare to your previous baseline Against competing providers Shop around Compare CDNs against each other to seek optimal performance Chances are CDNs may be stronger in some regions rather than others you may want to hold multiple vendor contracts that apply each CDN to its region of greatest strength Against in house performance Even after you contract with a vendor test the CDN against your servers What s the performance gain Again watch the differential the degree of improvement provided by the CDN Over time does the CDN continue to provide a significant improvement in performance Or is the differential shrinking as the months pass

b e sw H ir a cp a c e r t p te tip s

t oing with the flow best practices for mastering ghe 8 immutable laws of web experience Quality multimedia streaming

Measuring SLAs
Metric to Watch Availability Start up Time Rebuffer Ratio Streams to Test Test important streams across multiple file formats and bitrates that are delivered via original delivery infrastructure and the CDN s Frequency Test every 30 minutes from each node and test each stream for 30 or 60 seconds depending on stream length Testing Locations Monitor from top geographies using 4 6 locations Review Period Review availability start up time and rebuffer ratio every day and report on these metrics on a monthly basis

Watch the delivery sources You need to know precisely where your multimedia content is coming from In general there are two areas of concern CDN sources You may have contracted a given CDN by virtue of its proximity to an important market such as southeast Asia But is the CDN really distributing your videos from Singapore or are they sending them via California You can establish monitoring that tracks the true sources for you Third party content Even though advertising and consumer generated content doesn t come from you the performance of that content reflects on your site business and brand Apply your testing methodology to third party streamed content to be sure it s consistent with your site and with your visitors expectations Create workable service level agreements SLAs The service level agreement articulates the expectations of the client and the vendor s promise of performance Yet too many SLAs aren t worth the paper they re written on generating dissatisfaction frustration and in the worst cases litigation The best way to prevent dissatisfaction is to arrive at an SLA by way of mutual agreement Together you and the CDN can craft an agreement that is Quantifiable Every expectation and promise must be assigned a hard number That way you replace subjective understandings with objective goals both parties can see and measure Measurable If it can t be measured it can t be monitored Your agreement must be predicated on actions that can be measured against baselines and expectations Achievable Unrealistic goals suit no one s purposes Set objectives that all parties can reasonably expect to accomplish Flexible Online and multimedia technologies are evolving at a breakneck speed Be sure your SLA can accommodate new tools and new opportunities In addition to arriving at a mutual understanding of goals you and your vendor should share common ground on the execution of the SLA Be sure to specify Data set What will be measured and using which numbers The figures by which your CDN agrees to be measured must be consistent with the metrics you use to monitor in house performance this is the only way to identify meaningful differentials that reveal value Testing methodology Define what will be measured how it will be measured and from what point or points on the distribution path it will be measured Frequency How often will the tests be conducted and reviewed Finally consider using an independent third party to test and monitor your SLAs As an outside resource an independent tester can provide unbiased data both parties can trust

5

best practices

going with the flow best practices for mastering multimedia streaming

iv conclusion validation cHecklist
Ultimately your content is only as strong as your users ability to access it without error or frustration Use the following list of questions as a resource for assessing the strength of your multimedia stream YES
1

NO

MAYBE

Can customers reach your high value multimedia streams Have you set performance targets for your multimedia content delivery Can you measure multimedia delivery metrics e g up time and start time and quality metrics e g rebuffer ratios and error rates Are you able to compare streaming performance across different regions and times of day Have you established a baseline of current performance Do you have a consistent testing methodology Can you measure the differentials in performance over time for different stream Have you made and measured all the in house fixes you can accomplish yourself Are you able to objectively test competing hardware options Can you compare CDNs to identify top performers Can you confirm your distribution sources Does the performance of third party content confirm to your Web standards Are your service level agreements SLAs quantifiable measurable achievable and flexible Have you and your CDNs agreed to common data sets and testing methodologies Can you and your CDNs rely on objective data from a qualified independent party

2

3

4

5

6

7

8

9

10

11

12

13

14

15

If you re not able to answer yes to at least 12 of these 15 questions you may want to consider further assistance The experts at Gomez can help you address these and other questions crucial the success of your multimedia content You may contact a Gomez expert at no further obligation by calling 1 877 373 6732 or writing to answers gomez com

Gomez Incorporated 10 Maguire Rd Bldg 3 Suite 330 Lexington MA 02421 General 1 781 778 2700 Sales USA 1 781 778 2760 UK 44 0 207 554 5154 Germany 49 0 40 53299 167 www gomez com

2008 by Gomez Inc All rights reserved Gomez and the Gomez logo are registered trademarks of Gomez Inc All other trademarks and service marks are the property of their respective owners
Specifications may change Consult the technical documentation for the most current information Some features described are extra cost options Ask your sales representative for details

bp streaming 2008 001