How to Build Unbreakable Live Streams at Scale
What are the best workflows to develop in order to deliver reliable live streams at scale? 首席执行官亚当·米勒 游牧的技术, David Hassoun, Chief Technologist, 杜比 Cloud Media Solutions, 杜比.io, Peter Wharton, Chief Strategy & 云长, TAG视频系统老科里·史密斯(Corey Smith). Director, Advanced Production Technology, CBS体育数字频道, 派拉蒙 discuss how their organizations work to provide seamless live scaled streaming in this panel from 流媒体 West 2022.
“Live event streams can break down anywhere along the line,主持人说。, Eric Schumacher-Rasmussen, 椅子, 流媒体 会议和CMO， id3as. He asks Adam Miller of 游牧的技术 to kick off the discussion. “What do you see as the key touch points to focus on when building a workflow for reliability?”
Miller emphasizes that there are countless possible failure touch points with live streaming at scale. He says that at Nomad they try to break these possibilities down in an organized chart that narrows the categories to focus on into three parts. “What is the cost to make this redundancy better?他说. “What is the effort level to make it better? [And] where do you put your budget and time and energy? 当你这样做的时候, you're going to find that out of those hundred breakage points, probably about six really become most prominent. And that's going to be where you can put your energy to help increase the reliability rather than someone saying, ‘嘿, 我需要多个编码器.’ Well, put the energy towards where it's going to matter the most. So that's what we normally do when we start looking at redundancy.”
Peter Wharton of 标签和 notes that most people outside of the industry have little idea of the vast complexity of these live streaming systems. “You need to monitor the whole thing, especially if you're doing ad hoc building of systems or you're setting up a live system on demand,他说. “Because then you have all these moving parts and now, you're just building something instantaneously. So you have to make sure all those moving parts are working. It's not something you've had running for months, and you know how it works. But at the same time, every point in that workflow has a different value in the workflow.”
Wharton says that it’s a matter of adjusting for priorities. “As you go further down that food chain, every touch point which is going to impact the viewer is impacting fewer and fewer viewers,他说. “So, 因此, you have to also make sure that you are monitoring in a way that you're actually adjusting the cost of your monitoring to the value of the content at each point of the workflow. And that's a challenge to get all that right but still make sure it works everywhere. Because you can't spend the same amount of money for some edge of a CDN that affects one region that you do at the core.”
杜比的大卫·哈桑.io highlights the crucial need for validation testing of systems. “Every one of those key breakpoints, you really need to be able to hammer that, and you're going to have assumptions of how it can break, and those are the ones that you can at least test,他说. “But also try to look beyond what you didn't expect and understand what your different failure scenarios are, how you're going to transition, and make sure that that is smooth. It needs to be really worked through. It becomes muscle memory into all the teams that this is going to affect and what's going to happen in those situations.”
Miller mentions what he calls the “golden rule” of leaving what is working alone and not over-tinkering. “When it comes close to time to deploy, don't touch anything!他说. “People forget that, and they think, “Oh, let's just change out this encoder at the last minute.’ If you're trying to reliably distribute something, touch as little as possible. And if you're going to do it ten times, don't touch it at all. Build it once, and then just leave it and reuse it ten times. I find that a lot of people forget that golden rule of ‘Don't be changing things two minutes before.’”
Corey Smith of CBS Sports and 派拉蒙 stresses that “There’s a real art to figuring out both the telemetry coming in from the outside, [and] also how the customers experience the event.” He points out that CDNs are not always the most accommodating when stress-testing all possible failure points.
“当我在 Xbox,他说, “we were doing a lot of things to scale large customer events, whether it's the E3 Keynote or other Mars Moon Lander or whatever that happened to be on the console that particular day. But we took a lot of pride in actually testing to failure. 但是当你去 Akamai 或者是a 聚光灯下 or some other CDN provider and say, ‘嘿, I'm going to stress test my network out. Can you help support two and a half to three terabytes per second of traffic because I want to scale to two and a half million concurrence? How do you tell a CDN provider that's where you want to test to? They're going to laugh you out the door and say we can't absorb that on our network. But a lot of it is knowing where your traffic's going, being multi-CDN agnostic or being agnostic to a single CDN provider, but also getting feedback and telemetry from your clients so now you can make actual intelligent traffic decisions in near real time to what your customers are actually seeing. So if you are pushing a bit rate of 10 megabits plus, so you're doing 1080p Plus video at high quality, you don't exactly own all of the edge ecosystem that you're actually deploying to. You just own the video player tech, so to speak. You kind of want to get that feedback so you can say, ‘嘿, this city and provider in this particular part of the world is not doing so well and this is the CDN provider we need to start shedding traffic to. You have to build those kind of telemetry systems into the actual base of the application because you can't just flip it off and flip it back on during a live event. You have to be able to ebb and flow your traffic across the global internet and be able to do it seamlessly.”
Learn more about live streaming at scale at 流媒体东部2023.
相关文章
How do you maintain five-nines uptime for tentpole event streams reaching hundreds of thousands or millions of users? CBS体育数字频道's Corey Smith, Bulldog DM's John Petrocelli, and EdgeNext's Joshua Johnson offer tips and best practices for successfully scaling up your streams in this clip from 流媒体 Connect 2023.
2023年5月10日
How can live video help advertisers meet the challenges of a media consumption ecosystem perpetually in flux, 比如剪线, 品牌安全, 进行, 和更多的? BulldogDM's John Petrocelli, who works with many of the biggest brands in the business, and CBS体育数字频道's Corey Smith discuss how to make streams work for brands so the streams thrive and the brands and viewers keep coming back in this clip from 流媒体 Connect 2023.
2023年3月31日
End-to-end workflows for live streaming at scale are complex affairs, with troubleshooting challenges at each stage when delivery breaks down. An expert panel from 派拉蒙, Amagi, 标签和, and 游牧的技术 discusses key best practices for troubleshooting large-scale streams when the pressure is on.
2023年2月22日
How do you meet the challenges of monitoring stream performance and quality for dozens or thousands of live streams at once? 派拉蒙专家, 杜比, TAG视频系统, and 游牧的技术 discuss multistream monitoring tech and strategies in this panel from 流媒体 West 2022.
2023年2月15日
How much streaming reliability and quality are worth trading for ultra-low latency, and when is one at a premium over the others? Amagi的Brian Ring, 杜比,大卫·哈桑报道, 游牧的技术' Adam Miller, 派拉蒙的科里·史密斯, and Norsk CMO Eric Schumacher-Rasmussen discuss in this panel from 流媒体 West 2022.
2023年1月30日
What are the most dynamic approaches for localizing live event streams? Dan Turow of Evertz and Marisa Elizondo of fuboTV talk about the ways their organizations are working to best integrate local and user-preference-based content and experiences into live event streams.
11月15日2022
Companies and Suppliers Mentioned