MediaCentral : Engineers Peek

Following on from an earlier post about the fundamentals of MediaCentral I wanted to write a similar short Engineering post about the configuration and workings of the platform. If you know very little about MediaCentral then perhaps read my earlier post but I have put some background content here too.

This isn't a complete guide to MediaCentral and it isn't a 'how to' guide for building a system or maintain it, its just an overview of the 'Platform' and the very basics of how it works.

By the way, if you want to officially learn about building and maintaining a system then it's the WG453 ACSR Avid MediaCentral Platform course you should be attending.

Research and Manuals

The devil is in the detail, so finding the detail this is where I personally like to start. 

Avid has pretty good documentation when it comes to low down technical resources. All the release notes and configuration guides can be found on their KB. There are a number of guides under the MediaCentral Documentation page, but a good starting point, on any Avid product, is to look at the Release Notes and then dive into the supporting documentation from there. 

  Avid Knowledge Base is always a good place to start.

The only guide that serves as an introduction to MediaCentral is the MCS Concepts and Clustering Guide Chapter 2, but even this can be hard going for new users. To find simple information to introduce you to MediaCentral and gain a basic understanding of the system functions and how works, is a little more tricky. A straight forward Google of "how does Avid MediaCentral work?" sadly doesn't return all that much... 

I'm lucky (?) enough to have worked on a number of MediaCentral systems and so thought I'd try and give short a overview, using what little knowledge I have, combined with Avid documentation and hopefully highlight the most salient points.

Let's start with the fundamentals...

What does the MediaCentral give you?

MediaCentral is the gateway for remote users to access your Avid Production and News environments. Its key role is to allow users to communicate with Avid Interplay PAM or MAMs and/or iNEWS systems through a web interface. It allows this function by running background services that communicate between the various systems as well as supply the ability to create proxy media on the fly for the remote users to use. 

MediaCentral extends the ability of your production environment beyond your physical environment. Remote journalists and field editors have access to centrally held content such as video, stories and scripts through an Internet connection anywhere they need to work. They can edit video content and create new stories and scripts in the field or in the office from there desktop PC, no additional software needed. Other co-workers who wouldn't normally have the ability to access this content can now browse and log material, send material to an archive or publish content to the web, all through a single web interface. 

All the above can be achieved through a web interface supplied by MediaCentral

How does MediaCentral fit into my system?

MediaCentral, in its most simple form, is a server that runs the MediaCentral Platform software or rather services. So to deploy MediaCentral we simply need a server to load the software on to. This server is referred to as the MediaCentral Platform Services Server or MCS Server. However this server has a number of criteria that needs to be met. But before we start looking at the physical needs of the server we need to understand the environment where it will be used and in turn, by who.

MediaCentral can be used in a number of ways and it's functionality is usually dictated by its surrounding environment. For example, if the system were to be used in a Newsroom environment then it may well need to talk to the Newsrooms news gathering system (iNEWS for example) as well as the video content server (Interplay) It is this detail that will dictate the specification and number of servers used.

Here's the ways MediaCentral is commonly used.

• iNEWS Only
• Interplay PAM or MAM Only (also used with 'Cloud' MediaComposer)
• iNEWS and Interplay (perhaps the most commonly found)

Basic connections for a MediaCentral server in a common environment

Another key factor that we need to take into account is how many users will be using the service. This is especially important to factor in if the users are also able to playback video via MediaCentral. In order to support high user count and high quality source video codecs MediaCentral would have to be scaled 'Horizontally' with additional servers to facilitate both user and video stream demands. More on this later.

Hardware. The MCS Server 101

There was a time when Avid only used HP servers (DL360 and DL380 Gen7/8) with MediaCentral. HP has always been its preferred  'go to' partner especially when it comes to workstations for its Media Composer, but now DELL servers (Power Edge R620 and R630) have also been qualified. Even more interesting is the fact that virtual servers are also supported, but more on that later. 

DELL PowerEdge R630 suitable for use as a MediaCentral server.

HP DL360 Gen8 Server a commonly found server used for MediaCentral

When it comes to the hardware configuration, the servers used for MediaCentral Platform have very high specifications and there are a number of factors to take into account, especially if you are going to specifying your own server.

1. CPU
To handle system resources and especially to create proxy files on the fly, high processing power is needed. A minimum specification of dual Intel Xeon E5-2650 8-core 2.0 GHz (2.0GHz/8-core/20MB/95W) processors in recommended in the MCS Hardware Guide (2.10 used for this post.) But it's worth noting that Avid recommend performance processors only and not energy saving models showing how processor intense this server is. 

2. RAM
MediaCentral also requires a large amount of RAM. Depending upon your needs and processors installed a minimum of 96GB of RAM is recommended, with 128GB of RAM preferred for best performance. Its expected that each core of the CPUs has at least 4GB of RAM.

It may well be that if you have an earlier version of MediaCentral, 2.1 for example, then it may have shipped with less RAM than outlined above. This will be because you may have dual 6 core processors from an earlier system specification and so would have 12 x 4 =48GB of RAM. In some of these cases you may well need to increase the RAM on these system to move forward.

3. HDDs
The servers usually has 2 distinct sets of drives. One RAID1 set to hold the operating system and then a further RAID5 set that is used to hold cached video files for play back to clients (if needed.) In the case of the HP DL360 G8, this would usually be configured as follows...

4. NIC
The MediaCentral Server may need to be connected to ISIS or NEXIS to access the high resolution content to generate proxy files. In smaller configurations with less users or sites using lower bandwidth codecs (DV25, just as an example) the server may be connected by a single 1Gb Ethernet connection. For the majority of sites though a 10Gb Ethernet is needed to support higher codecs or user count. The supported card are listed below (Taken from the 2.10 Hardware guides) for the HP and DELL Servers

NICs and MediaCentral Servers

For most sites using MediaCentral the servers used will be specified by Avid and so you shouldn't have to worry too much about choosing the right processor and NIC. But as we have already seen with RAM configurations, specifications will change from version to version and in some cases, existing sites with MediaCentral may need hardware upgraded to meet the next versions needs.

It's also worth pointing out that Avid also makes a note about over provisioning your system, for example by over spec'ing your CPU's or RAM....


"You may wish to specify higher spec faster, more powerful CPUs to get additional capacity servers. It should be understood, however, that Avid has not measured server capacity in these cases. We can only say that more concurrent streams will be supported, but we cannot specify how many."
 - Avid Documentation


All of the information above should be discussed with your Avid representative as information and system specifications can change almost daily. That said, the above is a good guide to the core needs of any MediaCentral Server. Due to different and optional video playback methods and features, the number of servers required, and options which may need to be expanded, figuring out what to provision can be more complex and you will need the assistance of experts form within Avid to help establish the right solution.

Operating System

Avid has long been a strong user of Microsoft Windows Operating Systems for its core servers, Avid Interplay and Avid ISIS being good examples. MediaCentral however is completely Linux based and its driven almost exclusively at the command line level with no GUI. This means that if you are not a natural Linux operator, you had better get some time with the command line! (See resources below if you need to get to speed with Linux.) 

Avid is very specific about the build and version of Linux it wishes you to use. From the Avid Hardware Guide...


"MediaCentral Platform requires Red Hat Enterprise Linux 6.5 (RHEL), a commercially supported open source operating system. Although it is open source, RHEL is not free and Avid doesn't include it as part of the installation software. RHEL support has two important factors its offers:

• Technical support for OS-related issues. The level of support depends on the support subscription purchased by the customer.
• Indemnification. Because RHEL is an open source project, there is a risk that it will be discovered to infringe on patents and the patent holders will take legal action. Red Hat protects its customers by taking legal responsibility for the distribution.

There are two different ways to provision RHEL:

• HP and DELL offers RHEL renewable support subscriptions.
• Red Hat offers different tiers of annual support subscriptions directly."

 - Avid Documentation



So Red Hat it is. Depending upon how you source your hardware will ultimately dictate how you source your OS. 

If you already have a MediaCentral system you may well be running an earlier version of RHEL 6.3 which you will have to take to 6.5 which is recommended as a fresh install rather than an upgrade. A process that would need configuration files backing off the server and would be something worth having Avid involved with should you need to do this!

How available are you?

As the MediaCentral server is a critical platform and service to your web interface users, you'll want to look at how resilient you want to make the servers to ensure the highest availability and safe guard against server failures. Obviously Virtual servers offer perhaps the highest availability but physical servers offer the most computational power and storage performance and so these are preferred in the majority of cases, but for smaller environments Virtual systems are fast becoming the norm.

If your client demands (see later) are low and dual power supplies, RAID1 OS drives and RAID5 media drives are enough for you then a single server can be used, but for larger systems a slightly more complex approach needs to be adopted.

Clustering
If you need your system to have high resilience, you will need to install a second sever in a 'clustered' configuration. This means that should one server fail the second can take over. 'Clustering' is a more complex installation and its primary role is to ensure that the database that is essential to the configuration and function of MediaCentral is replicated from the active server onto the standby server. Adding a second server also increases the number of video streams that can be generated / number of users that can be served as we now have 2 servers that can generate proxy files for use.

If you have a high number of clients or high data-rate codecs (AVC-Intra 100 for example) you'll need to add in additional servers beyond this clustering.

GlusterFS
When adding a third or fourth MCS Server to a MediaCentral environment this is only done to allow for more users / steams of videos for web clients. These additional servers are 'load balancing' servers and are what we refereed to this earlier in this post when we talked about MediaCentral being 'scaling horizontally.'

To allow for the client demand, proxy files that are generated on one node are distributed across all nodes within the MCS configuration to allow multiple users access to the same clip without the need for its regeneration. This process is managed through a file system, GlusterFS, which is a scale-out network-attached storage file system now owned and maintained by Red Hat.

Keep this in mind.
When using MediaCentral with video playback there is always a Master node. Adding a second node (cluster) provides a fail-over node as well as another MCPS playback service to support more clients streaming media. Any additional nodes simply add another MCPS playback service to support more clients but do not provide any additional redundancy to the main set of services. The video playback requests are distributed to all nodes in the cluster and are load balanced between them.

Note how some services and data are in various states or missing on the Balancing servers when in a 'cluster'

Establishing how many servers you will need can be fiddly to calculate. The number of video proxys files that can be server by a server has 2 limitations. 

1. The CPU's can run out of resources.
2. The network connection between the MCS Server and ISIS/NEXIS storage system can become saturated. 

Avid helps with a number of tables in its MCS Hardware Guide, that show the number of streams a single server can sustain on either 1Gb or 10Gb connections. 

The table I'm using below is specific to the Sandy Bridge CPU. Stream counts with green shading indicate a CPU bound limit (the server is using its full compute capacity.) Stream counts with orange shading indicate an I/O bound limit (the server can process more streams, but is limited by the network connection).

Therefore, in simple terms, if I needed 40 people to access source material stored at XDCAM HD I would need 2 servers (24 streams per server, assuming it is 10GbE attached)

Example taken from the MCS Hardware Guide for MediaCentral v2.10 

But the hardware guide also advises that there is a constant overhead of 0.25 servers needed to run essential non-playback services. This means we also need to factor this into our needs. So lets add this into our previous example.

   No of servers = 0.25 + 40/24* = 1.92 servers.

* This ratio is simply the number of users over the number of video streams a single server can give for a specific resolution.

If you want high availability add another server to the above = 2.92 servers needed.


To sum up the use of Clustering and Glustering systems, Avid says in its Concept and Clustering guide...


"An MCS cluster differentiates itself from a single server by providing the following:

• Redundancy/High-Availability. Services are mirrored on the primary and secondary nodes which provide redundancy of databases, system settings and key services. If any node in the cluster fails, connections to that node are automatically redirected to another node.
• Scale/Load-Balancing. Multiple servers can be added to the cluster to increase the total possible number of client connections and playback streams. All incoming playback connections are
• routed to a single cluster IP address, and are subsequently distributed evenly across the nodes in the cluster.
• Replicated Cache. The media transcoded by one node in the cluster is automatically replicated on the other nodes. If “node2” receives a playback request for an asset that was already transcoded on “node1”, the replication process ensures that the media is available on “node2” without the need to re-transcode.
• Cluster Monitoring. The clustering software includes a utility that enables system administrators to monitor the status of all cluster resources and related services on all nodes. In addition, if a node fails or if a serious problem is detected, designated users are alerted to the issue through an automatically generated an e-mail."

 - Avid Documentation

MediaCentral software and services

One of the hardest thing to get your head around with MediaCentral is the acronyms that are used to name the underlying services as well as some of the third party file systems that are also needed. Where does Avid stop and a third party service or file system start? Sadly there's no getting way from knowing at least some of the core services names as the Avid manuals assume that you have firm understanding of them and what they all do and mean. But for now lets start gently...

The MediaCentral Platform software is a collection of services running on the Linux OS complimented with other third party file systems and databases such as GlusterFS and MongoDB. Avid has a number of core services that are referred to as the MCS services. 

There are 5 key groups of services looking after the likes of user authentication, system configuration settings, and provides proxy-based playback of video assets over the network to web-based clients. This later service, the ICPS (Interplay Central Playback Service) is the most intensive service demanding the most amount of CPU power and RAM and we'll look at this in more detail later. 

The 5 key Avid service groups that run on single and cluster master nodes are. 

1. Interplay Central Core Service (IPC)This 'middle-ware' manages communication between Interplay Central workstations and back-end systems.
2. Interplay Central User Management (UMS)This manages the users and groups and authenticates them.
3. Interplay Central Playback Service (ICPS)
   This service read contents from the ISIS or NEXIS and compresses it for playback use for Internet-connected clients.
4. User Setting Service (USS)It manages the system configuration, user management, and ICPS settings over the message bus.
5. Avid Common Services (ACS) BusThis is also referred to as 'the bus.' Cconfiguration & messaging uses RabbitMQ.

Depending upon the use of the system some of these services may not be needed, for example if used with iNEWS only, the ICPS service will not be needed. In the case of a system intended to be used for use with Interplay using the MediaCentral UX interface all 5 services will be needed needed.

NOTE: We could go on about the software and services used here but this is way outside fo the scope of this post. For much more detail about services and their uses and relationships use the MCS Concepts and Clustering Guide pdf I linked to earlier in this post.

As we have seen already, there are a number services that communicate between each other and the client. If were to dive down into these core services we would see there are mmore refined services with specific roles. These services have dependencies and failure in one service can have have a knock on to another. For example failure of the ISIS service means other services will report failures and in turn video will not be available for users to playback (unless already cached on the MCS servers cache vault.)

Avid does provide some rudimentary tools for monitoring the health of some of these services, for example. 
The command 

    service <servicename> status 

can be used in conjunction with the likes of the avid-all service which encapsulates a number of back-end services. So typing

    service avid-all status 

at the command line level of MediaCentral should return something like the screen grab below, with the sub services status showing OK against each one. 

All well with the avid-all services

There are a number of services that have have grouped together that can be monitored in such as way, but again this post is about the basics and the above is shown merely as an example of what can be seen for monitoring purposes. The ins and outs of monitoring systems is way beyond the scope of this post.

ICPS workings

The MediaCentral servers uses various databases to maintain user permissions and configurations. The most interesting service is that of the ICPS service that allows users to stream media from high resolution video clips held on the central storage system being used.

There are two options for media playback with the MediaCentral Platform: 

1. Frame based media
2. File based media.

To create frame-based playback the MCS decodes the stored media on the ISIS / NEXIS and streams images and audio to the remote client as individual frames. For file-based playback the MCS transcodes the media to flash based media (FLV) and caches the results.

Frame-based playback is the more CPU intensive of the two so fewer servers are needed for a site favouring file-based playback, but this can only be done for master clips, subclips, and basic sequences, but not on advanced sequences that can be made in MediaCentral. These flash based files also lack variable speed playback with the JKL keys. Because of these limits most sites tend to use frame-based playback for users. 

So how does the proxy creation playback work in frame playback? In simple terms the MCS creates a stream of sequential JPG files with associated PCM audio that is pumped out to the clients. 


"The MediaCentral Platform Services Server runs the MediaCentral Playback service. This is a generic playback device that provides the player for the MediaCentral | UX Browsing or Edit client through Javascript API. It concurrently reads multiple media assets and sequences residing on ISIS and compresses that media for playback on interconnected clients."
 - Avid Documentation

Playback process for ICPS when making Frame based media 

The ICPS service does an amazing job or providing content back to the users when you understand this 'transcode' is all done on the fly. 

The advantage of the frame based media approach is that the bandwidth needed is less than 4Mbs per client and the proxy calculations are a constant load the MCS CPU allowing for easier calculations of client demands etc. The ICPS service also buffers material to ensure clients have accurate in sync playback. Even when using the JKL keys! 

Configurations

The configuration of MediaCentral is through a combination of command line and the web interface. The majority of the technical details are carried out at the command line while a smaller set of users definable settings are presented through the web interface. These System Setting are only available to users who log on as an Administrator.

Web interface to configure MediaCentral 

    

System setting are broken down into groups as can be seen above. I'll just touch on a couple of interesting ones.

• Playback Service.
  This is where you can see/configure which Interplay you are connected to and which ISIS / NEXIS system you are using. You can also see the active state of the connection to ISIS / NEXIS here too.
• Send to Playback.
  This is where you build 'Send To Playback' profiles so that MCS users can indeed deliver content to specific servers. However its worth noting that this naming convention is not entirely accurate and should be thought as more as 'Send To' and this where you would build profiles to deliver to any 3rd party server, not just transmission devices.
  The above profiles may also reply upon other optional Interplay components such as Transcode and Transfer Engines to allow the envisaged workflow.  
• Interplay | Production
  Here we can specify various Interplay servers addresses so that MediaCentral can locate and interact with core system components.

For a more detailed view of these areas look at the Installation and Configuration guide for the version of MediaCentral you are running.

Users

MediaCentral users and groups are managed in a user database that is stored on the MediaCentral Services server. These users can be added manually or by synchronising users with an external list of users, such as an AD or LDAP server.

It's important to remember that user's will need to exist in multiple locations, Interplay and iNEWS for example. When a user first signs into MediaCentral | UX, they are prompted to enter their Interplay Production and iNEWS username and details. When these have been authenticated, the user provides only the MediaCentral user name and password on subsequent sign in.

Sign in once onto multiple systems (user names and passwords don't have to match)

Once a user exists within MediaCentral it will need to have a 'Role' assigned to it, it is the role that controls the users ability within MediaCentral.
'Roles' are created with various 'Layouts' assigned to them and it is these 'layouts' that dictate the ability of the Role. The Layouts allow users to switch between various windows into the web tool to have different functions. These Layouts are fixed and some depend upon plugin being installed in order to be available.

Possible layouts within MediaCentral at the time of writing

Example. I could have a Role called 'WebPublish' which I have given the 'Layouts' of Cut, Log, Story and Media Distribute. I can give this 'Role' to various Users or better still a Group.


Client Web browser 

The web client does have a rather frustrating limitation of only being able to use Chrome across both Mac and PC platforms, but you can use Safari on Mac systems.
Because MediaCentral needing the Netscape Plugin Application Programming Interface (NPAPI) to support JAVA in the browser and Chrome moving away from NPAPI to Pepper (PPAPI), Avid has had to created there own cut down version of Chrome version 45 and called it the MediaCentral | UX Desktop App. At the time of writing we are up to 1.4 but Avid is moving its video playback to HTML 5 to allow users to go back to their own browser of choice. Until this new feature is released then the MediaCentral | UX App stays. 

The HTML 5 can be exposed in the current build of MediaCentral but this is really a Beta function and so not recommended for day to day use. More details about HTML 5 play back can be found here.

It's also worth pointing out that there are Android and iOS Apps that can be used with smart phones and tablets to give users access to content in the palm of there hand. If you wish to use these devices be aware they have a technical knock-on when doing calculations for video streams and server loads, but we're not looking at this in this post.

MediaCentral User Interface also known as UX

There is a detailed guide to the whole UX interface in Avid's MediaCentral UX User Guide. It's a hefty read at 640 pages but its a good companion for when you need to trouble shoot user issues.

Licensing

As an engineer I spend more and more time looking after the really dull stuff, such as Licenses. That said, if there's no license then there is no running system, so it's a evil necessity that you have to understand and manage, no matter what software you have to maintain. MediaCentral is no exception.

MediaCentral itself has no license, but as it acts as an extension of either the iNEWS or Interplay environment it sits in, its licensing is pulled from either one or the other of these 2 systems. 


"MediaCentral Base Client License:
This license includes both an iNEWS client license and an Interplay Production license. This Interplay Production license (J type) allows browsing of assets stored in an Interplay Production database, creating basic or advanced sequences, and mobile connectivity. An iPad playing video counts as a base license. The only thing that a base license does not allow you to do is to connect to more than one database simultaneously (iNEWS and Interplay or using Media | Index to search another Interplay Production workgroup). If a site is configured for Media | Index and not configured for remote Interplay systems, then a base license can perform an indexed search (if other systems are configured the feature requires an advanced license). The J type license is also used by Media Composer and Interplay Access (Access will not use a license provided you launch and log in to any of the other applications first).

MediaCentral Advanced Client License: 
This license includes both an iNEWS client license and an Interplay Production license. The Interplay Production license (G type) allows both browsing of an Interplay Production database and creation of script sequences attached to iNEWS stories using that media. It also includes the ability to search remote Interplay Production databases and use the Deliver To Me function. An iOS device playing video attached to an iNEWS story is an advanced license."
 - Avid Documentation

REVIEW

So that's it for my little peep at MediaCentral. There's a vast amount that I have missed out but I just wanted to scratch at the surface to expose some of the engineering needs of the system. The post has already run much longer than I hoped and I've edited the content so heavily to keep it down I hope it still makes sense. 

MediaCentral systems can appear daunting to maintain and troubleshoot, especially if you have little Linux experience, but like anything, the more you do the better you get. If you have a smaller system with a single node then its maintenance and getting to grips with its configuration is a easier start. The bigger clustered systems do require a much more thorough understanding and quickly become complicated beasts that need expert knowledge when assistance is needed.

Log files are a constant need when looking after a system and MediaCentral generates lots of them for you, but being able to read these or even understand what you are looking for is an art in itself. Most of the time these logs would be sent on to Avid for them to investigate and come back with suggestions to address the issue. As ever there's an article on the Avid KB about how to find and collect the right logs. So even if you have no ability with Linux you can at least find the log files and pass them on ;o)

If you have little or no Linux experience then jump on a free Linux course (details of some below) and then get yourself on a MediaCentral WG453 course where you'll be able to build a system from scratch during the course with the aid of your tutor.

I'll be reviewing this article over the coming weeks and will add updates as needed but I hope you found this introduction of interest and a good starting point to gaining a better understanding of the system.

____________________________________________________________

Resources

Here's a few links to some useful resources for using with MediaCentral.

Get to grips with Linux

LINUX
Download a free copy of Linux. Pick you flavour, Ubuntu or Fedora are popular ones to go for and then jump online for Free Online Linux courses. 
  https://training.linuxfoundation.org/free-linux-training
  http://linuxsurvival.com/

MongoDB
Get background information about MongoDB to understand its ability and how/why Avid use it in MediaCentral.
Introduction to Sharding MongoDB
Later releases of MediaCentral switched to a sharded MongoDB, this article gives some more background information.
GlusterFS
This is the file system used to distribute the Proxy files across servers for load balancing.

Avid Linux Resource A hidden little GEM is the Avid Tech Talk episode with Jim Pappas. Its named "Learn to Love Linux" and is a great introduction to Linux and how you can interact with MediaCentral, but it doesn't offer MCS specific commands that we use directly with MCS to establish the state of the system. It's almost an hour long but I really do recommend watching it.

Jims Pappas - Learn to love Linux

Useful Avid KB Articles

Collecting-log-files-from-a-MediaCentral-server

Troubleshooting-MediaCentral-UX-Playback-Service

Avid-MediaCentral-Platform-Troubleshooting

Avid-MediaCentral-Health-Check-for-a-Single-Server

Avid-MediaCentral-Health-Check-for-a-Cluster