In our previous blog post we looked at how after ten years our Central London lab proved invaluable in building world class products. This blog post will highlight some of the big technological changes in our lab over the last ten years.
2015-2016 – Housekeeping and server building
Initially the main area of work was removing old hardware from the labs, removing a large amount of SD-only equipment. It was a major endeavour removing literally tons of legacy equipment.
At the time we regularly shipped a lot of hardware and the lab was invaluable in building and testing this equipment. Thankfully nowadays we mostly sell software but at the time having a location to perform regular automated installs of software was extremely important.
2016-2017 – The dawn of Uncompressed Video over IP
These were the first years of developing our Uncompressed IP Stack (initially ST 2022-6 and later ST-2110). We invested in an Arista 40G switch in our lab, and spent over eighteen months developing our first 10G ST 2022-6 sender and receiver using this stack.
One entire rack continues to be dedicated to Uncompressed IP, now with NMOS registry/controller, and dozens of permutations of Server and NIC (Network Interface Card) manufacturers and models. Historically, we supported the Intel 82599 10G card. But now we support Mellanox ConnectX5, ConnectX6 and now ConnectX7 cards, each with their own subtleties. This requires a lot of servers and a lot of testing.
2018-2019 – In-house hardware development
Developing our own SDI cards led to an increased focus on lab bench space and being able to test physical hardware and its behaviour. At the same time we began testing 25G network cards, as we started developing our ST-2110 stack.
These years were also the beginning of servers having AVX-512 CPU optimisations. As shown below being able to test these optimisations is key building high performance products.
2020-2021 – COVID accelerating major lab upgrades
100G Networking
The plan in early 2020 was to upgrade the core switch to 100G to aid development of our 25G and later 100G products. Used 100G switches were now affordable. The switch arrived in March 2020. But we were rudely interrupted…
COVID
Mandatory remote work led to a mad rush to make sure every single device was remotely controllable. Any spare encoder was put into service to support remote work. Our camera was left pointing at the street and has been doing so ever since, providing a valuable test signal.
This period was also the year of major network upgrades. An ancient central core Cisco switch was replaced with a modern Arista leaf-spine for corporate and internal multicast video traffic. Miles of cabling from each rack to the central core were replaced with intra-rack cabling and fibre to the spine.
Scopes
Special mention must go to the major effort involved in making SDI scopes (waveform monitors) remotely controllable. These are specifically designed to have an operator sit next to them. They all use DVI-A (Analogue), a weird flavour of DVI, at a resolution of 1024×768, a resolution not friendly to broadcast.
It was extremely important that teams could access this equipment remotely and in real-time as they developed our encoders and decoders. Throughout COVID we struggled with various DVI (or DVI carried in HDMI) to SDI converters glitching or otherwise being unreliable. COVID was a time to make-do even if things were not perfect.
Eventually after testing numerous products, we eventually found our knight in shining armour – the Monoprice “ALL to SDI” converter. This part, now end-of-life, was the only converter able to reliably convert the DVI-A to SDI so we managed to buy as many as we could. As shown below we make extensive use of these converters.
Using our low-latency encoders, pressing a button on the scope remote control changes the display as if it were right next to you. We are actually users of our own products every single day!
IP or SDI? Both in the end.
There was substantial discussion about whether we should move the entire lab to 2110 and attempting to keep SDI signals within particular racks. In the end, owing to our continual need to test SDI signals, we decided to stay in a mixed environment and install a larger SDI router.
2022-2023
TS Loops
A proper multicast network meant we could have transport streams on a loop for easier lab testing. We therefore setup dozens of dedicated streams of all sorts of weird and wonderful transport streams we collected over the years. This also makes for a good test of our decoder, as looping transport streams often cause third-party decoders to lock-up as they cannot handle timestamps jumping back in time or other discontinuities.
The TS loops are extremely valuable and used extensively for product testing.
PTP
Interestingly, we managed to survive for several years without a GPS locked PTP Grandmaster. But new NICs were not locking to a free-running PTP clock and it was clear we needed a proper GPS source. Owing to complications with roof access, we settled on using a window mounted PTP grandmaster. While not perfect, on a north-facing window, it allowed us to continue development on NICs with more sophisticated PTP implementations.
Monitoring
As part of this upgrade, we needed more complex monitoring. As a result, we introduced sophisticated monitoring of all parts of the network, including servers, networking and timing.
2023-2024
More Scopes
From what was a single SDI scope shared amongst staff, became a lot more scopes, thanks to the extremely low cost of used SDI scopes. This meant that multiple SDI analyses could occur in parallel, saving a lot of time. In total we have six SDI scopes available, each with their strengths and weaknesses. Incorporating these scopes involved buying the last available DVI to SDI converters mentioned above.
2025 – Back to the future with an Analogue router
Because we support all broadcast frame rates, we need to be able to work with various formats of blackburst and tri-level reference, often in parallel when different testing and development is in progress concurrently.
This also needs to be controllable remotely over IP, which many legacy broadcast appliances – particularly those from the days of analogue video – do not support.
Through careful selection of devices with the necessary capabilities, we have built a PTP co-timed, remotely controllable reference distribution setup, with which we are able to remotely route any reference format to any SDI device independently.
This system is an integration of everything from the latest broadcast standards through to the Telnet protocol from the 1970s!
TS loops to 10Gig
One day, after adding another TS stream to our TS loop system we saw that all of them started glitching. After some investigation we found that we had over 1Gbps of TS traffic emanating from a single server. After upgrading the network port on the server, we were happily running and this 10G system could be plugged directly into the main spine switch, giving us a huge amount of future capacity.
What have our customers gained from our lab?
Comprehensive Testing Capabilities
Our lab is equipped to test a wide range of combinations between different setups, including using products from Blackmagic, Intel or NVIDIA, as well as our own SDI cards.
This extensive testing capability allows us to mirror whatever set-up our customers may have, enabling us to meet their specific needs. It also empowers us to pay attention to a wide range of details. In fact, we take pride in the care we put into our products, ensuring that every aspect is finely tuned for optimal performance.
Commitment to Quality and Excellence
Operating 24/7 in the heart of central London, our lab sets us apart from competitors. While maintaining such a space requires investment, from air conditioning to maintenance and upgrades, we are proud to invest for the sake of excellence. We see our lab as a testament to our dedication to quality and customer satisfaction.
Ten Years of Excellence
As we celebrate our lab’s tenth birthday, it remains central to our commitment to refining our products and adapting them as the industry changes. By enabling us to deliver the highest quality solutions to our customers, our lab has played a crucial role in our success over the past decade. It has allowed us to innovate, adapt, and maintain the highest standards of quality in our products.
