It’s now been more than a year since the first ETSI NFV Plugtest,which provided an important testing platform and forum for dozens of NFV technology providers. It was the first time that so many vendors from a broad cross-section of the networking industry came together to assess the level of interoperability of their implementations between MANO, VNF and VIM vendors.
The results for 1st ETSI NFV Plugtest were better than expected. Hundreds of tests conducted among 35 commercial and open-source NFV implementations achieved more than 98 percent interoperability. We at DZS were further encouraged by overall execution rates for complex operations such as instantiation, scaling, and network service updates in which the Plugtest achieved an execution rate of almost 39 percent across all vendors.
Improving on this overall execution rate was a key initiative of the recently held 2nd ETSI NFV Plugtest at the ESTI headquarters in Sophia Antipolis, France. The scope and scale of this Plugtest were even more ambitious than the first. 41 commercial and open-source NFV implementations were under test with more than 100 test engineers conducting 189 tests during a breakneck one week period. The second Plugtest also featured a number of advanced testing scenarios such as multi-site, network path, enhanced platform awareness, fault and performance management, and NFV application platform API interfaces.
As shown in the following chart, Plugtest Two achieved nearly double the overall execution rate compared to the first Plugest. The execution rate success is even more encouraging considering the second Plugtest featured “50 percent more test cases addressing more advanced features,” as captured in the final ETSI report.
DZS actively participated in these advanced test cases including complicated multi-VNF management & orchestration, multi-site implementations with scaling, as well as tests for complex virtualization specifications such as single-root I/O virtualization (SR-IOV). DZS successfully executed 18 advanced test sessions between different VNF and VIM vendors to successfully complete all the supported test cases. DZS also on-boarded seven new virtualized network functions compared to our results from the first Plugtest. This brings the total of on-boarded VNFs during Plugtests to 28 (DZS has on-boarded more than 80 VNFs in total with more added regularly).Not everything went as smoothly as DZS (or the other participants) hoped or even expected. We did observe minor interoperability issues such as certain scaling behavior or issues with API documentation. There were also some error handling cases during high latency situations (500ms to 800ms). Of course, that’s one of the main reasons for participating in Plugtests such as these – to identify and remediate interoperability issues, especially for complex test cases. Better to discover these problems in a lab than within a customer network.
Not everything went as smoothly as DZS (or the other participants) hoped or even expected. We did observe minor interoperability issues such as certain scaling behavior or issues with API documentation. There were also some error handling cases during high latency situations (500ms to 800ms). Of course, that’s one of the main reasons for participating in Plugtests such as these – to identify and remediate interoperability issues, especially for complex test cases. Better to discover these problems in a lab than within a customer network.
This was also the first time a service provider was allowed to monitor the interoperability testing. This gave them the opportunity to provide feedback to ETSI and provide progress reports back to their own management teams. This participation by service providers is crucial to engagement and advancing how the Plugtest learnings are adopted in real world production plans.
The test results indicate that NFV as a whole took another incremental step toward full-scale, industry-wide deployment for real-world use cases. The great news is that we seem to be accelerating the pace of advancement, even in the face of increasing complexity. Attesting to this is the fact that ETSI is planning to hold its 3rd NFV Plugtest this summer (May 29 through June 8 2018) to further focus on Multi-VNF, real end customer use cases. This time, it will also be coordinated with OPNFV.
It is clear that DZS and the other vendors and operators are learning a lot from these Plugtests and, more importantly, applying these lessons to solving problems carriers routinely face in the real world. I can only imagine the achievements we will make at the next Plugtest. Outside of the testing itself, it is great to see the comradery between ETSI organizers and the engineers from all the vendors. The environment fostered is one of collaboration, healthy discussion and partnership for the purpose of advancing the industry forward.