Automate client testing on CTT|OPC Certification and Interoperability Testing|Forum|OPC Foundation

Hi João,

the CTT does come with a command line interface, which allows to start individual scripts while provide a configuration file. With this combination the CTT can be integrated in different tools like a CI System e.g. With the combination of the CTT and a CI system you can automate the UA Client testing. The CTT by itself unfortunately cannot automate the UA client testing, because it cannot validate the result of the test. E.g. saying we do have a test case where the StatusCode of a MonitoredItem is being changed to Bad_NoCommunication. Then the UA Client application is expected to not use the value for further processing and needs to indicate the Bad StatusCode to the user. Obviously the CTT does not have internal knowledge about the UA Client so it cannot determine, whether the value has been used or not. This needs to be validated inside the UA Client either manually by a tester or automatically by a CI system, which of course needs to be able to validate this aspect.

Regarding your second question. While we expect the vendor to execute every test defined, in the lab we are initially focusing on a subset of tests. This subset is well picked so the testers are not only validating a specific test for a feature but are also observing general aspects and the communication flow, so they'll notice when there is an issue, missing feature or strange things going on. If something like this is observed, the testing on this feature is extended to a greater detail level to ensure certified applications are meeting the very high quality standards the certification program implies.

I hope this helps and answers all your questions.

Regards,
Alexander Allmendinger

Sure thing.

Yes, of course we thought about different ways on how we could automate more of the client testing. But the problem always is the validation of the actions taken in the client. Let me give you another example, the client is often asked to provide information about certain communication issues in the log/trace of the application. Now we could again define a Method or Variable which provides the log output as a file, which then can be read for validation, but this is a very special behavior which needs to be implemented by the Client vendor and you still cannot be sure that the information actually is in the log. Same is for the case I described above, where we'd expect to see a bad StatusCode not being used in the Client. There you could say, it needs to be mapped to the Server side so we can validate it automatically. But the question still stands, has the Client used the value for example in scripts, IO Fields, Trace/Log, ... 

At the end it always comes down to ask the Client vendors to provide a very special Client implementation which would allow for automated testing while you still cannot be sure that it is doing what we expect. Instead we consider it more useful to actually go through the effort to integrate the CTT in a CI system. Those CI tests often already have access to the Log and internals of the application and therefore can be utilized easier for doing the validation.

I hope this helps,
Alex