Automate client testing on CTT|OPC Certification and Interoperability Testing|Forum|OPC Foundation

Avatar
Search
Forum Scope


Match



Forum Options



Minimum search word length is 3 characters - maximum search word length is 84 characters
Lost password?
sp_Feed sp_PrintTopic sp_TopicIcon
Automate client testing on CTT
Avatar
João Barbosa-Neto
Member
Members
Forum Posts: 8
Member Since:
01/17/2020
sp_UserOfflineSmall Offline
1
11/16/2020 - 12:04
sp_Permalink sp_Print

Greetings,

Is there a way to automate client testing through the CTT? If not, are there plans on making this be automated in some fashion? Does the test lab run all client scripts in the CTT manually?

Avatar
Alexander Allmendinger
Germany
Moderator
Members

Moderators

Moderators-Specifications

Moderators-Companion

Moderators-Implementation

Moderators-Certification

Moderators-ProductsServices
Forum Posts: 66
Member Since:
07/11/2017
sp_UserOfflineSmall Offline
2
11/17/2020 - 02:51
sp_Permalink sp_Print

Hi João,

the CTT does come with a command line interface, which allows to start individual scripts while provide a configuration file. With this combination the CTT can be integrated in different tools like a CI System e.g. With the combination of the CTT and a CI system you can automate the UA Client testing. The CTT by itself unfortunately cannot automate the UA client testing, because it cannot validate the result of the test. E.g. saying we do have a test case where the StatusCode of a MonitoredItem is being changed to Bad_NoCommunication. Then the UA Client application is expected to not use the value for further processing and needs to indicate the Bad StatusCode to the user. Obviously the CTT does not have internal knowledge about the UA Client so it cannot determine, whether the value has been used or not. This needs to be validated inside the UA Client either manually by a tester or automatically by a CI system, which of course needs to be able to validate this aspect.

Regarding your second question. While we expect the vendor to execute every test defined, in the lab we are initially focusing on a subset of tests. This subset is well picked so the testers are not only validating a specific test for a feature but are also observing general aspects and the communication flow, so they'll notice when there is an issue, missing feature or strange things going on. If something like this is observed, the testing on this feature is extended to a greater detail level to ensure certified applications are meeting the very high quality standards the certification program implies.

I hope this helps and answers all your questions.

Regards,
Alexander Allmendinger

Avatar
João Barbosa-Neto
Member
Members
Forum Posts: 8
Member Since:
01/17/2020
sp_UserOfflineSmall Offline
3
11/17/2020 - 06:36
sp_Permalink sp_Print

Thank you for the prompt reply Alex, much appreciated. Has there also been any discussion on standardizing client tests if the application under test is also a server? Could the CTT then control the client and validate its response handling by interacting with the server, e.g. through method calls?

Avatar
Alexander Allmendinger
Germany
Moderator
Members

Moderators

Moderators-Specifications

Moderators-Companion

Moderators-Implementation

Moderators-Certification

Moderators-ProductsServices
Forum Posts: 66
Member Since:
07/11/2017
sp_UserOfflineSmall Offline
4
11/17/2020 - 07:13
sp_Permalink sp_Print

Sure thing.

Yes, of course we thought about different ways on how we could automate more of the client testing. But the problem always is the validation of the actions taken in the client. Let me give you another example, the client is often asked to provide information about certain communication issues in the log/trace of the application. Now we could again define a Method or Variable which provides the log output as a file, which then can be read for validation, but this is a very special behavior which needs to be implemented by the Client vendor and you still cannot be sure that the information actually is in the log. Same is for the case I described above, where we'd expect to see a bad StatusCode not being used in the Client. There you could say, it needs to be mapped to the Server side so we can validate it automatically. But the question still stands, has the Client used the value for example in scripts, IO Fields, Trace/Log, ... 

At the end it always comes down to ask the Client vendors to provide a very special Client implementation which would allow for automated testing while you still cannot be sure that it is doing what we expect. Instead we consider it more useful to actually go through the effort to integrate the CTT in a CI system. Those CI tests often already have access to the Log and internals of the application and therefore can be utilized easier for doing the validation.

I hope this helps,
Alex

Avatar
João Barbosa-Neto
Member
Members
Forum Posts: 8
Member Since:
01/17/2020
sp_UserOfflineSmall Offline
5
11/19/2020 - 10:57
sp_Permalink sp_Print

Thank you again for the info Alex, it certainly has helped.

Forum Timezone: America/Phoenix
Most Users Ever Online: 510
Currently Online:
Guest(s) 18
Currently Browsing this Page:
1 Guest(s)
Top Posters:
Forum Stats:
Groups: 2
Forums: 10
Topics: 1341
Posts: 4545