Log In
Home
03/24/2017
Offline
Part 11, Section 6.5.3 defines modificationTime as "The time the modification was made. Support for this field is
optional. A null shall be returned if it is not defined". No further explanation is given as to what that means.
In section 6.4.3.3, the read-modified functionality is described thus: "it reads the modified values, StatusCodes, timestamps, modification type, the user identifier, and the timestamp of the modification from the history database for the specified time domain for one or more HistoricalDataNodes. If there are multiple replaced values the Server shall return all of them."
These descriptions suggest strongly that (A) modificationTime is the time when a previously persisted historic value was modified SUBSEQUENTLY, which means it may be, and will most usually be, neither its source not its server timestamp.
However, the end of 6.4.3.3 remarks: "A Server does not have to create a modification record for data when it is first added to the historical collection. If it does then it shall set the ExtraData bit and the Client can read the modification record using a ReadModified call. If the data is subsequently modified the Server shall create a second modification record which is returned along with the original modification record whenever a Client uses the ReadModified call if the Server supports multiple modification records per timestamp."
This means that, in principle, the server can persist the original value as a modified value, and then return it as such even if it has never been modified subsequently. Then it has to do something about its modificationTime. Now if we try to apply (A) to this, we can only conclude that it cannot really be applied, in which case we can say that (A) is invalid and we should (B) record either the source or the server timestamp as a modification time. This has the benefit that it is always unambiguously available, but, frankly, this interpretation seems to contradict everything else.
Another option is (C): just have NO modification timestamp in such a case, which is permissible by the definition of modificationTime.
Given everything that I read in the spec, I guess I can just implement either (A) or (B), and, should I want to regard the "original" value as "modified", either (A) + (C) or (B); and, of course, I can always just omit the modification timestamp in every case to avoid the complication to begin with.
However, is there any consensus as the expected behavior? What is recommended?
05/30/2017
Offline
Seems like you are making more complicated than it needs to be.
At every timestamp you can have at most 1 current value and 0 or more old values.
The data timestamp is the same for all of these values.
The current value is returned with the ExtraData bit when read with ReadRaw.
The old values are returned when you call ReadModified.
The old values have an additional timestamp to indicate when they are created.
The fact that one of these timestamps could be the same as the data timestamp is irrelevant.
Note that there is a use case for auto-create first old value when the historian is not processing live data. The modified record would indicate when the data was first added to the archive which could be some time after the data timestamp.
03/24/2017
Offline
Randy Armstrong said
The old values have an additional timestamp to indicate when they are created.
Thanks for your help! You spoke of "additional timestamp" and "data timestamp". I am afraid these are not defined by the spec unambiguously.
Is the "additional timestamp" the same as "modificationTime" of HistoryModifiedData?
If it is, can you clarify what "when they are created" means? This is not trivial, because that can mean "when the old values were originally created" (that is case B in my original post) and that can mean "when the modified values were created for (now old) original values" (case A of my post). Repeating myself, a straight-forward reading of the spec ("The time the modification was made") suggests (A).
I could try to rephrase this. Read-Raw and Read-Modified return DataValue structures. Each can have a source and/or a server timestamp. Read-Modified also returns modificationTime. I understand that any and all of these three timestamps could be the same, but let us consider a scenario where a historian with a completely blank history records a single value, for the very first time, for a given tag with source timestamp U and server timestamp V. Let, for clarity, U < V. Then, at server time W, V < W, the previously stored value is updated; in other words, a modified data record is created. Already at this point it is unclear whether the new value can be stored with server timestamp W. I do not think the spec says whether one or the other timestamp can be updated.
Anyway, a read-raw call will return source timestamp U and server timestamp either V or W. Now for the read-modified call, how should the original value be returned?
Source timestamp = U
Server timestamp = V
modificationTime = U? V? W? something else?
In a more complicated case, assume that I store the new value still with the old source timestamp U, but make its server timestamp X, V < X < W. It is even less clear whether this is permitted by the spec; if not please let me know. Then, again, when queried for modified data, what do I return for modificationTime? U? V? W? X?
Randy Armstrong said
Note that there is a use case for auto-create first old value when the historian is not processing live data. The modified record would indicate when the data was first added to the archive which could be some time after the data timestamp.
If you are referring the the end of section 6.4.3.3, which I quoted, I do not think the specification says there, or elsewhere, that this is about "not processing live data". The way I read it is that the server is at liberty to create (or return) modified data even if the data are not really modified. Even though this is of secondary importance for my inquiry, I would appreciate having a pointer at the spec that stipulates any constraints on that. If this is not specified, but is still considered good in the oral tradition, that would also be good to know 🙂
05/30/2017
Offline
I was trying to simplify the logic that should exist.
additional timestamp = modification time in specification.
data timestamp = source timestamp/server timestamp
the modification time unconnected to the data timestamp.
You should not assume any time relationship between the modification time and the data timestamps. They are independent values with completely different rules for setting them.
While it intuitively makes sense that a modification time >= data timestamp your code should never need to assume that.
That said there is an ambiguity in the specification when it comes to server timestamp.
if a value is updated with a different server timestamp then that would imply that a record is inserted at the server timestamp, deleted at the old server timestamp and modified at the current source timestamp. The update API cannot handle all of the different permutations which leads me to believe that the update API is only intended to operate on the source timestamp. The server timestamp only exists as a FYI and will never have modified records associated with it.
Please enter a mantis issue:
https://mantis.opcfoundation.o.....default=no
The way I read it is that the server is at liberty to create (or return) modified data even if the data are not really modified.
If it is not restricted it is allowed. There is no requirement that a modification be rejected because it has the same value as the current value.
03/24/2017
Offline
Randy Armstrong said
While it intuitively makes sense that a modification time >= data timestamp your code should never need to assume that.
I do not intend to make that assumption in production code. However, I asked explicitly for a clarification when they can be assumed to be different (when they are not, there is nothing to clarify) and, for the sake of the clarity, naturally sequenced.
I am not convinced that there is an ambiguity w.r.t. the server timestamp. Neither read nor update service has a way to request which timestamp should be used to locate data, and, moreover, Part 11 Section 4.3 says: "The request is in terms of SourceTimestamp but the reply could be in SourceTimestamp, ServerTimestamp or both timestamps." Therefore I agree with your conclusion that "The server timestamp only exists as a FYI and will never have modified records associated with it." What is still unclear, whether it can be modified for a given tag/source timestamp combo. But, for now, let's take it out of the picture and assume there are only source timestamp and modification time. So the original value has (source) timestamp U, and it is modified when the (wall clock, to avoid saying 'source' or 'server') time is W, U < W (for clarity, not in a general).
Should read-modified report that back as source timestamp = V, modification time W? I assume you will say 'yes'.
Now, let's try to consider the server timestamp nevertheless. I understand that the history-update service (as currently specified) may be unable to handle updates of the server timestamp. Assume it is NOT used, instead some proprietary means are used to modify historical values.
As before the original value is written with source timestamp U and server timestamp V, U < V. This is still perfectly within the spec.
Now, using the proprietary means, I update the value so that the new value has source timestamp U, server timestamp X (V < X), and the old value is written as a modification record with source timestamp U, server timestamp V, and modification time W (X < W). These timestamps are then available as such via the read-raw and read-modified calls. Would this behavior contradict anything in the spec? If not, would it be unadvisable for some other reason?
05/30/2017
Offline
source timestamp, server timestamp and modification time are 3 independent variables.
the source timestamp is a time series with some connection to real world process that is the origin of the data.
server timestamp or modification time are not time series but rather merely additional properties on a data point identified by the source timestamp.
You cannot make any assumptions about the order of data if you sort by server timestamp or modification time. Presume it is random.
03/24/2017
Offline
Here is another formulation of the same question.
Read-raw returns one value with source timestamp U and server timestamp X and the extra bit set.
Read-modified (for the same tag) returns one value with source timestamp U, server timestamp V and modification time W.
Is any of the following required or disallowed by the spec:
1. X == V
2. X != V
3. X == W
4. X != W
5. V == W
6. V != W
I have not found anything that would either require or disallow any of that.
In one of the previous messages you said "They are independent values with completely different rules for setting them." Where are those "rules" specified?
05/30/2017
Offline
They are independent values with completely different rules for setting them.
There is nothing in the spec therefore no rules apply. I was simply stating what is implied by the lack of rules in the spec since you seem to be convinced that some rules must exist.
So none of your cases are disallowed. The timestamps are independent and you cannot assume any relationship between them.
Moderators-Specifications
Moderators-Companion
Moderators-Implementation
Moderators-Certification
Moderators-COM
02/24/2014
Offline
So I have read this long discussion and have some comments. Part of the discussion appears to be somewhat contrived. So here are some general definition that timestamp and historical data need to follow:
From section 4.3 - "For HistoricalDataNodes, the SourceTimestamp is used to determine which historical data values are to be returned."
If a record is added, altered or deleted it becomes a modified value. Data that is collected in a standard manner should not be marked as modified data. See definition of modified data:
"HistoricalDataNode’s value that has been changed (or manually inserted or deleted) after it was stored in the historian
Note 1 to entry: For some Servers, a lab data entry value is not a modified value, but if a user corrects a lab value, the original value would be considered a modified value, and would be returned during a request for modified values. Also manually inserting a value that was missed by a standard collection system may be considered a modified value. Unless specified otherwise, all historical Services operate on the current, or most recent, value for the specified HistoricalDataNode at the specified timestamp. Requests for modified values are used to access values that have been superseded, deleted or inserted. It is up to a system to determine what is considered a modified value. Whenever a Server has modified data available for an entry in the historical collection it shall set the ExtraData bit in the StatusCode."
The definition of ServerTimestamp is the definition in Part 4:
"The serverTimestamp is used to reflect the time that the Server received a Variable value or knew it to be accurate.
In the case of a bad or uncertain status, serverTimestamp is used to reflect the time that the Server received the status or that the Server last tried to recover from the bad or uncertain status.
In the case where the OPC UA Server subscribes to a value from another OPC UA Server, each Server applies its own serverTimestamp. This is in contrast to the sourceTimestamp in which only the originator of the data is allowed to apply the sourceTimestamp.
If the Server subscribes to the value from another Server every ten seconds and the value changes, then the serverTimestamp is updated each time a new value is received. If the value does not change, then new values will not be received on the Subscription. However, in the absence of errors, the receiving Server applies a new ServerTimestamp every ten seconds because not receiving a value means that the value has not changed. Thus, the ServerTimestamp reflects the time at which the Server knew the value to be accurate.
This concept also applies to OPC UA Servers that receive values from exception-based data sources. For example, suppose that a Server is receiving values from an exception-based device, and that
the device is checking values every 0,5 seconds,
the connection to the device is good,
the device sent an update 3 minutes ago with a value of 1,234. In this case, the Server value would be 1,234 and the serverTimestamp would be updated every 0,5 seconds after the receipt of the value."
The historian also follows the part 4 definition of SourceTimestamp (but this appears to be accepted)
When modifying a value - you are changing the value, you should not be changing any of the timestamps, since the timestamp relate to the original data. The ServerTimestamp is not used for accessing data (it can be returned and a warning is included that this timestamp might be outside of any range of historical data that was requested). The modification time is associated with a group of records and it is not provided as part of the updateDataDetails structure (i.e. in the call) - it is just the stored by the server (if it is stored at all since it is not required) as the time the HistoricalUpdate occurred. In your listed case, the two timestamp means an normal operation occurred that collected a value and then an updated occured that changed the value. So in your list of matches, some could not happen, some are likely and usual:
1. X == V - this would be typical since usually it is not changed
2. X != V - this is allowed, but again not typical (not sure what case would make sense for this). why is the ServerTimestamp being changed? But there is no rule related to this. In general there are no rules for ServerTimestamps.
3. X == W - this could not happen since time moves forward - the update operation had to happen after the original record was written and the original ServerTimestamp would have had to be earlier then the later modification. i.e. X<W (assuming normal flow of time, if time is reset or changed on the device then all sorts of strange time related items might occur)
4. X != W - for your described steps I would say this is correct - like state above X<W (which is also X!=W)
5. V == W - Again this is not illegal but i would like to understand the use case that would want to set the serverTimestamp to the time that a modification occurred to the data? I would not make sense for any use case I can think of, but maybe I'm missing something - Also since the ServerTimestamp is provided in the DataValues structure that is being passed into the Updated function, and the Server is setting the time of the modification as part of the process it seem unlikely that both timestamps would be the same.
6. V != W - I would think this is typical, actually i would say V<W is typical.
That said if you are a client looking at the data only the SourceTimestamp has any relevance related to the data (the modification time and the user that made the modification, may of of interest to an auditor).
As i started this comment - I think much of what is being raised is contrived and not realistic to typical use cases of a historian.
Paul Hunkar - DSInteroperability
03/24/2017
Offline
Paul, thanks for your valuable feedback.
I probably should clarify that my context is of a historian server that is fed data from various sources, including other historians. So it cannot assume much about the data it sees, and in particular has to assume it may see the same tag - same source timestamp multiple times; for example, when a source historian has its data modified after it was already reported. And, as you remarked, the server timestamp is something that the receiving historian must assign when "the Server received a Variable value", it is not a trivial question whether it should keep the server timestamp it originally assigned, or whether it should assign a new server timestamp to each updated value it receives from the source. A manual update is also part of the picture, but this is easier in the sense that the decision about the server timestamp can be delegated to the user.
So, for example, I have to consider assigning server timestamp X to a value previously stored with server timestamp V, V < X, at which point I might decide that the modification time W = X, or X < W.
05/30/2017
Offline
And, as you remarked, the server timestamp is something that the receiving historian must assign when "the Server received a Variable value".
If the source has a ServerTimestamp should be the ServerTimestamp provided by the source and it should never change.
If the source does not have a ServerTimestamp should be set consistently based on rules set by the historian. The default rule should be to set ServerTimestamp = SourceTimestamp.
That said, the specification does not prohibit setting the ServerTimestamp based on when the value was added to the archive which could make it the same as the modification time.
To keep things simple I would just use ServerTimestamp = SourceTimestamp when no ServerTimestamp is provided by the data source.
03/24/2017
Offline
Randy Armstrong said
If the source has a ServerTimestamp should be the ServerTimestamp provided by the source and it should never change.
I believe this contradicts directly Part 4 Section 7.7.4 (also quoted previously): "In the case where the OPC UA Server subscribes to a value from another OPC UA Server, each Server applies its own serverTimestamp." Actually, this seems to contradict more than just this one sentence.
To keep things simple I would just use ServerTimestamp = SourceTimestamp when no ServerTimestamp is provided by the data source.
This, too, seems contradictory: "The serverTimestamp is used to reflect the time that the Server received a Variable value or knew
it to be accurate." Quite obviously, if a server reads and persists a value from another historian, the sourceTimestamp of the value will almost always be different from "the time that the Server received a Variable value".
As far as I can tell, the spec prescribes that the server timestamp, if it is used at all, shall be issued locally and never copied. Whether modified data may have different server timestamps (while having the same source timestamp) does not seem addressed by the spec, which I understand implies this is not disallowed, and whether the modification time, again if supported, may be the same as the server timestamp, is likewise not specified.
My conclusion is that whenever a server sees any value, it is legitimate to record it with the current server time as the server timestamp in the history, and if that results in multiple same tag/same source timestamp values thus recorded, the server timestamp can be returned as the modification time when the server is queried for modified data. To be clear, the server timestamp of the "next" version of the data should be returned as the modification time.
05/30/2017
Offline
In the case where the OPC UA Server subscribes to a value from another OPC UA Server
The implication is the UA server is aggregating server that makes the live value visible in its address space.
In your case, the live values are saved in an archive which is different. In this case, there is useful information that can be preserved by saving the ServerTimestamp.
As far as I can tell, the spec prescribes that the server timestamp, if it is used at all, shall be issued locally and never copied.
All these phrasing refer to chains of servers that expose the same value for downstream clients. A historian may only be a OPC UA client which means none of those statements apply. Even when a historian is also an aggregator that makes the live data available that servertimestamp would be set by the historian when sending live data to clients but that does not imply any requirements for storing historical data in the archive.
Moderators-Specifications
Moderators-Companion
Moderators-Implementation
Moderators-Certification
Moderators-COM
02/24/2014
Offline
Lets clarify some more:
your historian is aggregating data from other servers - sometimes it is historical data, sometimes it is collecting live data. The rules are different for this two cases:
I'm a normal server and I'm collecting live data from other servers (which I also historize) then I set the server timestamp when I get a value. I store the value in my historian (no modification record - it is just normal historical processing). This is the normal rules for aggregating servers. values are current (i.e. my timestamp is current time and the server timestamp on the incoming data is also near current time (subscription delay only)
I'm a historian and I'm pulling historical data from an underlining server (these days there are many device that have short term historical data, maybe an hour) so when I pull this data I store it in my server as historical data (I do not change this data - it is store as it was in the underlining server). Again I add no modification (I did not change a value). If the underlining historian had modification data - I should also maintain that data (un-changed). If this underlining server is only storing SourceTimestamp and your historian also requires ServerTimestamp (i.e. you can not store it without) then as Randy suggested - ServerTimestamp = SourceTimestamp. You never saw the live value so you have no clue that it was valid at any point in time. [this is for data that I obtained using historical access or some proprietary bulk transfer mechanism]. Key point I get blocks of historical data, I just store the block - don't touch it.
The modification is really used for industries where any change in a value that is not the normal process of live data collection has to be tracked and justified (i.e. I have a sensor that was broken and it provided no data, but for my report it is required. i.e. a field tech took manual reading and entered the data in the historian - this is modification and has information related to why it was inserted or changed) Modification is not the normal process of storing historical data.
An aggregating historical server that is collect data from underlining historian is not manipulating the values in any manner - it is just moving where that data is stored - so the data does not change. This is the same as a single historian that has on-line storage and an off-line storage. When it moves data to an off-line storage it does not change anything about the data - just where it is stored. (These days with the available disk sizes off-line storage is needed less often, but it still can occur - i.e. a removeable hard drive that when full is swapped out. Some processes require the historical data to be kept around for very long periods of time (the pharma industry is an example that can have the no modification and very long term storage requirements).
So one last case you described - is what happens if I have two sources for the same item - I get the live data and store it and I also get the historical data from the same underlining server? A common use case for this is a historian that can back fill data for when there is a communication loss between the server being aggregated and the long term historian server (or aggregation server that is also storing history). Then historical data is current as long as the connection is live, but when the data goes back quality (lost connection) then at some point later in time it backfills from the underlining server when the that server become accessible again. In this case what usually happens is the bad quality value is replaced (with a modification ) with the good quality value at that same source timestamp - but the other data is unchanged. (I would not be pulling historical data unless I needed to backfill - mostly for a lot of performance related issues) Again you are trying to report the state of the process, it is less important to collect data on when it was stored. In the vast majority of reports and uses of historical data only the source Timestamp is used (very rarely is someone concerned about when something was collected vs what was the value of the actual process - i.e. source timestamp show when it actually changed.)
It is more important that historical processing handle the retrieval processing correctly - be able to provided the boundary values - I.e. I want the value at 10:00:00 - the values stored were at 9:58:00 value 10.0 and then again at 10:02:00 value 15.0. Is the data exception reported for a slope or is it a step change? what did the underlining server if you pulled historical data from it? based on the answers the value at 10:00:)0 maybe 10.0 ot it maybe 12.5 (sloped line between the two points). This as stated in the spec is only about source timestamps.
Paul Hunkar - DSInteroperability
03/24/2017
Offline
Again I add no modification (I did not change a value). If the underlining historian had modification data - I should also maintain that data (un-changed).
The modification is really used for industries where any change in a value that is not the normal process of live data collection has to be tracked and justified (i.e. I have a sensor that was broken and it provided no data, but for my report it is required. i.e. a field tech took manual reading and entered the data in the historian - this is modification and has information related to why it was inserted or changed) Modification is not the normal process of storing historical data.
In a perfect world, indeed, I would just store the data, including any modified data records exactly like they were stored by the "source" historian.
My reality is not so perfect. As the UA spec says, read-processed or read-raw should return the latest version of data. The implication is that two subsequent reads may return different values (at the same tag and source timestamp), and the modified data may just not be maintained by the source historian. If it is read-processed that is used to derive the "original" data, the very concept of modified processed data does not exist. The data may not even come from an OPC UA source/historian, or, even if it ultimately comes from one, it may be transported via some other means, which will erase most of its OPC UA traits. The transport may be very dumb, too. It can retrieve a block of data, send it, and crash before it has a chance to record what it was that it forwarded last, and, upon restart, send the "same" block of data, except it may now have different values (at the same tag and source timestamp).
The bottom line is that I cannot prevent my historian from seeing (at least in theory) multiple values for the same tag/same source timestamp, while having no clue why they are different.
I can do basically two major things: keep one value per tag/source timestamp (and reject all the others); or keep multiple values. Unfortunately, no matter how I choose what value I get to keep, some customers are not happy that they cannot consult what other values (and when) were suppressed, so keeping all the values is what they want (sometimes). In fact, it is they who may have intelligence unavailable to the historian as to why the data were changing, and they can annotate and massage the data further - but first they want the historian to preserve the data.
Whether this sits well with the OPC UA concept of modified historical data is a good question, but I am afraid there is nothing else in OPC UA that would at least overlap with that.
05/30/2017
Offline
It is not clear what the issue is.
If there are multiple values at the same timestamp then one is the 'current' value and the rest are modified values.
If a client needs to create snapshots at different time then it can construct any time series it needs by using the modification time.
i.e. not every data analysis function needs to be done in the server. Sometimes clients can do the analysis after fetching data.
if you really do not know which value at a timestamp is the 'current' value you have to set the status to Uncertain or allow a client to update the value to the correct value.
Moderators-Specifications
Moderators-Companion
Moderators-Implementation
Moderators-Certification
Moderators-COM
02/24/2014
Offline
When you are aggregating data from another historian into your historian, you should only be using ReadRaw, reading processed data is performing calculations on the data (thus already changing it). Further when reading Raw values - you should not be asking for bounding values (since again they are calculated values - not real data but the result of a calculation) and should not be placed into your historian as if it was Raw data.
If you get the same sourceTimestamp a second time and the Value/status/serverTimestamp are the same, then I would just discard it. It is just a duplication of what was recieved. If the data came with modification information theni would save it since it is additional information.
I understand that if you are getting historical data from other sources (not OPC) it may be missing semantics. The discussion was about timestamps - I would not be storing a ServerTimestamp if I did not get one from the underlining source - it is extra information and making up some fictitious time stamp would not be good - if you don't know when the server that collected the actual data got a value - generating a ServerTimestamp with some later time does not make sense - as stated before if you require both timestamps then the only option is setting ServerTimestamp - sourceTimestamp. If you get data for a node with the same source timestamp (and no server Timestamp and no Modification information) but it has a different value. I would be hard-pressed to call either value as the current value and would agree with Randy and mark them both as uncertain - since you really don't know what the value actually was at the given sourceTimestamp.
Paul Hunkar - DSInteroperability
1 Guest(s)
Usage Policy