Note
For more information about ETAG, refer to the RFC7252 Section 5.10.6.
An entity-tag (ETAG) is used to identify changes of the resource that happen over time. ETAG is generated by the resource server, in our case device, using various methods such as a version, checksum, hash or a timestamp. When the client retrieves an entity-tag, it should treat it as an opaque value without making any assumptions about its content or structure. In CoAP, the ETAG is part of the CoAP options and is used to detect changes in resources. It is used to optimize the client processing time of the response from the device in case no change was detected - the ETAG wasn’t changed from the last request.
For more information about ETAG, refer to the RFC7252 Section 5.10.6.
To enable the ETAG feature in IoTivity-lite, use the CMake option -DOC_ETAG_ENABLED=ON.
In IoTivity-lite, the ETAG is generated using the following algorithm:
By following this algorithm, the ETAG is always increasing and unique for each changed resource among all resources in the device.
When handling load operations, the global ETAG value is determined by taking the maximum value among all existing ETAGs, combining it with the current time, and adding a random number (refer to oc_etag_load_and_clear). The inclusion of the time component serves to minimize potential collisions between the ETAGs of device resources and the cached ETAGs stored in clients. Additionally, the introduction of a random number introduces an element of uncertainty to the ETAG, particularly in cases where time synchronization is not precise.
Collision example: Consider a device that always sets the time to 1.1.1970 due to the lack of a real-time clock. This device has only one modifiable resource. Let’s say the device is running with an updated resource that has an ETAG value of 1000, and the client also has the same ETAG value of 1000 cached. However, due to a power loss and subsequent restart (with no stored ETAGs in persistent storage), the random number generator returns 1, which is the same as the previous run. Meanwhile, another client updates the resource at the exact same time as the previous run but modifies a different property, resulting in an ETAG of 1000 for that resource.
The initial client, without knowledge of the resource modification made by the second client, sends a request to access the resource using the previously cached ETAG value of 1000. The device responds with a VALID code, indicating that the ETAG remains valid. Nonetheless, the first client remains unaware that the resource has been altered by another client, and the device’s resource continues to possess the same ETAG.
static uint64_t g_etag;
static uint64_t etag_random() {
return (oc_random_value()%1000)+1;
}
void oc_etag_init()
{
g_etag = oc_clock_time() + etag_random();
}
// Get new ETAG for changed resource
uint64_t oc_etag_get(void)
{
uint64_t now = oc_clock_time();
if (now > g_etag)
{
g_etag = now;
}
g_etag = g_etag + etag_random();
return g_etag;
}
There are two methods for updating the ETAG when a resource changes:
oc_notify_resource_changed: This function internally triggers oc_notify_observers for the observable resource and oc_resource_update_etag.oc_resource_update_etag: This function updates the ETAG without informing observers about the change in content. It can be beneficial for resources that are not observable.In order to improve network traffic efficiency, clients have the option to store received ETAGs from GET responses in a cache. To determine if the ETAG is still valid, the client can send the ETAGs in a request to the device. If the ETAG is valid, the device will respond with a VALID code and no body, and corresponding matched ETAG. If the ETAG is not valid, the device will respond with a CONTENT code containing the current ETAG and the corresponding body.
For OCF interfaces, the ETAG remains unaffected even if different representations of the resource are available, except for the batch interface (oic.if.b). The ETAG is specifically tied to the state of the resource itself, regardless of the interface used.
When accessing the /oic/res resource, the ETAG represents the highest ETAG value among all the device resources in that collection and the collection ETAG itself. If multiple resources within the collection undergo simultaneous changes, the ETAG will be set to the highest value among all the modified resources.
If a client sends a GET request with an ETAG that matches the highest ETAG value among all the resources, the response will consist of a VALID code without a body. However, if the ETAG provided by the client does not match the highest ETAG value, the response will include a CONTENT code along with the content of all the resources and the highest ETAG value among them.
To support the use of ETAGs, the representation of the resource in the batch response has been extended by introducing the etag property, following this structure:
[
{
"href": "ocf://<device_id><href>",
"rep": {
...
},
"etag": "1-8 byte opaque value"
},
...
]
To identify the specific resource ETAG used in the batch ETAG response, the client should conduct a comparison between the ETAG value of each resource within the batch response and the ETAG value provided in the CoAP option by the client. In case the collection resource ETAG has been used, the ETAG is not included in the body response.
The incremental changes feature enables clients to request modifications to resources that have occurred since the most recent notification from observation or GET request. To request incremental changes from a device through a GET request, clients can include an ETAG-0 Value in the CoAP option, along with multiple CoAP query parameters using the following format: incChanges=<ETAG-1 Value in base64>,<ETAG-2 Value in base64>..<ETAG-20 Value in base64>, incChanges=<ETAG-21 Value in base64>,..<ETAG-40 Value in base64>,incChanges=.... Each ETAG provided by the client corresponds to a different resource’s ETAG, and the ETAG-0 Value in the option represents the latest ETAG Value among all resources. The ETag-0 value is encoded in binary format as it resides within the CoAP option, while the remaining ETag-1+ values are encoded in base64 format, separated by a comma ,. This choice is due to their storage in query parameters, which require compliance with UTF-8 encoding.
If we were to illustrate this in terms of HTTP, it would resemble the following:
GET /oic/res?incChanges=<ETAG-1 Value in base64>,..<ETAG-20 Value in base64>&incChanges... HTTP/1.1
ETag: <ETAG-0 Value>
If the value of ETAG-0 Value is the same as the highest ETAG among all the resources on the device, it indicates that the client is already up to date with the latest changes. Consequently, the device will respond with the VALID code, as mentioned in the previous section.
The query’s capacity to include ETAGs is limited to 255 due to restrictions on the option size and the maximum transmission unit (MTU) for datagrams in UDP transport, typically set at 1152 bytes in most cases. As a result, to ensure proper transmission, the entire CoAP packet size must not exceed the MTU. For optimal performance with a 1152-byte datagram, it is advisable to limit the usage to a maximum of 60 ETAGs (20 ETAGs per query).
In the case of TCP transport, the limit is determined by the protocol data unit size, referred to as OC_PDU_SIZE in iotivity-lite.
The device sends incremental changes (shown by the CONTENT response code) if the given ETAGs are smaller than the global ETAG. If one of the client’s ETAGs matches a device resource, the device will transmit all resources with higher ETAGs from that point. In short, you’ll receive only changed or all resources.
During device startup, the device developer loads the ETAGs using oc_etag_load_and_clear and clears the existing storage. Additionally, when properly shutting down IoTivity-lite, the ETAGs are stored in the storage using oc_etag_dump. Improper shutdown results in the ETAGs not being stored, leading to their absence upon restart. The device developer should call oc_etag_load_and_clear after creating all resources and before the first run by calling oc_main_poll_v1, and oc_etag_dump before calling oc_main_shutdown.
When utilizing the oc_etag_dump function to safeguard ETAGs, both the ETAG associated with each resource and the hash (SHA1) of its content are stored. Should the hash of the content differ from the stored hash during the loading process using oc_etag_load_and_clear, the existing ETAG is invalidated, prompting the creation of a fresh ETAG. The process of computing the hash entails initiating a GET request via the x.plgd.if.etag interface, if supported by the resource. Alternatively, the oic.if.baseline interface is employed. Notably, the x.plgd.if.etag interface allows for the exclusion of certain properties from the hash computation. For example, the /x.plgd.dev/time resource has a time property that consists of a timestamp. This specific property does not contribute to the hash computation.
int ret = oc_main_init(&handler);
...
// Create all custom resources
...
oc_etag_load_and_clear();
while (!quit)
{
oc_clock_time_t next_event_mt = oc_main_poll_v1();
...
}
oc_etag_dump();
...
// Remove all custom resources
...
oc_main_shutdown();
oc_etag_dump function to confirm the success of the operation. Storage failures often occur due to insufficient storage space.oc_etag_load_and_clear function, which requires the client to resynchronize with the device to establish a new global ETAG.oc_etag_load_and_clear process. This ensures that ETAGs assigned to previous resources are not reused by new resources, thereby avoiding conflicts and maintaining consistency.In order to monitor resource changes and determine if a resource has been modified on the device, the CoAP gateway utilizes the Entity Tag (ETAG) mechanism.
To enable the use of ETAGs in the plgd CoAP gateway, activate it by setting the value in the Helm chart: .coapgateway.deviceTwin.useETags: true.
For Batch Observation, the ETAG is associated with the overall state of resources. Prior to initiating resource observation, the CoAP gateway retrieves the latest ETAG for numbers of resource(N-latest ETAGs) among all device resources from the Hub Database. When initiating the resource observation, the CoAP gateway sends the ETAGs to the device with the query incChanges. If the received highest ETAG matches the highest ETAG among the device resources, the device responds with a code VALID. However, if the received ETAG does not match, the device responds with a code CONTENT and includes the current ETAG. Consequently, when a resource changes, the device sends the updated ETAG back to the CoAP gateway via a notification. The CoAP gateway transmits the ETAGs together with the Content by using the NotifyResourceChanged method to the resource-aggregate. This command is then converted into a ResourceChanged event, which is saved in a database and distributed through the event bus.
The special query to the database efficiently retrieves the N-latest ETAGs from all device resources without loading the complete set of data. This optimized query solely focuses on performance and retrieves only the required ETAGs, excluding any additional information.
The N-latest ETAGs retrieved from the database is configurable in CoAP gateway. In the worst-case scenario, the CoAP gateway retrieves N-latest ETAGs from the database, but none of them will be utilized in the batch observation.
To enable batch observation in IoTivity-lite, you need to activate it using the CMake option -DOC_DISCOVERY_RESOURCE_OBSERVABLE_ENABLED=ON.
CONTENT code, as described in the Incremental Changes section.To identify the resource responsible for updating the ETAG during batch observation, you can refer to the device’s debug logs. To enable the device debug logs, use the -DOC_DEBUG_ENABLED=ON flag, and set the filter function using oc_log_set_function.
plgd makes it simpler to build a successful IoT initiative – to create a proof of concept, evaluate, optimize, and scale.
