Writing Instances: Upsert¶

We assume that you have generated a SDK for the WindTurbine model and have a client ready to go.

The SDK generated by SDK supports nested upsert

In [2]:

from wind_turbine import WindTurbineClient

from wind_turbine import WindTurbineClient

In [3]:

pygen = WindTurbineClient.from_toml("config.toml")

pygen = WindTurbineClient.from_toml("config.toml")

Constructing new Turbine¶

When constructing a new WindTurbine we need to use the generated data classes from pygen. We can import this as follows:

In [4]:

from wind_turbine import data_classes as data_cls

from wind_turbine import data_classes as data_cls

The location of the data classes is determined by the parameter top_level_package which is set when you generate the SDK, either using the generate_sdk_notebook (simplified wrapper around) or generate_sdk. If you don't set it, it will be default set to the external_id of the data model converted to snake_case. For this example, the external_id=WindTurbine thus the top_level_package = wind_turbine.

Lets construct a new windmill with TimeSeries. Note the example below is not complete (some TimeSeries and components are missing), but is kept short to make it easier to grasp

In [5]:

new_turbine = data_cls.WindTurbineWrite(
    external_id="windmill:demo",
    capacity=10.0,
    windfarm="Fornebu",
    name="Windmill ATH",
    rotor=data_cls.RotorWrite(
        external_id="windmill:demo:rotor",
    ),
    nacelle=data_cls.NacelleWrite(
        external_id="windmill:demo:nacelle",
    ),
    blades=[
        data_cls.BladeWrite(
            external_id="windmill:demo:blade1",
            is_damaged=False,
            name="Blade 1",
        ),
        data_cls.BladeWrite(
            external_id="windmill:demo:blade2",
            is_damaged=False,
            name="Blade 2",
        ),
        data_cls.BladeWrite(
            external_id="windmill:demo:blade3",
            is_damaged=True,
            name="Blade 3",
        ),
    ],
)

new_turbine = data_cls.WindTurbineWrite( external_id="windmill:demo", capacity=10.0, windfarm="Fornebu", name="Windmill ATH", rotor=data_cls.RotorWrite( external_id="windmill:demo:rotor", ), nacelle=data_cls.NacelleWrite( external_id="windmill:demo:nacelle", ), blades=[ data_cls.BladeWrite( external_id="windmill:demo:blade1", is_damaged=False, name="Blade 1", ), data_cls.BladeWrite( external_id="windmill:demo:blade2", is_damaged=False, name="Blade 2", ), data_cls.BladeWrite( external_id="windmill:demo:blade3", is_damaged=True, name="Blade 3", ), ], )

When writing nested data we can specify edges either with an external id for the end node, or another data class.

The advangage of using a nested data class is that we can express edges without being explicit. In the example above, we are able to express that the blades Blade 1-3 are connected to the windmill windmill ATH and that the windmill ATH is also linked to a nacelle and rotor.

Inspecting Resources to create¶

We can inspect the nodes, edges and other resources that will be created by using the .to_instances_write on the new wind turbine object.

In [6]:

resources = new_turbine.to_instances_write()

resources = new_turbine.to_instances_write()

In [7]:

len(resources.nodes), len(resources.edges), len(resources.time_series)

len(resources.nodes), len(resources.edges), len(resources.time_series)

Out[7]:

(6, 0, 0)

In [8]:

resources.nodes

resources.nodes

Out[8]:

	space	external_id	instance_type	sources
0	sp_wind	windmill:demo	node	[{'properties': {'blades': [{'space': 'sp_wind...
1	sp_wind	windmill:demo:blade1	node	[{'properties': {'is_damaged': False, 'name': ...
2	sp_wind	windmill:demo:blade2	node	[{'properties': {'is_damaged': False, 'name': ...
3	sp_wind	windmill:demo:blade3	node	[{'properties': {'is_damaged': True, 'name': '...
4	sp_wind	windmill:demo:nacelle	node	NaN
5	sp_wind	windmill:demo:rotor	node	NaN

In [9]:

resources.edges

resources.edges

Out[9]:

In [10]:

resources.time_series

resources.time_series

Out[10]:

Creating new WindTurbine¶

Optinal Reading: Why pygen.upsert and not pygen.wind_turbine.upsert?

In contrast from other methods, the .upsert method is on the pygen instead of the individual API class. So instead of pygen.wind_turbine.upsert, we use pygen.upsert.

The reason for this is that the new_turbine we created above is enhanced by pygen with all the information needed to write it correctly to our data model. This means that all .upsert methods are the same, this is in contrast to methods such as .list and .retrieve which are specialized for each data type.

Furthermore, the reason for not duplicating the .upsert methods on each API class (pygen.wind_turbine.upsert, pygen.blade.upsert, and so on) is that encourages an anti-pattern (bad practice), in which nodes and edges are created in small batches. It is much more efficient to create all nodes and edges in as few batches as possible.

In [11]:

created = pygen.upsert(new_turbine)

created = pygen.upsert(new_turbine)

Note that the call above created 6 nodes. If you had linked metmasts, it would have created edges as well.

In [12]:

created.nodes

created.nodes

Out[12]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo	1	True	2024-12-29 11:23:45.015	2024-12-29 11:23:45.015
1	sp_wind	node	windmill:demo:blade1	1	True	2024-12-29 11:23:45.015	2024-12-29 11:23:45.015
2	sp_wind	node	windmill:demo:blade2	1	True	2024-12-29 11:23:45.015	2024-12-29 11:23:45.015
3	sp_wind	node	windmill:demo:blade3	1	True	2024-12-29 11:23:45.015	2024-12-29 11:23:45.015
4	sp_wind	node	windmill:demo:nacelle	1	False	2024-11-16 17:30:21.812	2024-11-16 17:30:21.812
5	sp_wind	node	windmill:demo:rotor	1	False	2024-11-16 17:30:21.812	2024-11-16 17:30:21.812

In [13]:

created.edges

created.edges

Out[13]:

In [14]:

created.time_series

created.time_series

Out[14]:

We can inspect the newly created windmill by calling retrieve with the external id

In [15]:

pygen.wind_turbine.retrieve(new_turbine.external_id)

pygen.wind_turbine.retrieve(new_turbine.external_id)

Out[15]:

	value
space	sp_wind
external_id	windmill:demo
data_record	{'version': 1, 'last_updated_time': 2024-12-29...
node_type	None
capacity	10.0
description	None
name	Windmill ATH
blades	[windmill:demo:blade1, windmill:demo:blade2, w...
datasheets	[]
metmast	None
nacelle	windmill:demo:nacelle
power_curve	None
rotor	windmill:demo:rotor
windfarm	Fornebu

In [ ]:

Upsert Parameters replace, None values and allow_version_increase¶

The upsert method have several parameters that control how the upsert call should be done. In this section, we will go through each of these flags.

Parameter: replace¶

The replace flag decide what to do if the item we are upserting already exists. If replace is set to True all properties of the existing item will be replaced by the properties set in the upsert call, and the properties not included will be to null. If replace is set to False, then only the properties included in the upsert call will be updated.

Let's demonstrate this by creating a new blade and update it. We start by creating a new blade and call upsert on it.

A blade has two properties: name and is_damaged. In addition, it has edges to the sensor_positions connected to the blade. In this example, we will focus on the two properties, which are both nullable, meaning that they are optional.

In [16]:

from wind_turbine import data_classes as data_cls

from wind_turbine import data_classes as data_cls

In [17]:

new_blade = data_cls.BladeWrite(
    external_id="windmill:demo:blade4",
    name="Demo Blade",
)

new_blade = data_cls.BladeWrite( external_id="windmill:demo:blade4", name="Demo Blade", )

In [18]:

created_blade = pygen.upsert(new_blade)

created_blade = pygen.upsert(new_blade)

In [19]:

created_blade.nodes

created_blade.nodes

Out[19]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo:blade4	1	True	2024-12-29 11:24:55.634	2024-12-29 11:24:55.634

In [20]:

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)

In [21]:

retrieved_blade

retrieved_blade

Out[21]:

	value
space	sp_wind
external_id	windmill:demo:blade4
data_record	{'version': 1, 'last_updated_time': 2024-12-29...
node_type	None
is_damaged	None
name	Demo Blade
sensor_positions	None

We can see that the is_damaged property is not set, while we have the name property set. We will now update the blade with the is_damaged property set to True.

In [22]:

updated_blade = data_cls.BladeWrite(
    external_id="windmill:demo:blade4",
    is_damaged=True,
)

updated_blade = data_cls.BladeWrite( external_id="windmill:demo:blade4", is_damaged=True, )

In [23]:

update = pygen.upsert(updated_blade, replace=False)

update = pygen.upsert(updated_blade, replace=False)

In [24]:

update.nodes

update.nodes

Out[24]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo:blade4	2	True	2024-12-29 11:25:36.498	2024-12-29 11:24:55.634

In [25]:

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)

In [26]:

retrieved_blade

retrieved_blade

Out[26]:

	value
space	sp_wind
external_id	windmill:demo:blade4
data_record	{'version': 2, 'last_updated_time': 2024-12-29...
node_type	None
is_damaged	True
name	Demo Blade
sensor_positions	None

We see that the blade property is_damaged is now set to True, while the name property is unchanged. This is because we set replace to False. If we set replace to True, then the name property would be set to null. Let's create a new update were we set the is_damaged property to False and use replace set to True.

In [27]:

blade_update2 = data_cls.BladeWrite(
    external_id="windmill:demo:blade4",
    is_damaged=False,
)

blade_update2 = data_cls.BladeWrite( external_id="windmill:demo:blade4", is_damaged=False, )

In [28]:

update2 = pygen.upsert(blade_update2, replace=True)

update2 = pygen.upsert(blade_update2, replace=True)

In [29]:

update2.nodes

update2.nodes

Out[29]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo:blade4	3	True	2024-12-29 11:27:00.629	2024-12-29 11:24:55.634

In [30]:

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)

In [31]:

retrieved_blade

retrieved_blade

Out[31]:

	value
space	sp_wind
external_id	windmill:demo:blade4
data_record	{'version': 3, 'last_updated_time': 2024-12-29...
node_type	None
is_damaged	False
name	None
sensor_positions	None

Notice that the name property is now set to null as we set replace to True.

In [32]:

# Cleanup
pygen.delete(new_blade.external_id)

# Cleanup pygen.delete(new_blade.external_id)

Out[32]:

InstancesDeleteResult(nodes=[NodeId(space='sp_wind', external_id='windmill:demo:blade4')], edges=[])

None values¶

pygen automatically detects the nullable values that are set to None and sends them to CDF.

Let's demonstrate this by creating a new blade with the is_damaged property set to True, and then update it to be not set.

In [41]:

new_blade = data_cls.BladeWrite(external_id="windmill:demo:write_none", name="Demo Blade", is_damaged=True)

new_blade = data_cls.BladeWrite(external_id="windmill:demo:write_none", name="Demo Blade", is_damaged=True)

In [42]:

pygen.upsert(new_blade).nodes

pygen.upsert(new_blade).nodes

Out[42]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo:write_none	1	True	2024-12-29 11:32:34.539	2024-12-29 11:32:34.539

In [43]:

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)
retrieved_blade

retrieved_blade = pygen.blade.retrieve(new_blade.external_id) retrieved_blade

Out[43]:

	value
space	sp_wind
external_id	windmill:demo:write_none
data_record	{'version': 1, 'last_updated_time': 2024-12-29...
node_type	None
is_damaged	True
name	Demo Blade
sensor_positions	None

In [44]:

updated_blade = data_cls.BladeWrite(
    external_id="windmill:demo:write_none",
    is_damaged=None,
)

updated_blade = data_cls.BladeWrite( external_id="windmill:demo:write_none", is_damaged=None, )

In [45]:

pygen.upsert(updated_blade).nodes

pygen.upsert(updated_blade).nodes

Out[45]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo:write_none	2	True	2024-12-29 11:32:37.399	2024-12-29 11:32:34.539

In [46]:

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)
retrieved_blade

retrieved_blade = pygen.blade.retrieve(new_blade.external_id) retrieved_blade

Out[46]:

	value
space	sp_wind
external_id	windmill:demo:write_none
data_record	{'version': 2, 'last_updated_time': 2024-12-29...
node_type	None
is_damaged	None
name	Demo Blade
sensor_positions	None

Notice that the is_damaged property is now set to None.

In [47]:

# Cleanup
pygen.delete("windmill:demo:write_none")

# Cleanup pygen.delete("windmill:demo:write_none")

Out[47]:

InstancesDeleteResult(nodes=[NodeId(space='sp_wind', external_id='windmill:demo:write_none')], edges=[])

Parameter: allow_version_increase¶

If you notice in the last examples, that when updating the blade, the version of the returning node is increasing. This is because each time we do a change to the blade node it is registered and the version is increased. You can control this behavior by setting the existing_version property in data_record of the blade node. Let's demonstrate with an example

In [89]:

from cognite.client.exceptions import CogniteAPIError
from wind_turbine import data_classes as data_cls

from cognite.client.exceptions import CogniteAPIError from wind_turbine import data_classes as data_cls

Introduction¶

In [90]:

new_blade = data_cls.BladeWrite(external_id="windmill:demo:allow_version_increase", name="Demo Blade", is_damaged=True)

new_blade = data_cls.BladeWrite(external_id="windmill:demo:allow_version_increase", name="Demo Blade", is_damaged=True)

In [91]:

pygen.upsert(new_blade).nodes

pygen.upsert(new_blade).nodes

Out[91]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo:allow_version_increase	1	True	2024-12-29 11:39:38.345	2024-12-29 11:39:38.345

We see that we have version 1 of the blade. If we want to avoid overwriting this blade by accident, we can set the existing_version property to lower than the version we want to avoid overwriting in the data_record of the blade node. This will be 0 in this case.

In [92]:

new_blade2 = data_cls.BladeWrite(
    external_id="windmill:demo:allow_version_increase",
    is_damaged=False,
    data_record=data_cls.DataRecordWrite(existing_version=0),
)

new_blade2 = data_cls.BladeWrite( external_id="windmill:demo:allow_version_increase", is_damaged=False, data_record=data_cls.DataRecordWrite(existing_version=0), )

In [93]:

try:
    pygen.upsert(new_blade2)
except CogniteAPIError as e:
    print(e)

try: pygen.upsert(new_blade2) except CogniteAPIError as e: print(e)

A version conflict caused the ingest to fail. | code: 400 | X-Request-ID: e69cddfb-5fdd-9e7d-bce4-20cedee43c01 | cluster: greenfield
The API Failed to process some items.
Successful (2xx): []
Unknown (5xx): []
Failed (4xx): [sp_wind:windmill:demo:allow_version_increase, ...]

Allow Version Increase¶

This can cause problems when we want to migrate data from one project to another, or from one data model to another. It is a common pattern that we use the pygen generated SDK to retrieve from one project and then use the .as_write method to turn the retrieved read format of a node into the write format. We want to ensure we always will overwrite the nodes in the new project. Then, we can use the allow_version_increase flag to ensure that we always overwrite the all the nodes and edges we are writing will have set existing_version to None which will ensure that we always overwrite the nodes and edges.

First, we retrieve the blade.

In [97]:

retrieved_blade = pygen.blade.retrieve(new_blade.external_id)
retrieved_blade

retrieved_blade = pygen.blade.retrieve(new_blade.external_id) retrieved_blade

Out[97]:

	value
space	sp_wind
external_id	windmill:demo:allow_version_increase
data_record	{'version': 1, 'last_updated_time': 2024-12-29...
node_type	None
is_damaged	True
name	Demo Blade
sensor_positions	None

Before we write the blade back, somebody else updates it:

In [98]:

pygen.upsert(
    data_cls.BladeWrite(external_id="windmill:demo:allow_version_increase", name="Updated", is_damaged=True)
).nodes

pygen.upsert( data_cls.BladeWrite(external_id="windmill:demo:allow_version_increase", name="Updated", is_damaged=True) ).nodes

Out[98]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo:allow_version_increase	2	True	2024-12-29 11:42:59.519	2024-12-29 11:39:38.345

Then, we try to write back the retrieved blade

In [99]:

writeable_blade = retrieved_blade.as_write()

writeable_blade = retrieved_blade.as_write()

In [100]:

writeable_blade

writeable_blade

Out[100]:

	value
space	sp_wind
external_id	windmill:demo:allow_version_increase
data_record	{'existing_version': 1}
node_type	None
is_damaged	True
name	Demo Blade

In [101]:

try:
    pygen.upsert(writeable_blade)
except CogniteAPIError as e:
    print(e)

try: pygen.upsert(writeable_blade) except CogniteAPIError as e: print(e)

A version conflict caused the ingest to fail. | code: 400 | X-Request-ID: 9de7d7cf-6b6e-9f2f-ba3f-6b61ec243363 | cluster: greenfield
The API Failed to process some items.
Successful (2xx): []
Unknown (5xx): []
Failed (4xx): [sp_wind:windmill:demo:allow_version_increase, ...]

We force the issue by setting allow_version_increase=True

In [102]:

pygen.upsert(writeable_blade, allow_version_increase=True).nodes

pygen.upsert(writeable_blade, allow_version_increase=True).nodes

Out[102]:

	space	instance_type	external_id	version	was_modified	last_updated_time	created_time
0	sp_wind	node	windmill:demo:allow_version_increase	3	True	2024-12-29 11:43:52.328	2024-12-29 11:39:38.345

We see that the allow_version_increase flag ensures that we always overwrite the nodes and edges.

In [103]:

# Cleanup
pygen.delete("windmill:demo:allow_version_increase")

# Cleanup pygen.delete("windmill:demo:allow_version_increase")

Out[103]:

InstancesDeleteResult(nodes=[NodeId(space='sp_wind', external_id='windmill:demo:allow_version_increase')], edges=[])

Creating from JSON Format¶

See the quick start guide data population for an example of creating instances from JSON.

Deleting Instances¶

You can delete by passing and external ID or an Item or a sequence of external id or items to the delete method.

We can delete the newly created windmill

In [74]:

pygen.wind_turbine.list()

pygen.wind_turbine.list()

Out[74]:

	space	external_id	capacity	name	blades	datasheets	nacelle	rotor	windfarm	data_record
0	sp_wind	hornsea_1_mill_3	7.0	hornsea_1_mill_3	[hornsea_1_mill_3_blade_A, hornsea_1_mill_3_bl...	[windmill_schematics]	hornsea_1_mill_3_nacelle	hornsea_1_mill_3_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
1	sp_wind	hornsea_1_mill_2	7.0	hornsea_1_mill_2	[hornsea_1_mill_2_blade_B, hornsea_1_mill_2_bl...	[windmill_schematics]	hornsea_1_mill_2_nacelle	hornsea_1_mill_2_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
2	sp_wind	hornsea_1_mill_1	7.0	hornsea_1_mill_1	[hornsea_1_mill_1_blade_A, hornsea_1_mill_1_bl...	[windmill_schematics]	hornsea_1_mill_1_nacelle	hornsea_1_mill_1_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
3	sp_wind	hornsea_1_mill_4	7.0	hornsea_1_mill_4	[hornsea_1_mill_4_blade_C, hornsea_1_mill_4_bl...	[windmill_schematics]	hornsea_1_mill_4_nacelle	hornsea_1_mill_4_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
4	sp_wind	hornsea_1_mill_5	7.0	hornsea_1_mill_5	[hornsea_1_mill_5_blade_B, hornsea_1_mill_5_bl...	[windmill_schematics]	hornsea_1_mill_5_nacelle	hornsea_1_mill_5_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
5	sp_wind	windmill:demo	10.0	Windmill ATH	[windmill:demo:blade1, windmill:demo:blade2, w...	[]	windmill:demo:nacelle	windmill:demo:rotor	Fornebu	{'version': 1, 'last_updated_time': 2024-12-29...

Same as .upsert, the delete method is located on the pygen and not the API class.

In [75]:

deleted = pygen.delete(new_turbine)

deleted = pygen.delete(new_turbine)

In [76]:

len(deleted.nodes), len(deleted.edges)

len(deleted.nodes), len(deleted.edges)

Out[76]:

(6, 0)

After the delete call the new windmill is gone

In [77]:

pygen.wind_turbine.list()

pygen.wind_turbine.list()

Out[77]:

	space	external_id	capacity	name	blades	datasheets	nacelle	rotor	windfarm	data_record
0	sp_wind	hornsea_1_mill_3	7.0	hornsea_1_mill_3	[hornsea_1_mill_3_blade_A, hornsea_1_mill_3_bl...	[windmill_schematics]	hornsea_1_mill_3_nacelle	hornsea_1_mill_3_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
1	sp_wind	hornsea_1_mill_2	7.0	hornsea_1_mill_2	[hornsea_1_mill_2_blade_B, hornsea_1_mill_2_bl...	[windmill_schematics]	hornsea_1_mill_2_nacelle	hornsea_1_mill_2_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
2	sp_wind	hornsea_1_mill_1	7.0	hornsea_1_mill_1	[hornsea_1_mill_1_blade_A, hornsea_1_mill_1_bl...	[windmill_schematics]	hornsea_1_mill_1_nacelle	hornsea_1_mill_1_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
3	sp_wind	hornsea_1_mill_4	7.0	hornsea_1_mill_4	[hornsea_1_mill_4_blade_C, hornsea_1_mill_4_bl...	[windmill_schematics]	hornsea_1_mill_4_nacelle	hornsea_1_mill_4_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...
4	sp_wind	hornsea_1_mill_5	7.0	hornsea_1_mill_5	[hornsea_1_mill_5_blade_B, hornsea_1_mill_5_bl...	[windmill_schematics]	hornsea_1_mill_5_nacelle	hornsea_1_mill_5_rotor	Hornsea 1	{'version': 8, 'last_updated_time': 2024-12-17...

In addition, since we passed in the entire object and not just an external ID, all the edges and nodes connected to the windmill:demo was also deleted.

In [ ]: