Performance issues (cue export takes hours) in evalv3 with loops and nested data structures #3879
-
|
Hi, The problem I encounter is related to loops in evalv3 I guess. To be concrete I have a heavily nested data structure which shows the hierachy of the Assets. A non exhaustive example of the datastructure: import "net"
#Location: [Location=string]: {
racks: [RackName=string]: #Rack
devices: [Name=string]: #Device
}
#Devices: [Region=string]: [SiteGroup=string]: [Site=string]: {
status: "active"
clusters?: #ClusterTypes
locations: #Location
vlans: #VLANs
}
#VLANs: [VLANName=string]: {
id!: number
description?: string
status: *"active" | string
role!: string
tenant!: string
vrfs: [VRF=string]: {
description?: string
unique: *true | bool
tenant?: string
prefixes: [Prefix=net.IPCIDR & string]: {
role?: string
status: string
description?: string
tenant?: string
}
}
}
#Device: {
device_role: string
device_type: string
face?: string
position?: number
status: *"active" | #DeviceStatus
tenant!: string
interfaces?: [IfName=string]: {
label: *"\(IfName)" | string
enabled: *true | bool
mac_address: string | =~"^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$"
mode: *"access" | "access" | "tagged" | "tagged-all"
mgmt: *false | bool
type!: string
vrf?: string
untagged_vlan?: number
tagged_vlans?: [...number]
}
...
}The actual data population is done with OpenTofu where I need easy maps which I can pass to for_each easier than those complex structures (which are more user friendly for our devs and show how the objects are related). So to achieve this I wrote a transformer "function", which loops the structure and the rearranges it into another format. Which #TransformLocations: this={
#input: {
locations!: #Location
site!: string
}
locations: #NetboxTerraformAPI_Devices_Schema.locations
racks: #NetboxTerraformAPI_Devices_Schema.racks
devices: #NetboxTerraformAPI_Devices_Schema.devices
for location_raw, location_data in this.#input.locations
let location = "\(#CreateSlug & {#input: "\(location_raw)", _})" {
locations: "\(location)": {
slug: location
site: this.#input.site
}
for device, device_data in location_data.devices {
devices: "\(device)": {
device_role: device_data.device_role
device_type: device_data.device_type
status: device_data.status
tenant: device_data.tenant
"location": "\(location)"
site: this.#input.site
if device_data.dell_servicetag != _|_ {
dell_expresscode: device_data.dell_expresscode
dell_servicetag: device_data.dell_servicetag
}
if device_data.dfs_equipment_number != _|_ {
dfs_equipment_number: device_data.dfs_equipment_number
dfs_asset_number: device_data.dfs_asset_number
dfs_asset_subnumber: device_data.dfs_asset_subnumber
}
if device_data.cluster != _|_ {
cluster: device_data.cluster
}
}
}
for rack, rack_data in location_data.racks {
racks: "\(rack)": {
"location": "\(location)"
site: this.#input.site
role: rack_data.role
tenant: rack_data.tenant
form_factor: rack_data.type
desc_units: true
width: rack_data.width
status: rack_data.status
u_height: rack_data.height
}
for device, device_data in rack_data.devices {
devices: "\(device)": {
device_role: device_data.device_role
device_type: device_data.device_type
status: device_data.status
tenant: device_data.tenant
rack_position: device_data.position
rack_face: device_data.face
"location": "\(location)"
"rack": rack
site: this.#input.site
if device_data.dell_servicetag != _|_ {
dell_expresscode: device_data.dell_expresscode
dell_servicetag: device_data.dell_servicetag
}
if device_data.dfs_equipment_number != _|_ {
dfs_equipment_number: device_data.dfs_equipment_number
dfs_asset_number: device_data.dfs_asset_number
dfs_asset_subnumber: device_data.dfs_asset_subnumber
}
if device_data.cluster != _|_ {
cluster: device_data.cluster
}
}
}
}
}
}
#TransformDeviceToTFApi: {
#input_devices: #Devices
#NetboxTerraformAPI_Devices_Schema
for region, region_data in #input_devices
for site_group, site_group_data in region_data
for site, site_data in site_group_data {
regions: (region): slug: #CreateSlug & {#input: region, _}
site_groups: (site_group): slug: #CreateSlug & {#input: site_group, _}
sites: site: {
slug: #CreateSlug & {#input: site, _}
status: site_data.status
"region": region
group: site_group
}
let _d = #TransformLocations & {#input: {locations: site_data.locations, "site": site}, _}
locations: _d.locations
racks: _d.racks
devices: _d.devices
if site_data.clusters != _|_
for cluster_type, cluster_type_data in site_data.clusters
for cluster_group, cluster_group_data in cluster_type_data
for cluster, cluster_data in cluster_group_data {
cluster_types: (cluster_type): slug: "\(#CreateSlug & {#input: cluster_type, _})"
cluster_groups: (cluster_group): slug: "\(#CreateSlug & {#input: cluster_group, _})"
clusters: (cluster): {
tenant: cluster_data.tenant
group: cluster_group
"site": site
type: cluster_type
if cluster_data.description != _|_ {
description: cluster_data.description
}
}
}
for vlan, vlan_data in site_data.vlans {
vlans: (vlan): {
vid: vlan_data.id
"site": site
if vlan_data.description != _|_ {
description: vlan_data.description
}
if vlan_data.tenant != _|_ {
tenant: vlan_data.tenant
}
if vlan_data.role != _|_ {
role: vlan_data.role
}
if vlan_data.status != _|_ {
status: vlan_data.status
}
}
for vrf, vrf_data in vlan_data.vrfs {
vrfs: (vrf): {
if vrf_data.description != _|_ {
description: vrf_data.description
}
if vrf_data.tenant != _|_ {
tenant: vrf_data.tenant
}
enforce_unique: vrf_data.unique
}
for prefix, prefix_data in vrf_data.prefixes {
prefixes: "\(prefix)-\(vrf)": {
"prefix": prefix
status: prefix_data.status
"site": site
"vlan": vlan
"vrf": vrf
if prefix_data.description != _|_ {
description: prefix_data.description
}
if prefix_data.role != _|_ {
role: prefix_data.role
}
if prefix_data.tenant != _|_ {
tenant: prefix_data.tenant
}
}
}
}
}
}
}I also think the problem decreases with less nesting in the loops. if site_data.clusters != _|_
for cluster_type, cluster_type_data in site_data.clusters
for cluster_group, cluster_group_data in cluster_type_data
for cluster, cluster_data in cluster_group_data {
cluster_types: (cluster_type): slug: "\(#CreateSlug & {#input: cluster_type, _})"
cluster_groups: (cluster_group): slug: "\(#CreateSlug & {#input: cluster_group, _})"
clusters: (cluster): {
tenant: cluster_data.tenant
group: cluster_group
"site": site
type: cluster_type
if cluster_data.description != _|_ {
description: cluster_data.description
}
}
} if site_data.clusters != _|_ {
for cluster_type, cluster_type_data in site_data.clusters {
cluster_types: (cluster_type): slug: "\(#CreateSlug & {#input: cluster_type, _})"
for cluster_group, cluster_group_data in cluster_type_data {
cluster_groups: (cluster_group): slug: "\(#CreateSlug & {#input: cluster_group, _})"
for cluster, cluster_data in cluster_group_data {
clusters: (cluster): {
tenant: cluster_data.tenant
group: cluster_group
"site": site
type: cluster_type
if cluster_data.description != _|_ {
description: cluster_data.description
}
}
}
}
}
} |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 7 replies
-
|
Can you clarify if you are trying v0.13.0-alpha.3 or a specific commit in master? When you evaluate with evalv3, how does the peak memory usage compare to evalv2? Are you able to share your configuration with us, perhaps privately via https://cue.dev/products/unity/ if it's on a GitHub repo? That would be the easiest way for us to be able to reproduce and investigate the issue. We are indeed aware of one remaining performance regression from evalv2 to evalv3, as evalv3 does not have memory management (i.e. memory reuse) yet, whereas evalv2 did have it. In some cases, this can lead to 10-100x more memory and wall time use on evalv3, as shown in #3334 (comment). We are looking into this optimization next. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, thanks for the quick reply. I'm on (which was master when I cloned and built it a few days ago) Memory is not that of an issue currently (but was before on 0.13.0 with evalv3 but unfortunately I don't remember the revision) only the export times when multiple loops and nesting is involved are strange in comparison to v2. Actually evalv2 seems to use more memory than v3. I cannot test the reality with v2 anymore because some modules don't work with v2 anymore due to errors when exporting. It is not on GitHub but I maybe could create an example which I can share. If I want to use unity, how can I use it in my repo? I already tried to use it and installed the application and authorized it for an repo. But how can I run it? |
Beta Was this translation helpful? Give feedback.
-
|
If you can create a small example from your config which shows evalv3 being orders of magnitude slower than evalv2, that would be really useful, as that's ultimately the kind of thing we want for a performance regression test. Regarding the Unity app - you basically install it on a repo containing your config or code, whether it's public or private, and then you tell us what command we can use to test your project, and in your case reproduce the performance regression. Usually it's something akin to |
Beta Was this translation helpful? Give feedback.
-
|
Hm ok, I got an example repo here: https://github.com/joelMuehlena/cue-netbox-example the output can be generated using |
Beta Was this translation helpful? Give feedback.
-
|
I just meassured with the real data again and got |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing https://github.com/joelMuehlena/cue-netbox-example, I've started looking into it. I ran your So the evalv3 slow-down appears to be about 4x. What is interesting is that this seems to relate to closedness; if I run the new evaluator without the closedness checks (i.e. I'm currently reducing to see if I can raise a small, self-contained performance issue about this. |
Beta Was this translation helpful? Give feedback.
-
|
The reductions below go towards memory management, i.e. #3334, and I think you're partially affected by that. However, I think the bigger issue here is a slow-down caused by closedness. As shown above, |
Beta Was this translation helpful? Give feedback.
-
|
I reduced it to the following: whose stats are as follows: As you can see, evalv3 still shows a 4x performance regression, even though the elapsed times are pretty small at this point. I think this is the lack of memory management in evalv3 described in #3334 (comment), as you can see in the Go stats that we allocate more than 3x as many objects, and more than 5x as many bytes. |
Beta Was this translation helpful? Give feedback.
-
|
I collected the same stats from your full repo: Similar results here; evalv3 does 2x as many Go allocs and 10x as many bytes, and you can see via v2's "reused" that memory management helped quite a lot. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, |
Beta Was this translation helpful? Give feedback.
-
|
Great - you can certainly use |
Beta Was this translation helpful? Give feedback.
-
|
@joelMuehlena FYI, https://review.gerrithub.io/c/cue-lang/cue/+/1213537 fixes another performance issue that is affecting your project. With |
Beta Was this translation helpful? Give feedback.
The reductions below go towards memory management, i.e. #3334, and I think you're partially affected by that.
However, I think the bigger issue here is a slow-down caused by closedness. As shown above,
CUE_DEBUG=opendefcan bring as much as a 6x improvement. I've raised #3881 for that.