Inconsistencies when unifying with templates

[mikelorant]

What version of CUE are you using (cue version)?

$ cue version
cue version v0.11.1

Does this issue reproduce with the latest stable release?

Have reproduced this issue on both v0.11.1 and v0.12.0-alpha.1.

What did you do?

I am attempting to unify two invisible structs where one of the structs is using a template.

The following example can be used to see the issue:

cue export bug.cue

bug.cue

_foo: {
  a: i: 0
  b: i: 1
}

_bar: {
  c: i: 2
  d: i: 3
}

_bar: {
	[Key=_]: {
		j: 1
	}
}

baz: _foo
baz: _bar

What did you expect to see?

Only c and d keys should have the j key added to their values.

Expected output is:

{
    "baz": {
        "a": {
            "i": 0
        },
        "b": {
            "i": 1
        },
        "c": {
            "i": 2,
            "j": 1
        },
        "d": {
            "i": 3,
            "j": 1
        }
    }
}

What did you see instead?

All keys under baz had the j key added to their values.

Actual output is:

{
    "baz": {
        "a": {
            "i": 0,
            "j": 1
        },
        "b": {
            "i": 1,
            "j": 1
        },
        "c": {
            "i": 2,
            "j": 1
        },
        "d": {
            "i": 3,
            "j": 1
        }
    }
}

[mikelorant]

Simplifying to remove the template does show the expected result:

_foo: {
  a: i: 0
  b: i: 1
}

_bar: {
  c: {
    i: 2
    j: 1
  }
  d: {
    i: 3
    j: 1
  }
}

baz: _foo
baz: _bar

Generating _bar from a template and then merging it with another struct seems to create the field leak.

[myitcv]

Thanks for raising, @mikelorant. I've transferred this to a discussion for now, until we clearly identify a bug/feature request that isn't covered elsewhere.

The behaviour you are seeing is as expected given how templates are defined. The "issue" is that the template for j: 1 is part of the result of the unification:

baz: _foo
baz: _bar

i.e. the template ends up also applying to the fields of baz that were contributed by _foo.

To directly answer your question with a workaround, here is one approach (play):

_foo: [string]: _source: "_foo" 
_foo: {
  a: i: 0
  b: i: 1
}

_bar: {
  c: i: 2
  d: i: 3
}

_bar: {
	[Key=_]: {
        j: 1        
	}
}

for v in [_foo, _bar] for kk, vv in v {
  baz: (kk): vv
}

This is, however, nowhere near as elegant.

There are a couple of options here:

1. Rethink the way that you are structuring things. I'm always reticent to give such a response, because it might come across as "you are doing things wrong!" That's very much not the case here, as you will see from the next point. But if you could provide some more detail on what you're ultimately trying to achieve here, that might help to either provide an alternative approach, or instead further affirm the next point.
2. Your question has hit upon a point that I have recently discussed with @rogpeppe regarding missing builtins. I've added a brief mention of that in Proposal: core builtin extensions #943 (comment), and will (in case you're interested) be adding more detail in Definitions, closedness, embedding, abstractions etc #3653 on the back of that. TL;DR - something like a finalize() builtin (or more precisely a way to "strip" templates) would be a "more elegant" way to achieve the same result.

[mikelorant]

Thanks for the explanation. My delayed response is that it actually took me a few days to actually understand what you were explaining.

If I am correct, everything gets merged together and the template now becomes part of the output which iterates over the full set.

This has become an order of operations problem.

I have made a mistake by turning this into an XY problem and should have explained what I am trying to achieve.

Stepping back, I am trying to create a concise config file structure for my application to use. I want to split the configuration that the user defines into a separate piece and let Cue expand it into a complete structure that can be mapped to a Go struct.

The structure I want the user to define would look like this:

// Config that will go in the home directory.
home: {
  zsh: _
  nano: _
  ssh: target: "ssh"
  ...
}

// Config that will go in to the XDG config directory.
config: {
  git: _
  ghostty: _
}

// Now merge them together under a link struct.
link: home
link: config

The ouput I want from this (if it turns into something JSON-like):

link: {
  zsh: { }
  nano: { }
  ssh: { target: "ssh" }
  git: { base: "config" }
  ghostty: { base: "config" }
}

Basically, I want to merge both structs but have something added to all entries in the "config" struct. I want to make sure they are all validated as well.

The template was the critical part that could fill in required fields and validate what is provided.

Was my approach flawed in thinking that I could add something to a struct and then use a template that could transform it into the result that could match my Go struct?

I wanted to also hide some of the complexity from the user (so I did not want to have the Cue validation in my user editable file). The templating allows for a common block to have less repetition (in this case the desire not to have to constantly repeat base: "config" for each item.

Hopefully I have clearly explained what I am trying to achieve and why I took the approach that I did. What would you recommend?

Thanks again for your advice.

[DavidGamba]

The beauty of CUE is that the config can live in multiple files without issues. I would split your schema file from your config file. The schema file can be embedded into your Go application and the user never has to even know about link.

Schema file:

#home: [string]: {
	target?: string
}

home: #home

#config: [string]: {
	base: "config"
}

config: #config

#link: [string]: {
	target?: string
	base?:   string
}

link: #link
for v in [home, config] for kk, vv in v {
	link: (kk): vv
}

Config file:

// Config that will go in the home directory.
home: {
	zsh:  _
	nano: _
	ssh: target: "ssh"
}

// Config taht will go in the XDG config directory
config: {
	git:     _
	ghostty: _
}

Playground

This is an example of how I use Golang's embed to embed the CUE schema in my apps:
schema.cue
config.go
And a sample user config that exposes nothing about using the schemas:
user-config.cue

[myitcv]

Apologies for the delay in replying here. Busy week! I will reply asap.

[myitcv]

After a very extended delay, I'm coming back to this thread. As we discussed on the recent community call, we've been extremely busy with evalv3 and curated modules.

Thanks to @DavidGamba for his response. I don't have any further to add, based on the way that CUE currently works. Using comprehensions in that way will achieve what you want. And for some comfort, what you are doing sounds entirely reasonable to me!

That said @mikelorant I've just raised #3804 to more explicitly capture the language-level concern of how to do this in a more principled way. Because it's not the first time that it's come up.

Just one comment on @DavidGamba's suggestion:

The schema file can be embedded into your Go application

I would agree, with one mild additional suggestion: consider the schema as the source of truth, and the application's actual implementation something that follows that source of truth. The main reason being, this will allow your users to validate their configurations before calling the application. Shifting left with validating configurations like this is "a good thing". You might like to think of it as schema-first development. It pulls forward conversations about the interfaces, APIs etc that exist between different parts of a system. And configuration is a key part of that. For example, an older version of your application consuming a configuration that was updated with fields that are only understood in a newer version of the application (think rollback of a rollout).