Batch edit for multiple trees

[sharadsw]

Fixes #6127
Fixes #6011
Fixes #6315

Warning

This PR does not directly contain a migration but since it's based on #5417, make sure to use a dataset where #5417 was tested.

This PR enables batch editing on Tree tables and enables support for querying on all tree fields when choosing a specific rank (no longer limited to Author and Full Name). Functionally, this PR will work the same as the workbench when uploading to a tree rank.

Batch editing with tree as base table

• The name field is required when querying on a tree rank
• When choosing a specific rank (and not any rank), Batch Edit expects to have all ranks lower than the chosen rank in the query
  • If the user does not add the "missing" fields, they will be prompted to pick the trees they wish to batch edit
    
    • Users can pick which trees they want in case of conflicting ranks
    • Batch edit will only add ranks from selected trees
    • If no trees are selected, batch edit will use ALL trees
  • When selected ranks do not exist in multiple trees but there are missing ranks:
    
  • Example: Say you have the following ranks for trees Tree 1 and Tree 2:
    • Tree 1 -> Species and Tree 1 -> Subspecies
    • Tree 2 -> Species (lowest rank)
  • And you make a query with:
    • Tree 1 -> Species -> name
  • Batch edit will implicitly add missing ranks (based on which trees were selected) and augment them to the dataset. The dataset will then look like
    • Tree 1 -> Species -> name
    • Tree 1 -> Subspecies -> name
    • Tree 2 -> Species -> name

Batch editing a Tree path

• When editing a tree 'path' using a Tree as a base table with Batch Edit, the records are not edited directly. Rather the record is cloned and uploaded to the tree, which means functionally it will work the same as the workbench. The implementation in this PR will be more apparent in Batch edit for relationships #6283 when trying to edit tree records through a relationship. eg: Determinations (more info here: Batch edit for multiple trees #6196 (comment))
  

Editing a single node

• A single node's simple fields can be edited using a query containing a (any rank) query field
  

Checklist

• Self-review the PR after opening it to make sure the changes look good and
  self-explanatory (or properly documented)
• Add automated tests
• Add relevant issue to release milestone
• Add relevant documentation (Tester - Dev)
• Add a reverse migration if a migration is present in the PR

Testing instructions

Warning

NOTE: This PR will have the same capabilities as in the workbench when uploading to a Tree table if you edit the name field (i.e: Edited records will not be considered 'updated' cells but rather be a new upload to the tree).

However, other simple fields of the Tree can still be edited and will highlight as updated cell.

Test #6011

• Create a query with a Tree base table (eg: Taxon, Storage, Geography, Lithostrat, Chronostrat, Tectonic Unit)
• Verify all tree fields can be queried with

Test batch edit with a single tree in db

1. Edit a single node

• Make a tree query with (any rank) and some other simple fields
• Click Batch Edit
• Edit fields
• Test validate/commit
• Verify edited cells were highlighted as Updated Cell
• Test rollback

2. Edit a tree path

• Make a tree query with multiple rank names
• Batch edit
• Edit fields
• Test validate/commit/rollback
• Verify changes are highlighted as New Cell and the tree path is a new upload to the Tree

3. Test missing ranks

• Make a tree query with a rank that isn't the lowest rank
• Click batch edit
• Verify there is a missing rank dialog indicating the lower ranks will be added
• Click continue
• Verify missing ranks were added correctly

Test batch edit with multiple trees in the db

• Repeat steps for editing single node
• Repeat steps for editing a tree path
• Repeat steps for missing ranks
  • Verify missing ranks dialog allows you to select trees to edit with
  • Verify ranks of selected trees are added correctly
  • Verify ALL trees are used when no trees are selected in the dialog

[sharadsw]

The query builder currently only shows rank names to filter on even when you have multiple trees. If you have multiple trees with the same rank name and rankId, that rank will only have one entry in the QB and it will refer to both trees. I'm not sure what we can do to avoid duplicate names or the long list of names in the error here unless we make the QB mapper have the same behavior as the Workbench mapper where you can specify which tree you want to query on. (see: #6127 (comment))

[realVinayak]

RE #6196 (comment).
Yeah this is the correct idea! But, shouldn't be needing the params at all. I'm not sure why it was included in the initial query builder by Ben.

[realVinayak]

RE: #6196 (comment).

Yeah, that's what I was stuck on before and was a hard problem. Actually, there are atleast two ways of approaching it:

First way:

Allow query builder to support specific trees (related: #5251). This would involve a bit of frontend change, although, I've no memory / idea of how hard that will be, best person might actually be Jason. On the backend (assuming treedef is available in queryfieldspec)

a. We still need to make the worst case assumption that the user selects a rank from any tree (so we still need to take the maximum of treedefitems). This part is already done. See

specify7/specifyweb/stored_queries/query_construct.py

Lines 48 to 52 in 7e478b9

	treedefs = get_treedefs(query.collection, table.name)
	# We need to take the max here. Otherwise, it is possible that the same rank
	# name may not occur at the same level across tree defs.
	max_depth = max(depth for _, depth in treedefs)

b. Need to restrict this to just one tree.

specify7/specifyweb/stored_queries/query_construct.py

Lines 68 to 71 in 7e478b9

	treedefs_with_ranks: List[Tuple[int, int]] = [tup for tup in [
	(treedef_id, _safe_filter(item_model.objects.filter(treedef_id=treedef_id, name=tree_rank).values_list('id', flat=True)))
	for treedef_id, _ in treedefs
	] if tup[1] is not None]

Simplest way will be to add a treeid member field in TreeRankQuery when parsing. Somewhere here:

specify7/specifyweb/stored_queries/queryfieldspec.py

Lines 189 to 194 in 7e478b9

	tree_rank = TreeRankQuery(
	name=tree_rank_name,
	relatedModelName=node.name,
	type="many-to-one",
	column=node.idField.column
	)

c. Now, when say a new queryfieldspec with a treeid is encountered, the case-when block will select the correct case where the treedefitem_id is encountered. In the current implementation if say ranks from treedefs 5 and 6 are searched, then both are searched. This is because of the following two places:

specify7/specifyweb/stored_queries/query_construct.py

Lines 92 to 94 in befd6a6

	# We don't want to include treedef if the rank is not present.
	new_filters = [*query.internal_filters, getattr(node, treedef_column).in_(defs_to_filter_on)]
	query = query._replace(internal_filters=new_filters)

specify7/specifyweb/stored_queries/query_construct.py

Lines 139 to 142 in befd6a6

	# To make things "simpler", it doesn't apply any filters, but returns a single predicate
	# @model is an input parameter, because cannot guess if it is aliased or not (callers are supposed to know that)
	def get_internal_filters(self):
	return sql.or_(*self.internal_filters)

(The or clause makes the magic. Although, the comment in second is outdated --- I removed model param because it was buggy).

d. By the time you get here, the tree query will work for a specific tree. Next is to make it work with batch-edit. All we need to do is make a format like this (taken from #5091 (comment)).

"mustMatchTreeRecord": { 
   "ranks": { 
     "SomeTreeName~>Class": { 
       "treeNodeCols": { 
         "name": { 
           "matchBehavior": "ignoreAlways", 
           "nullAllowed": true, 
           "default": null, 
           "column": "Class" 
         } 
       }, 
       "treeId": 1 
     }, 
 } 

Note: I'm pretty pretty sure treeId was not necessary anymore (double check this, it makes things more simpler. We can still make things work though.).

e. Add a function like get_wb_name to TreeRankQuery. Assuming that there is a treeId member field, it'll be something like (in pseudocode). Yes, getting self.treeId (so the ID) instead of the treeName -- not a mistake! -- it makes things simpler later

def get_wb_name(self): return self.treeId + "~>"  + self.name # Please use the constant for "~>" though.

f. Replace

specify7/specifyweb/stored_queries/batch_edit.py

Line 399 in 7e478b9

node.name.lower() if not isinstance(node, TreeRankQuery) else node.name

by node.name.lower() if not isinstance(node, TreeRankQuery) else node.get_wb_name().

And that should be all. This is because the relationship name (and hence the rank) will contain treedefid. If the "treeId" prop is required (like in here:

specify7/specifyweb/frontend/js_src/lib/tests/fixtures/uploadplan.1.json

Line 319 in eb3c820

"treeId": 1

). Trace the code out in TreeRecord.py (as introduced in #5091) and see if it actually is needed. I can spend time for a more in-depth review laer. That file (after 5091) is waaay too bloated (and I intended on refactoring as part of the merge of production and 4929). If it looks like it is actually needed, construct TreeRankRecord itself. To do that, extract the id from the key here (which was added by get_wb_name, and then later get the name of the treedef). The key, in question, is here:

specify7/specifyweb/stored_queries/batch_edit.py

Line 934 in 7e478b9

for (key, upload_table) in to_one_upload_tables.items()

. Yeah, quite hacky, but things are unnecessarily complicated from bloating in TreeRecord.py.

g. You'd not need to make adjustments to parsing of batch-edit pack.

h. Things should just work out of the box by this time now.

Second way:

a. Don't make any changes to the frontend.
b. Before getting row plan map in batch-edit query below, augment the query such that if a rank appears, then you add all trees that contains that rank (in that discipline). So, essentially, you are making the choice for the user here (that's why no frontend changes).

specify7/specifyweb/stored_queries/batch_edit.py

Line 1034 in 7e478b9

row_plan = RowPlanMap.get_row_plan(visible_fields)

That is, suppose you have two trees Fossil and Rocks and both contain rank species. And user's query was (Determination -> Taxon -> Species -> Name), then rewrite this to contain two query fields:

(Determination -> Taxon -> Species (Fossil) -> Name)
(Determination -> Taxon -> Species (Rocks) -> Name)

So, each such field will get diverged into however many trees contain it.
If you have some kind of filter, things become more complicated (in the worst case, you'd need to generate four fields instead of two -- the extra two will be for OR null).
This approach is quite complicated to implement correctly. However, the all the backend changes from method 1 will still work here (so those are common). Maybe ask UX to see what is acceptable?

I can look over the changes in TreeRecord.py more deeply later (on Tuesday/Thursday morning/evening). Querying a specific tree support on the backend is needed regardless. Definitely see if you have a better idea. On the frontend btw, it'll be probably be beneficial to add the treedef name in the list in BatchEdit.tsx that asks you to select extra fields. Also, if you feel things can be simplified in batch-edit in general (even for simple fields PR), definitely do that (and lmk! only fair since I called other code bloated).

[beach53]

Vinnie You are keeping all this stuff in your head? I'm worried about you. James H. Beach Specify Collections Consortium Biodiversity Institute University of Kansas 1345 Jayhawk Boulevard Lawrence, KS 66045, USA ***@***.*** www.specifysoftware.org Office: +1 785.864.4645 Cell: +1 785.331.8508

…

On Sun, Feb 9, 2025 at 12:51 AM Vinayak Jha ***@***.***> wrote: RE: #6196 (comment) <#6196 (comment)>. Yeah, that's what I was stuck on before and was a hard problem. Actually, there are atleast two ways of approaching it: First way: Allow query builder to support specific trees (related: #5251 <#5251>). This would involve a bit of frontend change, although, I've no memory / idea of how hard that will be, best person might actually be Jason. On the backend (assuming treedef is available in queryfieldspec) a. We still need to make the worst case assumption that the user selects a rank from any tree (so we still need to take the maximum of treedefitems). This part is already done. See https://github.com/specify/specify7/blob/7e478b967a4b2be2e5cfbc868535a574f8535c88/specifyweb/stored_queries/query_construct.py#L48-L52 b. Need to restrict this to just one tree. https://github.com/specify/specify7/blob/7e478b967a4b2be2e5cfbc868535a574f8535c88/specifyweb/stored_queries/query_construct.py#L68-L71 Simplest way will be to add a treeid member field in TreeRankQuery when parsing. Somewhere here: https://github.com/specify/specify7/blob/7e478b967a4b2be2e5cfbc868535a574f8535c88/specifyweb/stored_queries/queryfieldspec.py#L189-L194 c. Now, when say a new queryfieldspec with a treeid is encountered, the case-when block will select the correct case where the treedefitem_id is encountered. In the current implementation if say ranks from treedefs 5 and 6 are searched, then *both* are searched. This is because of the following two places: https://github.com/specify/specify7/blob/befd6a625df3ef0999d5df8bc14b4aea4302acd8/specifyweb/stored_queries/query_construct.py#L92-L94 https://github.com/specify/specify7/blob/befd6a625df3ef0999d5df8bc14b4aea4302acd8/specifyweb/stored_queries/query_construct.py#L139-L142 (The *or* clause makes the magic. Although, the comment in second is outdated --- I removed model param because it was buggy). d. By the time you get here, the tree query will work for a specific tree. Next is to make it work with batch-edit. *All* we need to do is make a format like this (taken from #5091 (comment) <#5091 (comment)>). "mustMatchTreeRecord": { "ranks": { "SomeTreeName~>Class": { "treeNodeCols": { "name": { "matchBehavior": "ignoreAlways", "nullAllowed": true, "default": null, "column": "Class" } }, "treeId": 1 }, } Note: I'm pretty pretty sure treeId was not necessary anymore (double check this, it makes things more simpler. We can still make things work though.). e. Add a function like get_wb_name to TreeRankQuery. Assuming that there is a treeId member field, it'll be something like (in pseudocode). *Yes, getting self.treeId (so the ID) instead of the treeName -- not a mistake! -- it makes things simpler later* def get_wb_name(self): return self.treeId + "~>" + self.name # Please use the constant for "~>" though. f. Replace https://github.com/specify/specify7/blob/7e478b967a4b2be2e5cfbc868535a574f8535c88/specifyweb/stored_queries/batch_edit.py#L399 by node.name.lower() if not isinstance(node, TreeRankQuery) else node.get_wb_name(). And that should be all. This is because the relationship name (and hence the rank) will contain treedefid. If the "treeId" prop is required (like in here: https://github.com/specify/specify7/blob/eb3c8200bad92af9f43c063655ba47f8a02badf2/specifyweb/frontend/js_src/lib/tests/fixtures/uploadplan.1.json#L319). Trace the code out in TreeRecord.py (as introduced in #5091 <#5091>) and see if it actually is needed. I can spend time for a more in-depth review laer. That file (after 5091) is waaay too bloated (and I intended on refactoring as part of the merge of production and 4929). If it looks like it is actually needed, construct TreeRankRecord itself. To do that, extract the id from the key here (which was added by get_wb_name, and then later get the name of the treedef). The key, in question, is here: https://github.com/specify/specify7/blob/7e478b967a4b2be2e5cfbc868535a574f8535c88/specifyweb/stored_queries/batch_edit.py#L934. Yeah, quite hacky, but things are unnecessarily complicated from bloating in TreeRecord.py. g. You'd not need to make adjustments to parsing of batch-edit pack. h. Things should just work out of the box by this time now. Second way: a. Don't make any changes to the frontend. b. Before getting row plan map in batch-edit query below, augment the query such that if a rank appears, then you add all trees that contains that rank (in that discipline). So, essentially, you are making the choice for the user here (that's why no frontend changes). https://github.com/specify/specify7/blob/7e478b967a4b2be2e5cfbc868535a574f8535c88/specifyweb/stored_queries/batch_edit.py#L1034 That is, suppose you have two trees Fossil and Rocks and both contain rank species. And user's query was (Determination -> Taxon -> Species -> Name), then rewrite this to contain two query fields: (Determination -> Taxon -> Species (Fossil) -> Name) (Determination -> Taxon -> Species (Rocks) -> Name) So, each such field will get diverged into however many trees contain it. If you have some kind of filter, things become more complicated (in the worst case, you'd need to generate four fields instead of two -- the extra two will be for OR null). This approach is quite complicated to implement correctly. However, the all the backend changes from method 1 will still work here (so those are common). Maybe ask UX to see what is acceptable? I can look over the changes in TreeRecord.py more deeply later (on Tuesday/Thursday morning/evening). Querying a specific tree support on the backend is needed regardless. Definitely see if you have a better idea. On the frontend btw, it'll be probably be beneficial to add the treedef name in the list in BatchEdit.tsx that asks you to select extra fields. Also, if you feel things can be simplified in batch-edit in general (even for simple fields PR), definitely do that. — Reply to this email directly, view it on GitHub <#6196 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACTDB4RLNEIZZB2DRCWN5ZT2O33FPAVCNFSM6AAAAABWRIGB6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBWGA4TOMZSHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[realVinayak]

Eh, just wanted to give Sharad more context about this - I recall how hard it was to fix bugs in xml-editor, and that's after we had videos from Max! If it makes you feel any better, I don't remember much beyond changes I did in batch-edit and nested-to-manys.

[sharadsw]

NOTE: Querying on Taxon names that involve different trees don't work correctly in this PR.
In prod:

Deltodus and Test Genus belong to different trees and the results work correctly. In this branch, a query like this returns no result at all.

[realVinayak]

huh that is weird. By the way, is the Deltodus duplicated? What does the query look like?

[sharadsw]

Weird, turns out one of the Deltodus actually belongs to Species instead: https://calvertmarinemuseum20250206-production.test.specifysystems.org/specify/view/taxon/38726/

[realVinayak]

Ah that's fine (repeat of that taxon). Curious what the sql query itself looks like on this branch for that query not working. It should be log level debug

[sharadsw]

@realVinayak Prod: https://pastebin.com/imefgzi0
Batch edit: https://pastebin.com/ADt4HrGx

Here's a diffcheck in case pastebin doesn't like me: https://www.diffchecker.com/PxUxsDFd/

Prod looks at the treedefid as well in the case whens. Looks like I missed something somewhere

[realVinayak]

that's perfect thanks. I can look at that in an hour or so. I did see the treedefid discrepancy in the code, but that would not matter (so yes, the code from production is redundant).

[realVinayak]

This is a wacky bug, and it also happens in issue-5413 (so batch edit too) but I think you already knew that. The diffs helped. Specifically, I noticed this:

That is, bug had to be here:

specify7/specifyweb/stored_queries/execution.py

Lines 882 to 889 in 4814f11

	if props.implicit_or:
	implicit_ors = [
	reduce(sql.or_, ps) for ps in predicates_by_field.values() if ps
	]
	if implicit_ors:
	where = reduce(sql.and_, implicit_ors)
	query = query.filter(where)

The reason is because TreeRankQuery instances are dynamically constructed. The proper way to fix this would be use NamedTuples for fields and relationships. See #628.

I guess a decent way could be to

class TreeRankQuery(Relationship):
    # FUTURE: used to remember what the previous value was. Useless after 6 retires
    original_field: str

    def __hash__(self):
        return hash((TreeRankQuery, self.relatedModelName, self.name))

    def __eq__(self, value):
        return (
            isinstance(value, TreeRankQuery)
            and value.name == self.name
            and value.relatedModelName == self.relatedModelName
        )

EDIT: I jumped the gun on the explanation. The bug originates here:

specify7/specifyweb/stored_queries/execution.py

Line 880 in 38a632d

predicates_by_field[fs.fieldspec].append(predicate)

The way ORs work is by constructing the same queryfieldspec but implicitly ORing them. Above, the fs.fieldspecs are all different in the cases where TreeRankQuery occurs in the join_path (because the class instances are dynamic). Not a problem for regular fields and relationships because they are created just once at the beginning and are then "static".

[sharadsw]

NOTE:
After internal discussion, the conclusion that was reached for batch editing with multiple trees was:

• No change to frontend queries as we need the ability to search on ranks across all trees
• The error dialog (Batch edit for multiple trees #6196 (comment)) will instead be a prompt that notifies the users what fields are missing
• The missing fields will then be added on the backend when creating the batch edit dataset

[sharadsw]

@realVinayak Since the frontend queries are not to be changed and we want to add missing fields to the dataset directly, I think the second way you mentioned sounds appropriate. Can you clarify what you mean by adding two extra fields for OR null?

Also, I tried hardcoding some things around to get a feel for what changes need to be made. When I make a change to a Taxon and commit, it gets highlighted as a new cell and a new node gets uploaded to the tree. The behavior is the same in #4929 before the prod merge (commit 2d86ba4). Is this expected?

[realVinayak]

Sorry for the delay.

Question 1
Yeah, the second one seems like the way to go. Actually, I tried implementing it myself, and realized I misspoke when I said adding two extra fields for the null. My initial thought process was to make it so the user makes a query like (Species->name == "whatever"), then this needs to be broken into (assuming species occurs in treedefs 1 and 2)

(Species->name == "whatever" and treedef_id == 1) OR (Species->name == "whatever" and treedef_id == 2)

Note here that the OR is needed. Otherwise, you'd be constructing

(Species->name == "whatever" and treedef_id == 1) AND (Species->name == "whatever" and treedef_id == 2)

which can never be true. So, I thought this could be fixed by making the below (which is what the message above meant)

([Species->name == "whatever" OR Species->name is null] and treedef_id == 1)  AND ([Species->name == "whatever" OR Species->name is null] and treedef_id == 2)

But the last one is actually wrong because then the cases where both are null will also be added.

In conclusion, the first variant is correct. Since only one of the OR conditions can be true at a time anyways, it's good.

Question 2
Yes, that's "expected". Actually, we can't sanely do batch-edit on tree ranks (notice the specificity), it's all a lie. There's just isn't a clear definition for it, so I just made it exactly like workbench, but that record data is cloned. That is, say you have something like

And you bring it in batch-edit, like below and change the name (the yellow), it'll try cloning all the attributes of the previous "TestSpeciesOther", and try inserting like workbench. So, you can never actually do batch-edit on tree ranks.

This might seem like there's then no need of supporting tree ranks, but, bulk changing determinations can be done this way (the changed node will be matched and created if not found). Does this make sense? I'd recommend asking grant if he has sometime to discuss this, I did demo this quite a bit before.

BTW, here is the corresponding query (for the above screenshots)

You can access the query at http://localhost:5050/specify/query/261/. Lmk if that link doesn't work /j.

Misc

Why do we care about the "highest" rank?
Just to be on the same page about trees.

Highest means from the bottom. Lower rankid, more higher in the tree.

There is no harm in always showing all the ranks to the user. That is, every time a relationship to tree (with ranks) appears, we can show all the ranks for all the trees. This simplifies implementation quite a bit (all that findMissingRank stuff won't be needed). However, this is bad for usability because, you know, some have like 27 ranks + multiple trees can't make it better. So, we show the least ranks in least trees needed to get stuff done. For a single tree, it'll be all the ranks below the highest.

But, wait -- do we always need that? If the user selects "Species" and "Genus", are there cases where we can show just those two ranks? Yes, in fact, if taxon is the base table, you can do that. But, if you are something like CO->det->taxon, we need to show all the ranks below the highest (genus).

(however, what about below the lowest rank, so instead of everything below genus, what about everything below species? why would that not work? -- think about this! there are cases where this won't work)

In the case of multiple tree, we can do the exact same thing, but for all the trees the selected ranks can be part of. In fact, we can only show just those trees that the user selected via ranks (rather than showing all trees).

Need to be careful here, there can be more than one path to the exact same tree. Think about "CO->CollectingEvent->CEA->taxon" and "CO->det->taxon".

misc again

Batch-edit is two steps as you know:

1. Query to dataset
2. Batch-edit within workbench framework

I actually got the first part done for multiple trees yesterday (sunday) - I was curious about it. It does follow the requirements that were mentioned. I don't expect you to use it (that's why I have not made a PR on sp7 repo). However, if you're curious about it, here is the PR equivalent on my fork of sp7 repo (comparing my changes to this branch): https://github.com/realVinayak/specify7/pull/1. The main function you'd be tracing is this. The above screenshots are actually from that branch. You might recognize the different tree names. Here are some brief feature notes:

Good things

1. Fields are added on the backend. Currently, they are not localized. However, the treedef name are added to the column name (to better disambiguate) -- you can see that in the above workbench screenshot (that's from the above PR), where "botany" and "paleo taxonomy tree" is added.
2. Only the most required fields, for the least ranks, in the least trees are shown. That is, if you selected Genus, and there happens to be ["Species"] and ["Species", "Subspecies"] below in two different trees, it'll just show those three. For example, in the above screenshot, the other ranks from the different trees were automatically added.
3. If you select "Species", and there is another tree with ranks ["rank1", "rank2"], that later tree won't be included (because ranks aren't there)
4. Filters are automatically, and correctly handled. The OR trick did it.
5. Currently, it acts on the RowPlanMap, so if there are more than 1 path to the same tree, they are still correctly handled.
6. It's generic for all trees

Bad things:

1. Doesn't show missing fields on the frontend. Currently, if just directly constructs the dataset from the query (without a popup). However, this can still be done -- it's more simpler than backend logic.
2. Doesn't do any localizing -- needs hacks.
3. This is just the first part of batch-edit. haven't spent more than 5 minutes on the workbench/upload/tree_record.py file when coding this, I don't know if the unit tests run correctly (probably need changes) and if the workbench part (component 2) of batch-edit works or not (probably doesn't). However (this might help you in your implementation) -- batch edit for treerecord is significantly more simpler than for uploadtables (because we can't have in-place edits, so all we're doing is just clone). I'd recommend going back to 4929 before the prod merge and checking it out for yourself how less complicated it is, compared to uploadtables (non-tree relationship) edits.
4. It doesn't necessarily order the columns correctly -- that's complicated.
5. It's quite complicated in general, will require unit tests
6. Might make future changes (if sp7 supports multiple treedef query) harder, although, it's quite considerate towards it.

Reflections
I did think about different variations -- doing insertions of fields on the frontend, vs on the backend. Since all the query construction happens on the backend, it does make things a bit easier -- at the cost of localizing, but that can be fixed with hacks... BUT, any of this is not supposed to mean there isn't a better and simpler implementation than what I did in that PR.

[sharadsw]

@realVinayak Thank you! Your approach is a lot cleaner than what I was trying to do. I'm gonna use your code if that's okay.

I'm not sure TreeRankRecord is needed for the upload plan. It seems upload works the same as long as the key follows the format <treename>~><rankname> but might as well keep TreeRankRecord just to keep it consistent between WB and Batch Edit. Upload will use the same code regardless

[realVinayak]

I'm gonna use your code if that's okay.

Yeah that's fine - it's open source and public. Still, if you feel you can simplify logic more, please do so!

It seems upload works the same as long as the key follows the format......

Yeah, I ended up making a conservative assumption that I need to use TreeRankRecord. I think the apply_batch_edit_pack for TreeRecord would need some adjustement (since, internally, the batch_edit.py uses the key of format <treename>~><rankname>~><treedef_id> but that won't be a hard change (just would require change to apply_batch_edit_pack).

EDIT: I just looked at the treerecord.py file. The changes necessary (from what it seems like) should be here (and not in apply_batch_edit_pack, spoke too fast).

specify7/specifyweb/workbench/upload/treerecord.py

Lines 1077 to 1081 in aa81431

	if tdi.name not in self.batch_edit_pack:
	continue
	columns = [pr.column for pr in self.parsedFields[tdi.name]]
	info = ReportInfo(tableName=self.name, columns=columns, treeInfo=None)
	pack = self.batch_edit_pack[tdi.name]

Specifically, just format the key to be something like <treename>~><rankname>~><treedef_id> format. Seems like the only change necessary here (to get it working, not necessarily bug free) would be to make those lines to be

ref_key = f"{tdi.treedef.name}~>{tdi.name}~>{tdi.treedef_id}"
tree_rank_record = TreeRankRecord(tdi.name, tdi.treedef_id)
if ref_key not in self.batch_edit_pack:
    continue
columns = [pr.column for pr in self.parsedFields[tree_rank_record]]
info = ReportInfo(tableName=self.name, columns=columns, treeInfo=None)
pack = self.batch_edit_pack[ref_key]

I haven't tested above but that's the idea. Still need to trace out later to see if there are any gotchas from the prod code changes.

[lexiclevenger]

Test #6011

• Create a query with a Tree base table (eg: Taxon, Storage, Geography, Lithostrat, Chronostrat, Tectonic Unit)
• Verify all tree fields can be queried with

Test batch edit with a single tree in db

1. Edit a single node

• Make a tree query with (any rank) and some other simple fields
• Click Batch Edit
• Edit fields
• Test validate/commit
• Verify edited cells were highlighted as Updated Cell
• Test rollback

2. Edit a tree path

• Make a tree query with multiple rank names
• Batch edit
• Edit fields
• Test validate/commit/rollback
• Verify changes are highlighted as New Cell and the tree path is a new upload to the Tree

3. Test missing ranks

• Make a tree query with a rank that isn't the lowest rank
• Click batch edit
• Verify there is a missing rank dialog indicating the lower ranks will be added
• Click continue
• Verify missing ranks were added correctly

Test batch edit with multiple trees in the db

• Repeat steps for editing single node
• Repeat steps for editing a tree path
• Repeat steps for missing ranks
  • Verify missing ranks dialog allows you to select trees to edit with
  • Verify ranks of selected trees are added correctly
  • Verify ALL trees are used when no trees are selected in the dialog

  ---

  Looks good, all trees can be batch edited. However, I couldn't roll back any of the data sets. Maybe due to #6427? Here is the error mesage:

[lexiclevenger]

Here is a data set that won't roll back: https://ciscollectionsbatchedit220250210-issue-6127.test.specifysystems.org/specify/workbench/28

[sharadsw]

@lexiclevenger Can you try again? Should be fixed now

[sharadsw]

Created a different issue for cases where rollback fails due to a disambiguation error: #6443

Datasets without a disambiguation error should have a working rollback

[emenslin]

Test #6011

• Verify all tree fields can be queried with

Test batch edit with a single tree in db

1. Edit a single node

• Test validate/commit
• Verify edited cells were highlighted as Updated Cell
• Test rollback

2. Edit a tree path

• Test validate/commit/rollback
• Verify changes are highlighted as New Cell and the tree path is a new upload to the Tree

3. Test missing ranks

• Verify there is a missing rank dialog indicating the lower ranks will be added
• Verify missing ranks were added correctly

Test batch edit with multiple trees in the db

• Repeat steps for editing single node
• Repeat steps for editing a tree path
• Repeat steps for missing ranks
  • Verify missing ranks dialog allows you to select trees to edit with
  • Verify ranks of selected trees are added correctly
  • Verify ALL trees are used when no trees are selected in the dialog

---

Missing ranks are added in a random order, is this supposed to happen? This is an example with a single taxon tree but it also happens with multiple trees.

04-25_13.55.mp4

Also I did run into this api rank problem causing unload protect a couple times, it seems to only happen when batch editing on 'any rank' but I'm not sure if it happens anywhere else.

04-25_14.35.mp4

[sharadsw]

@emenslin Looks like ranks were being sorted alphabetically. I have changed it so that the user's order of ranks in the query is kept as is.

Created a separate issue for the unload protect error: #6448

[sharadsw]

@emenslin Actually, I had to revert the change. Trying to fix column ordering breaks tree datasets in relationships. I think we'll have to push it to a different milestone. Can you create a issue for it?

[lexiclevenger]

Test batch edit with a single tree in db

1. Edit a single node

• Test validate/commit
• Verify edited cells were highlighted as Updated Cell
• Test rollback

2. Edit a tree path

• Test validate/commit/rollback
• Verify changes are highlighted as New Cell and the tree path is a new upload to the Tree

3. Test missing ranks

• Verify there is a missing rank dialog indicating the lower ranks will be added
• Verify missing ranks were added correctly

Test batch edit with multiple trees in the db

• Repeat steps for editing single node
• Repeat steps for editing a tree path
• Repeat steps for missing ranks
  • Verify missing ranks dialog allows you to select trees to edit with
  • Verify ranks of selected trees are added correctly
  • Verify ALL trees are used when no trees are selected in the dialog

---

Looks good! I didn't run into any issues and was able to roll back all of the data sets as long as they didn't have disambiguation errors.

moved to separate issues: #6448 and #6452