Batch Edit: Support for editing basic fields

[sharadsw]

Fixes #5413
Fixes #6209
Uses code from #4929

⚠️ Note: This PR affects database migrations. See migration testing instructions.

This PR pretty much has the same code as #4929 but updated to production. To minimize the scope for testing, batch edit for relationships is disabled in this PR and will be enabled in the follow up PR - batch edit for relationships. There are still some merge conflicts resulting from some newer code in prod which will be resolved in the next PR.

Batch Edit Workflow

• Create a Query for the data you want to edit
• Click on Batch Edit in the query results
• Update any values
• Validate
• Valid edited cells highlight as Updated Cell if its a simple field
• Click on commit to apply the changes

Permissions

Only users with the Batch Edit permissions are allowed to Batch Edit on a query. For other users, the Batch Edit button will not be visible in the query results

Behaviors

• Fields that are a relationship to the Base Table will be readonly in the Batch Edit dataset
• Valid edited cells will be highlighted as Updated Cell
  
• Committing the changes will apply the changes to database
• Rollback will undo all changes

Checklist

• Self-review the PR after opening it to make sure the changes look good
  and self-explanatory (or properly documented)
• Add automated tests
• Add relevant issue to release milestone

Testing instructions

• Create a query
• Click on Batch Edit. This will create a new Batch Edit dataset with columns involving relationships disabled (readonly)
• Change values you wish to edit
• Click Validate
• Verify changed values are highlighted as updated cells
• Click Commit. This will apply your changes
• Re-run the original query and verify the values changed as expected
• Click rollback
• Verify values were rolled back to original

Regression tests

• Retest Support COT catalog number format configuration in WorkBench #5489

[sharadsw]

POTENTIAL IMPROVEMENTS?

• Data Mapper in Batch Edit?
  • Should the Data Mapper be accessible in Batch Edit datasets? Since Batch Edit datasets can only be created from a Query, it seems confusing that users can still access the Data Mapper from a Batch Edit dataset with most of the mapper functionalities being disabled. It seems the mapper is only needed for changing Batch Edit Preferences. Could move the preferences dialog to Editor page instead.

[realVinayak]

Nice observation. My fault for keeping a hidden feature secret: Another, use for data mapper is to control “ignore when blank”-type behavior per field basis. That’s actually supported in that PR ( even for batch edit data mapper, you can modify this per-field setting. Ask grant for more context )

the only things not read only are

1. Batch edit prefs
2. Field behavior (never ignore, …)
3. must match (also, could you leave a note somewhere to make trees must match by default?)

search for usages of “readonlyspec”.

For purposes of this PR, you can theoretically not add data mapper (you also don’t need batch edit prefs, they don’t do anything when there are no relationships)

[sharadsw]

@combs-a I'm not able to recreate it. I run into a validation error if I try to set the preparation quantity greater than the count or the availability:

What I did was:

• Create a CO with preparation count 5
• Create 2 Gift records
  • One with preparation qty 3
  • One with preparation qty 2
• Tried to update preparation qty > 5 in Batch edit

[sharadsw]

@realVinayak Doesn't look like fld_change_info can be used here without refactoring it to do something similar as is_equal from 4814f11. I'm happy to keep is_equal for now and note this in #6170

specify7/specifyweb/specify/api.py

Lines 536 to 544 in 1b2ab81

	def fld_change_info(obj, field, val) -> Optional[FieldChangeInfo]:
	if field.name != 'timestampmodified':
	value = prepare_value(field, val)
	if isinstance(field, FloatField) or isinstance(field, DecimalField):
	value = None if value is None else float(value)
	old_value = getattr(obj, field.name)
	if str(old_value) != str(value): # ugh
	return FieldChangeInfo(field_name=field.name, old_value=old_value, new_value=value)
	return None

• The string comparison is unreliable when comparing objects with a primitive. Not sure how we can handle it while keeping it generic
  • We get a scope change error when comparing discipline/scope values since the comparison becomes Discipline object (3) == 3 => false
• For Decimal values, the old value remains a Decimal and the new value is a float and so it still gets detected as a change for lat/long
  • i.e str(Decimal(33.1493800000)) == str(33.14938) => 33.1493800000 == 33.14938 => false

Made a note of the lat/long text issues in #6126, thanks!

[sharadsw]

@specify/testing The lat/long decimal issue should now be fixed. The lat/long text update + matching issues is added to #6126 and will be tested when we enable relationships.

[realVinayak]

Ah yes, you're right -- it's more involved than I thought, sorry for wasting your time. I assumed that the previous code is bug-free, but it is not (I tested this on sp7demofish, and latitude are always considered to be changed when saving a locality form -- that's another, unrelated to batch edit, bug). I agree, just keep what you have in that case.

Probably also need to document the lat/long text update (plus other fields that get updated, refer to the code for it). Maybe the discourse documentation is enough, or if there should be a warning when you use locality as the base table.

Just for reference (no changes needed):

RE: We get a scope change error when comparing discipline/scope values since the comparison become

Can replace field.name by field.attname and that'll fix this. Attname will convert discipline to discpline_id (which will be 3),

[combs-a]

@sharadsw I've tried to recreate my issue again a few times, and wasn't able to as well--it should be kept in mind, but it might've just been something very specific.

• Verify changed values are highlighted as updated cells
• Re-run the original query and verify the values changed as expected
• Verify values were rolled back to original

Regression tests

• Retest Support COT catalog number format configuration in WorkBench #5489

LatLon has been kind of fixed, but there are also potentially integers; ints are being converted to floats and that marks them as updated cells when they shouldn't be.

Also, I tried out COs that use COTs with the AlphaNumByYear type, and batch edit flagged the correct format as invalid, and incorrect formats as valid. Is fixing this within the scope of the PR? Wanted to check since the same thing works fine in the Workbench.

The incorrect format is AAAAAA, which was the default catno format before I changed it for FunnyTurtles.

[sharadsw]

@combs-a Catalog number validation is broken because relationships are disabled and so the value gets validated with the collection's default formatter. I created a new issue for it (#6248) but it will be fixed when relationships are enabled in #6126

About the lat/long issue, I'm not sure how we can reliably handle all cases there. The underlying cause for that row being updated is the issue Vinny mentioned where lat1text and lat2text get updated internally.
Specifically, here lat1text and lat2text get updated from 12 to 12.0 but latitude1 and longitude1 are not detected as changed since 12.0 == 12.

We could potentially parse the values similar to the frontend so that any integer values pretending to be floats get converted to integers (ie. 12.0 -> 12) in the dataset but that will lead to another issue where a value actually saved as 12.0 also gets converted to 12 and leads to a change detected in batch edit

Another potential solution could be to have comments on the cell in the frontend when lat/long text values update indicating the reason behind the row being shown as updated. Similar to how we have comments on validation error cells. Unfortunately, we don't have a way to do that right now but could be considered in future.

Either way I have created a separate issue for the underlying cause which will have to be looked at in a later milestone #6251

[combs-a]

• Verify changed values are highlighted as updated cells
• Re-run the original query and verify the values changed as expected
• Verify values were rolled back to original

Regression tests

• Retest Support COT catalog number format configuration in WorkBench #5489

---

All tests passed, good work!

[emenslin]

• Verify changed values are highlighted as updated cells
• Re-run the original query and verify the values changed as expected
• Verify values were rolled back to original
• Retest Support COT catalog number format configuration in WorkBench #5489

I didn't run into any new issues but since this is such a big PR I would like if we could get another testing review just to make sure nothing was missed.

[lexiclevenger]

• Verify changed values are highlighted as updated cells
• Re-run the original query and verify the values changed as expected
• Verify values were rolled back to original

Regression tests

• Retest Support COT catalog number format configuration in WorkBench #5489

---

Looks good, nice job!

[specifysoftware]

This pull request has been mentioned on Specify Community Forum. There might be relevant details there:

https://discourse.specifysoftware.org/t/specify-7-10-2-release-announcement/2455/1