Skip to content

Commit be17b84

Browse files
authored
Batch edit for relationships (#6283)
* Enable trees in queries * Use query construct code from #4929 * Update unit test * Enable nested to-many in Workbench * Update test * Remove param_count * Display tree name in query error * Add upload plan changes * Update tests * Lint code with ESLint and Prettier Triggered by 82033cd on branch refs/heads/issue-2331 * Update TreeRankQuery to fix implicit ORs - see: #6196 (comment) * Allow removing last row in Batch Edit * Make a missing rank info dialog which proceeds to dataset creation * Augment tree queries with missing ranks for batch edit * Lint code with ESLint and Prettier Triggered by 65f7d21 on branch refs/heads/issue-6127 * Adjust apply_batch_edit_pack for multiple trees * Add a discipline type in tests * Enable relationships * Lint code with ESLint and Prettier Triggered by 350ee9c on branch refs/heads/issue-6126 * Enable data mapper and batch edit preferences * Fix localizations * Consider remote to ones as to many in upload plan * Add remote to ones method * Un-enforce TreeRankRecord in upload plan * Add loading action to missing ranks dialog * Move table name to same line in missing fields dialog * Handle case when rank name has spaces * Add a close button to missing ranks dialog * Fix frontend missing field calculation * Fix frontend to many tree error * Lint code with ESLint and Prettier Triggered by 9407376 on branch refs/heads/issue-6127 * Restrict to manys only for tree fields * Avoid cloning to-ones when committing - This was caused because we treat remote to-ones as to-many in the upload plan (affects COGs) * Fix to many for tree in relationships * Change revert to rollback in pref localization * Use TreeRankRecord in upload plan * Fix multiple rank in row error * Fix multiple rank in row error * Fix navigator * Fix tests * Group missing ranks by tree * Lint code with ESLint and Prettier Triggered by 7f2149d on branch refs/heads/issue-6127 * Pass filtered treedef ids to the backend - Adds checkboxes to tree names in missing ranks dialog - Splits the main batch edit file into 4 smaller files * Lint code with ESLint and Prettier Triggered by 1c29ec9 on branch refs/heads/issue-6127 * Filter trees used when rewriting batch edit dataset * Fix tests * Use TreeRankRecord in upload plan * Remove unused string * Fix visual order - For multiple trees, columns will be grouped by tree first * Revert "Fix visual order" This reverts commit a8a2ad6. * Fix tests * Lint code with ESLint and Prettier Triggered by f0822bf on branch refs/heads/issue-2331 * Handle (any rank) mapping for Batch Edit upload plans * Lint code with ESLint and Prettier Triggered by 3945527 on branch refs/heads/issue-6126 * Disable spauditlog for BE * Add title when batch edit is disabled * Fix deleted cells for many-to-one dependents * Use variant permissions for creating record sets * Fix undefined name error - Using tables doesn't work when the data hasn't loaded correctly * Disable changing batch edit prefs after upload * Add validation error for scope change * Add localization for other WB errors * Lint code with ESLint and Prettier Triggered by 34a3459 on branch refs/heads/issue-6126 * Remove description for null record - Removed for UX reasons. Users do not need to manually remove null record strings * Ensure at least 1 to-many column gets added to batch edit datasets * Flag to-many in tree only queries * Lint code with ESLint and Prettier Triggered by f36521c on branch refs/heads/issue-6127 * Lint code with ESLint and Prettier Triggered by 0647fa6 on branch refs/heads/issue-6126 * Enable nested to-many in Workbench (#6216) * Enable nested to-many in Workbench * Update test * Add upload plan changes * Update tests * Lint code with ESLint and Prettier Triggered by 82033cd on branch refs/heads/issue-2331 * Fix tests * Lint code with ESLint and Prettier Triggered by f0822bf on branch refs/heads/issue-2331 * Lint code with ESLint and Prettier Triggered by f27581b on branch refs/heads/issue-2331 * Lint code with ESLint and Prettier Triggered by cc1f85b on branch refs/heads/issue-6127 * Disable editing any rank tree relationships * Check for lowercase tree table names when rewriting tree rank row plan * Lint code with ESLint and Prettier Triggered by 633a7da on branch refs/heads/issue-6126 * Handle None rank * Batch edit: Disable editing dataset after rollback (#6428) * Add rolledback to SpDataset * Lint code with ESLint and Prettier Triggered by 7d2a86d on branch refs/heads/issue-6390 * Add text to indicate dataset cannot be edited * Make hot columns readonly based on context * Lint code with ESLint and Prettier Triggered by dcbb593 on branch refs/heads/issue-6390 * Reorder migration * Fix tree column order * Fix tests * Fix tests * Upgrade celery version (#6437) * Add rolledback to SpDataset * Lint code with ESLint and Prettier Triggered by 7d2a86d on branch refs/heads/issue-6390 * Add text to indicate dataset cannot be edited * Make hot columns readonly based on context * Lint code with ESLint and Prettier Triggered by dcbb593 on branch refs/heads/issue-6390 * Reorder migration * Upgrade celery and its dependencies * Revert back to sort columns * Enable matched and changed when readonly * Add missing import * Re-add lost code
1 parent 0346d96 commit be17b84

31 files changed

+470
-101
lines changed

requirements.txt

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@ setuptools>=50.0.0
22
tzdata
33
wheel
44
backports.zoneinfo==0.2.1
5-
kombu==5.2.4
6-
celery[redis]==5.2.7
5+
kombu==5.5.2
6+
celery[redis]==5.5.1
77
Django==4.2.18
88
mysqlclient==2.1.1
99
SQLAlchemy==1.2.11
@@ -12,4 +12,4 @@ pycryptodome==3.21.0
1212
PyJWT==2.3.0
1313
django-auth-ldap==1.2.17
1414
jsonschema==3.2.0
15-
typing-extensions==4.3.0
15+
typing-extensions==4.12.2

specifyweb/frontend/js_src/lib/components/BatchEdit/index.tsx

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -123,13 +123,16 @@ export function BatchEditFromQuery({
123123
})
124124
);
125125

126+
const isDisabled =
127+
queryFieldSpecs.some(containsSystemTables) ||
128+
queryFieldSpecs.some(hasHierarchyBaseTable) ||
129+
containsDisallowedTables(query);
130+
126131
return (
127132
<>
128133
<Button.Small
129-
disabled={
130-
queryFieldSpecs.some(containsSystemTables) ||
131-
queryFieldSpecs.some(hasHierarchyBaseTable)
132-
}
134+
disabled={isDisabled}
135+
title={isDisabled ? batchEditText.batchEditDisabled() : undefined}
133136
onClick={() => {
134137
loading(
135138
treeRanksPromise.then(async () => {
@@ -205,5 +208,13 @@ const hasHierarchyBaseTable = (queryFieldSpec: QueryFieldSpec) =>
205208
queryFieldSpec.baseTable.name.toLowerCase() as 'collection'
206209
);
207210

211+
// Using tables.SpAuditLog here leads to an error in some cases where the tables data hasn't loaded correctly
212+
const DISALLOWED_TABLES = ['spauditlog'];
213+
214+
const containsDisallowedTables = (query: SpecifyResource<SpQuery>) =>
215+
DISALLOWED_TABLES.some(
216+
(tableName) => query.get('contextName').toLowerCase() === tableName
217+
);
218+
208219
// Error filters
209220
const filters = [containsFaultyNestedToMany, containsSystemTables];

specifyweb/frontend/js_src/lib/components/WbActions/index.tsx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ export function WbActions({
5858
useBooleanState();
5959
const [operationCompleted, openOperationCompleted, closeOperationCompleted] =
6060
useBooleanState();
61+
6162
const { mode, refreshInitiatorAborted, startUpload, triggerStatusComponent } =
6263
useWbActions({
6364
datasetId: dataset.id,
@@ -262,6 +263,10 @@ function useWbActions({
262263
const refreshInitiatorAborted = React.useRef<boolean>(false);
263264
const loading = React.useContext(LoadingContext);
264265

266+
/**
267+
* NOTE: Only validate and upload use startUpload
268+
* For rollback, we directly call the API inside the RollbackConfirmation component
269+
*/
265270
const startUpload = (newMode: WbStatus): void => {
266271
workbench.validation.stopLiveValidation();
267272
loading(

specifyweb/frontend/js_src/lib/components/WbPlanView/Mapper.tsx

Lines changed: 96 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ import type { MappingElementProps } from './LineComponents';
4040
import { getMappingLineProps, MappingLineComponent } from './LineComponents';
4141
import { columnOptionsAreDefault } from './linesGetter';
4242
import {
43+
BatchEditPrefsView,
4344
ChangeBaseTable,
4445
EmptyDataSetDialog,
4546
mappingOptionsMenu,
@@ -102,17 +103,36 @@ export type MappingState = State<
102103
readonly autoMapperSuggestions?: RA<AutoMapperSuggestion>;
103104
readonly openSelectElement?: SelectElementPosition;
104105
readonly validationResults: RA<MappingPath>;
106+
readonly batchEditPrefs?: BatchEditPrefs;
105107
}
106108
>;
107109

110+
export type ReadonlySpec = {
111+
readonly mustMatch: boolean;
112+
readonly columnOptions: boolean;
113+
readonly batchEditPrefs: boolean;
114+
};
115+
116+
export type BatchEditPrefs = {
117+
readonly deferForNullCheck: boolean;
118+
readonly deferForMatch: boolean;
119+
};
120+
121+
export const DEFAULT_BATCH_EDIT_PREFS: BatchEditPrefs = {
122+
deferForMatch: true,
123+
deferForNullCheck: false,
124+
} as const;
125+
108126
export const getDefaultMappingState = ({
109127
changesMade,
110128
lines,
111129
mustMatchPreferences,
130+
batchEditPrefs,
112131
}: {
113132
readonly changesMade: boolean;
114133
readonly lines: RA<MappingLine>;
115134
readonly mustMatchPreferences: IR<boolean>;
135+
readonly batchEditPrefs?: BatchEditPrefs;
116136
}): MappingState => ({
117137
type: 'MappingState',
118138
showHiddenFields: getCache('wbPlanViewUi', 'showHiddenFields') ?? false,
@@ -124,6 +144,7 @@ export const getDefaultMappingState = ({
124144
focusedLine: 0,
125145
changesMade,
126146
mustMatchPreferences,
147+
batchEditPrefs,
127148
});
128149

129150
// REFACTOR: split component into smaller components
@@ -133,19 +154,23 @@ export function Mapper(props: {
133154
readonly onChangeBaseTable: () => void;
134155
readonly onSave: (
135156
lines: RA<MappingLine>,
136-
mustMatchPreferences: IR<boolean>
157+
mustMatchPreferences: IR<boolean>,
158+
batchEditPrefs?: BatchEditPrefs
137159
) => Promise<void>;
138160
// Initial values for the state:
139161
readonly changesMade: boolean;
140162
readonly lines: RA<MappingLine>;
141163
readonly mustMatchPreferences: IR<boolean>;
164+
readonly readonlySpec?: ReadonlySpec;
165+
readonly batchEditPrefs?: BatchEditPrefs;
142166
}): JSX.Element {
143167
const [state, dispatch] = React.useReducer(
144168
reducer,
145169
{
146170
changesMade: props.changesMade,
147171
lines: props.lines,
148172
mustMatchPreferences: props.mustMatchPreferences,
173+
batchEditPrefs: props.batchEditPrefs,
149174
},
150175
getDefaultMappingState
151176
);
@@ -264,7 +289,13 @@ export function Mapper(props: {
264289
const validationResults = ignoreValidation ? [] : validate();
265290
if (validationResults.length === 0) {
266291
unsetUnloadProtect();
267-
loading(props.onSave(state.lines, state.mustMatchPreferences));
292+
loading(
293+
props.onSave(
294+
state.lines,
295+
state.mustMatchPreferences,
296+
state.batchEditPrefs
297+
)
298+
);
268299
} else
269300
dispatch({
270301
type: 'ValidationAction',
@@ -294,6 +325,11 @@ export function Mapper(props: {
294325
mappingPathIsComplete(state.mappingView) &&
295326
getMappedFieldsBind(state.mappingView).length === 0;
296327

328+
const disableSave =
329+
props.readonlySpec === undefined
330+
? isReadOnly
331+
: Object.values(props.readonlySpec).every(Boolean);
332+
297333
return (
298334
<Layout
299335
buttonsLeft={
@@ -337,38 +373,61 @@ export function Mapper(props: {
337373
})
338374
}
339375
/>
340-
<MustMatch
341-
getMustMatchPreferences={(): IR<boolean> =>
342-
getMustMatchTables({
343-
baseTableName: props.baseTableName,
344-
lines: state.lines,
345-
mustMatchPreferences: state.mustMatchPreferences,
346-
})
347-
}
348-
onChange={(mustMatchPreferences): void =>
349-
dispatch({
350-
type: 'MustMatchPrefChangeAction',
351-
mustMatchPreferences,
352-
})
353-
}
354-
onClose={(): void => {
355-
/*
356-
* Since setting table as must match causes all of its fields to
357-
* be optional, we may have to rerun validation on
358-
* mustMatchPreferences changes
359-
*/
360-
if (
361-
state.validationResults.length > 0 &&
362-
state.lines.some(({ mappingPath }) =>
363-
mappingPathIsComplete(mappingPath)
364-
)
365-
)
376+
{typeof props.batchEditPrefs === 'object' ? (
377+
<ReadOnlyContext.Provider
378+
value={
379+
props.dataset.uploadresult?.success ??
380+
props.readonlySpec?.batchEditPrefs ??
381+
isReadOnly
382+
}
383+
>
384+
<BatchEditPrefsView
385+
prefs={props.batchEditPrefs}
386+
onChange={(prefs) =>
387+
dispatch({
388+
type: 'ChangeBatchEditPrefs',
389+
prefs,
390+
})
391+
}
392+
/>
393+
</ReadOnlyContext.Provider>
394+
) : null}
395+
<ReadOnlyContext.Provider
396+
value={props.readonlySpec?.mustMatch ?? isReadOnly}
397+
>
398+
<MustMatch
399+
getMustMatchPreferences={(): IR<boolean> =>
400+
getMustMatchTables({
401+
baseTableName: props.baseTableName,
402+
lines: state.lines,
403+
mustMatchPreferences: state.mustMatchPreferences,
404+
})
405+
}
406+
onChange={(mustMatchPreferences): void =>
366407
dispatch({
367-
type: 'ValidationAction',
368-
validationResults: validate(),
369-
});
370-
}}
371-
/>
408+
type: 'ChangeMustMatchPrefAction',
409+
mustMatchPreferences,
410+
})
411+
}
412+
onClose={(): void => {
413+
/*
414+
* Since setting table as must match causes all of its fields to
415+
* be optional, we may have to rerun validation on
416+
* mustMatchPreferences changes
417+
*/
418+
if (
419+
state.validationResults.length > 0 &&
420+
state.lines.some(({ mappingPath }) =>
421+
mappingPathIsComplete(mappingPath)
422+
)
423+
)
424+
dispatch({
425+
type: 'ValidationAction',
426+
validationResults: validate(),
427+
});
428+
}}
429+
/>
430+
</ReadOnlyContext.Provider>
372431
{!isReadOnly && (
373432
<Button.Small
374433
className={
@@ -396,11 +455,12 @@ export function Mapper(props: {
396455
>
397456
{isReadOnly ? wbText.dataEditor() : commonText.cancel()}
398457
</Link.Small>
399-
{!isReadOnly && (
458+
{!disableSave && (
400459
<Button.Small
401460
disabled={!state.changesMade}
402461
variant={className.saveButton}
403-
onClick={(): void => handleSave(false)}
462+
// This is a bit complicated to resolve correctly. Each component should have its own validator..
463+
onClick={(): void => handleSave(isReadOnly)}
404464
>
405465
{commonText.save()}
406466
</Button.Small>
@@ -557,7 +617,7 @@ export function Mapper(props: {
557617
customSelectSubtype: 'simple',
558618
fieldsData: mappingOptionsMenu({
559619
id: (suffix) => id(`column-options-${line}-${suffix}`),
560-
isReadOnly,
620+
isReadOnly: props.readonlySpec?.columnOptions ?? isReadOnly,
561621
columnOptions,
562622
onChangeMatchBehaviour: (matchBehavior) =>
563623
dispatch({

0 commit comments

Comments
 (0)