Skip to content

GO-4792: make csv export for anytype #2170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

AnastasiaShemyakinskaya
Copy link
Contributor

@AnastasiaShemyakinskaya AnastasiaShemyakinskaya commented Feb 27, 2025

  1. Create separate directory with object types csv (tasks.csv, pages.csv, relations.csv and etc). These files contain the list of objects with given type and their relations in headers.
object_types
├── page.csv
│ | source | title | detail |
│ |------|------------|-------------|
│ | page.md | First Page | details |
│ | page1.md| Second Page | details |
│
├── task.csv
│ | source | title | status |
│ |------|---------|---------|
│ | task.md | Buy milk | pending |
│
├── note.csv
│ | source | title |
│ |------|--------------|
│ | note.md | Meeting notes |
│
  1. Object types csv contain links to markdown files with objects.

  2. Collections and sets are converted to csv files with table, based on the first dataview view and visible relations. If object from collection is presented in result export, the row with object details is added to result table.

├── page.md
├── test.csv
│ title |  visible detail |
│ |-----------|--------------|
│ |Page title |  visible detail |

Signed-off-by: AnastasiaShemyakinskaya <[email protected]>
Signed-off-by: AnastasiaShemyakinskaya <[email protected]>
Copy link

github-actions bot commented Feb 27, 2025

Previous Coverage 50.5% of statements
New Coverage 50.5% of statements
Patch Coverage 53.5% of changed statements (223/417)

Coverage provided by https://github.com/seriousben/go-patch-cover-action

if info.Id == addr.AnytypeProfileId {
return false
}
if !isProtobuf && (!validTypeForNonProtobuf(sbType) || !validLayoutForNonProtobuf(info.Details)) {
if !shouldExportRelationsAndType && !validTypeForNonProtobuf(sbType) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validTypeForNonProtobuf could be renamed to validDocumentType according to export naming

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@@ -47,11 +43,11 @@ func (d *dot) Add(space smartblock.Space, st *state.State) error {
return nil
}

func (d *dot) Convert(sbType model.SmartBlockType) []byte {
func (d *dot) Convert(st *state.State, sbType model.SmartBlockType, filename string) []byte {
panic("not supported on windows")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it is not supported on mobiles as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

return str
}

func getStringValueFromDetail(details *domain.Details, key domain.RelationKey) string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also support int lists, e.g. restrictions

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

"github.com/anyproto/anytype-heart/pkg/lib/pb/model"
)

func ExtractHeaders(spaceIndex spaceindex.Store, keys []string) ([]string, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it ok for this function, if len(keys) != len(records) ?
We do not return error in current implementtion

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

return headersKeys, headersName, nil
}

func findDataviewBlock(st *state.State) *model.Block {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could return the very *model.BlockContentDtaview

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


func (c *Converter) extractHeaders(dataview *model.BlockContentDataview, spaceId string) ([]string, []string, error) {
var headersKeys []string
for _, relation := range dataview.Views[0].Relations {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we look into relations of first view?
It seems to be a product question, but we have some data inside dtaview block to calculate good view candidate for export. E.g. activeView field carries id of last active view within current session. view.Type handles type of view. Grid and List fit to CSV export, while Calendar or Graph not so much

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is product questions, I suppose

func NewMDConverter(s *state.State, fn FileNamer) converter.Converter {
return &MD{s: s, fn: fn}
func NewMDConverter(fn FileNamer, store objectstore.ObjectStore, knownDocs map[string]*domain.Details) converter.Converter {
return &MD{fn: fn, objectTypeFiles: csv.ObjectTypeFiles{}, store: store, listCsv: csv.NewConverter(store, knownDocs), knownDocs: knownDocs}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems it is better to initialize new struct vertically

return &MD{
		fn: fn, 
		objectTypeFiles: csv.ObjectTypeFiles{}, 
		store: store, 
		listCsv: csv.NewConverter(store, knownDocs), 
		knownDocs: knownDocs,
	}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Copy link

Testomat.io Report 🔴 SMOKE-TEST FAILED
Tests ✔️ 13 tests run
Summary 🔴 4 failed; 🟢 9 passed; 🟡 0 skipped
Duration 🕐 20 minutes, 56 seconds
Testomat.io Report 📊 Run #1326e40f
Job 🗂️ Smoke Tests / smoke-test
Operating System 🖥️ Linux X64

✅ Passed Tests (9)

  • Revoking an invite link (3 minutes, 13 seconds)
  • User cancels their join request (1 minute, 26 seconds)
  • User deletes the space and rejoins later (1 minute, 44 seconds) Passed on retry: 2
  • Owner changes the rights of a user from Viewer to Editor (1 minute, 57 seconds)
  • Owner approves a join request with Editor permissions (1 minute, 36 seconds)
  • Owner declines a join request (1 minute, 10 seconds)
  • Owner removes a participant from the space (1 minute, 25 seconds)
  • Owner deletes a space (1 minute, 27 seconds) Passed on retry: 2
  • Owner approves leave request (1 minute, 26 seconds)

🟥 Failures (4)

🔴 User deletes the space and rejoins later

Error: Test failed: The account did not sync within the expected time. Stack trace: Error: Timeout: Condition not met within the specified time. at Timeout. (/home/runner/work/anytype-heart/anytype-heart/support/api/services/utils.ts:231:11) at listOnTimeout (node:internal/timers:594:17) at processTimers (node:internal/timers:529:7) at Timeout. (/home/runner/work/anytype-heart/anytype-heart/support/api/services/utils.ts:231:11) at listOnTimeout (node:internal/timers:594:17) at processTimers (node:internal/timers:529:7) at World. (/home/runner/work/anytype-heart/anytype-heart/step_definitions/api/accountSteps.ts:233:13)

################[ Steps ]################
Given the server "default" 1 is running (3670.69ms)
And the user 1 is using client 1 (0.25ms)
And the user creates a new account on "staging" (1822.46ms)
And the account is synced within 60 seconds (62058.41ms)
Given the user creates an object 1 in the account (0.00ms)
And the user can open the object 1 (0.00ms)
And the user has shared his space and generated an invite link (0.00ms)
And the account is synced within 60 seconds (0.00ms)
And the server "default" 2 is running (0.00ms)
And the user 2 is using client 2 (0.00ms)
And the user creates a new account on "staging" (0.00ms)
And the user sends requests to join the space (0.00ms)
When the user 1 approves the join request of user 2 with "Viewer" rights (0.00ms)
And the user 2 is using client 2 (0.00ms)
And the user 2 leaves the shared space (0.00ms)
And the user sends requests to join the space (0.00ms)
Then the request is automatically approved as "Viewer" again (0.00ms)
And both accounts are deleted (0.00ms)
And both servers are stopped (0.00ms)

🔴 Owner deletes a space

Error: Test failed: The account did not sync within the expected time. Stack trace: Error: Timeout: Condition not met within the specified time. at Timeout. (/home/runner/work/anytype-heart/anytype-heart/support/api/services/utils.ts:231:11) at listOnTimeout (node:internal/timers:594:17) at processTimers (node:internal/timers:529:7) at Timeout. (/home/runner/work/anytype-heart/anytype-heart/support/api/services/utils.ts:231:11) at listOnTimeout (node:internal/timers:594:17) at processTimers (node:internal/timers:529:7) at World. (/home/runner/work/anytype-heart/anytype-heart/step_definitions/api/accountSteps.ts:233:13)

################[ Steps ]################
Given the server "default" 1 is running (3595.29ms)
And the user 1 is using client 1 (0.20ms)
And the user creates a new account on "staging" (1872.64ms)
And the account is synced within 60 seconds (62036.83ms)
Given the user creates an object 1 in the account (0.00ms)
And the user can open the object 1 (0.00ms)
And the user has shared his space and generated an invite link (0.00ms)
And the account is synced within 60 seconds (0.00ms)
And the server "default" 2 is running (0.00ms)
And the user 2 is using client 2 (0.00ms)
And the user creates a new account on "staging" (0.00ms)
And the user sends requests to join the space (0.00ms)
When the user 1 approves the join request of user 2 with "Editor" rights (0.00ms)
And the user 1 deletes the shared space (0.00ms)
And the user 2 is using client 2 (0.00ms)
Then the user can't open the object 1 (0.00ms)
And both accounts are deleted (0.00ms)
And both servers are stopped (0.00ms)

🔴 Sync on staging nodes

Error: Test failed: The account did not sync within the expected time. Stack trace: Error: Timeout: Condition not met within the specified time. at Timeout. (/home/runner/work/anytype-heart/anytype-heart/support/api/services/utils.ts:231:11) at listOnTimeout (node:internal/timers:594:17) at processTimers (node:internal/timers:529:7) at Timeout. (/home/runner/work/anytype-heart/anytype-heart/support/api/services/utils.ts:231:11) at listOnTimeout (node:internal/timers:594:17) at processTimers (node:internal/timers:529:7) at World. (/home/runner/work/anytype-heart/anytype-heart/step_definitions/api/accountSteps.ts:233:13)

################[ Steps ]################
Given the server "default" 1 is running (3606.82ms)
And the user is using client 1 (0.28ms)
And the user creates a new account on "staging" (1875.67ms)
And the user creates an object 1 in the account (9.92ms)
And the user can open the object 1 (10005.82ms)
And the account is synced within 80 seconds (82091.28ms)
And the server 1 is stopped (0.00ms)
And data is deleted (0.00ms)
And the server "default" 2 is running (0.00ms)
And the user is using client 2 (0.00ms)
When the user logs in to their account on "staging" (0.00ms)
Then the account should have an analyticsId, profile picture and name (0.00ms)
Then the user can open the object 1 (0.00ms)
And the account is deleted (0.00ms)
And the server 2 is stopped (0.00ms)

🔴 Sync on staging nodes (Will be retried 🔄)

Error: Test failed: The account did not sync within the expected time. Stack trace: Error: Timeout: Condition not met within the specified time. at Timeout. (/home/runner/work/anytype-heart/anytype-heart/support/api/services/utils.ts:231:11) at listOnTimeout (node:internal/timers:594:17) at processTimers (node:internal/timers:529:7) at Timeout. (/home/runner/work/anytype-heart/anytype-heart/support/api/services/utils.ts:231:11) at listOnTimeout (node:internal/timers:594:17) at processTimers (node:internal/timers:529:7) at World. (/home/runner/work/anytype-heart/anytype-heart/step_definitions/api/accountSteps.ts:233:13)

################[ Steps ]################
Given the server "default" 1 is running (3700.45ms)
And the user is using client 1 (0.25ms)
And the user creates a new account on "staging" (1861.55ms)
And the user creates an object 1 in the account (8.25ms)
And the user can open the object 1 (10004.70ms)
And the account is synced within 80 seconds (82074.39ms)
And the server 1 is stopped (0.00ms)
And data is deleted (0.00ms)
And the server "default" 2 is running (0.00ms)
And the user is using client 2 (0.00ms)
When the user logs in to their account on "staging" (0.00ms)
Then the account should have an analyticsId, profile picture and name (0.00ms)
Then the user can open the object 1 (0.00ms)
And the account is deleted (0.00ms)
And the server 2 is stopped (0.00ms)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants