Metadata service, client, UI overview

[ellisonbg]

This is an issue that provides an overview of the proposed metadata service, client, and UI being developed in this repo.

Background

Entities in the JupyterLab universe (notebooks, text files, datasets, models, visualizations, etc.) often have rich context and metadata associated with them. Examples include:

• Authors
• Related works and publications
• Data catalogs
• Institutions
• Publishers
• Code snippets

This rich context is incredibly useful to groups of people working with code and data. This goal of this work is to build a metadata architecture to enable Jupyter users to collaboratively create and explore metadata for any entity in the Jupyter universe.

What metadata standard

We have considered a number of existing metadata standards, and the one that is emerging as a top candidate is that of https://schema.org/. It appears to be rich enough to describe the different types of metadata we encounter in the Jupyter universe. In talking with potential users of this system, that flexibility seems to be important.

See jupyterlab/jupyterlab#5733 for additional discussion about metadata schema.

Implementation

The current proposal is to create a Jupyter notebook server extension that is a GraphQL service for the relevant subset of the schema.org metadata schema. We haven't worked through what subset of the schema is relevant for Jupyter, but it will probably be the document and data related ones (probably won't start with things like https://schema.org/FlightReservation).

The usage of GraphQL is important because we imagine a wide range of complex UIs being built to display and edit this highly nested metadata. Being able to get back and edit rich data in single queries will be really helpful on the frontend.

We haven't decided if this notebook server extension will be written in Python or node.js (or both), but it shouldn't really matter.

For a client, we are imagining a TypeScript based library that provides a thin, well-typed API for talking to the service.

The notebook server and TypeScript client library should be entirely independent of JupyterLab and useful outside of it.

Finally, we plan on creating a JupyterLab extension that offers a user experience for editing and viewing metadata for entities in JupyterLab. Initial work will focus on notebooks, data sets, text documents.

Initially, this repo will create our explorations of the notebook server, TypeScript client, and JupyterLab UI extension, but these may be separated out over time.

[ellisonbg]

Another big question that is coming up is how to treat code snippets. While I list them above as metadata, we may want to have a separate code snippet extension in JupyterLab.

[saulshanabrook]

Some notes from our meeting:

The primary initial use case is one JupyterLab server that has multiple simultaneous users. One person opens a CSV file and adds some metadata about it, like that it has a author or something, and then another user who has the file open sees this field, when look at the metadata for that file.

We discussed using files instead of GraphQL to store this, i.e. like one file per metadata object, but then talked about then how we couldn't have two users edit the same object at the same time, one would get clobbered.

Resources:

• https://github.com/prisma/prisma: Support MySQL and PostgreSQL databases
  • See Production deploy recipe prisma/prisma#2460 for supporting SQLite or in memory
• https://comunica.github.io/Article-ISWC2018-Demo-GraphQlLD/ GraphQL and JSON-LD

  • In this work, we propose GraphQL-LD as a technique for combining the worlds of GraphQL and the Semantic Web. We provide an implementation of this approach, and demonstrate this with a set of example queries.

    In future work, we intend to formalize our GraphQL-LD conversion algorithm. Furthermore, we intend to improve the way in which we determine which variables should be considered singular or plural. OWL’s InverseFunctionalProperty or JSON-LD framing are potential options that we consider for this.

    In summary, this work allows GraphQL developers to query the Linked Open Data cloud. But also Linked Data experts can use it an an alternative to SPARQL.

  • https://github.com/rubensworks/graphql-to-sparql.js

[rgbkrk]

Let me just say I'm really happy about the direction with GraphQL in Jupyter. As far as tooling is concerned, I'd prefer to use the really well supported node.js backend tooling for GraphQL. It also helps that you can share types from backend to frontend. This likely means having to use something like nbserverproxy though.

[saulshanabrook]

I'd prefer to use the really well supported node.js backend tooling for GraphQL

Me too. Do you have any suggestions for key packages that would be helpful to look at? Subscriptions seem pretty useful, and I was looking at this: https://github.com/prisma/prisma-examples/tree/master/typescript/graphql-subscriptions

EDIT: looks like rec is to move to Apollo Server 2 instead of using yoga graphql-hive/graphql-yoga#449 (comment)

[rgbkrk]

Yeah, I'd recommend Apollo Server.

@captainsafia and I have been working on a new server that provides a GraphQL API for managing communication between a Jupyter kernel and clients.

[captainsafia]

You can find the code for this work at https://github.com/nteract/nteract/tree/master/packages/kernel-relay.

It's designed to provide more interaction-based endpoints as opposed to resource-based endpoints in REST APIs. For example, I want to launch a kernel, I want to subscribe to the status of a kernel, I want to execute this code snippet, etc.

[bollwyvl]

Yeah, I'd recommend Apollo...

💯 to apollo's front end stack.

Server.

+0 in hub deployments, as we've already got configurable-http-proxy there.

👎 to nodejs on the single user server. I'll toss out this prototype based on graphene which, while having some growth challenges, in 2019 is still a lot more supportable on end-user machines than nodejs. Also if we get too far down the "reference implementations of frontend and backend can share code," there's a pretty good chance there will never be another implementation of either.

Stepping back from either implementation, would be to have a canonical:

1. serialization schema a la nbformat (graphql doesn't dictate one, but turns out you gotta have something)
2. json-ld context
3. a graphql schema graphql SDL
   ...which "a jupyter metadata server" must conform to, akin to nbformat, but with only an implemented conformance test suite.

Given that an (opinionated) GraphQL schema from a json-ld context, i think this gives us the most robustness. There isn't a lot of precedence (or support) for combining un-coordinated GraphQL type libraries at present, and I'd really hate it if this feature set regressed on the hackability of our other tools: if a community wants to add and search by their microscope metadata schema to something, I don't want the answer to be fork, because it doesn't "fit" in "schema.org" or "Jupyter's GraphQL thing"

[saulshanabrook]

👎 to nodejs on the single user server. I'll toss out this prototype based on graphene which, while having some growth challenges, in 2019 is still a lot more supportable on end-user machines than nodejs.

One thing to keep in mind is that I believe we would need graphql subscriptions support. In my last chat with Brian, one of the big drivers of using graphql is the ability for two users to be editing the same metadata at the same time and to have the changes mirrored between the two. Subscriptions would allow each client to get notified when another client made a change with the updated data, I believe (I am not a GraphQL expert, someone correct me if you don't need subscriptions for this use case).

They aren't included in Graphene at the moment (graphql-python/graphene#430 (comment), graphql-python/graphene#393), but a few people have added custom support to use Django's channels with graphene: https://github.com/eamigo86/graphene-django-subscriptions, https://github.com/datadvance/DjangoChannelsGraphqlWs

json-ld context

TBH I am not familiar with what JSON-LD context really is. I will have to investigate that. I get scared whenever I see RDF/SPARQL cause it's a whole other world that I don't know much about. Do we keep a bunch of RDF files in disk? Run a SPARQL server?

AFAIK the initial work on the metadata service isn't about using graphql or schema.org for notebook's themselves. Instead, my understanding is that it's more like a seperate context microservice. You might have comments on a notebook, and those would be stored persistently with some reference to the notebook, like it's file path and the cell ID of the comment.

[ellisonbg]

Thanks everyone. One of the benefits of going with GraphQL is that in the long term, the details of the server implementation, persistence mechanism, python/node.js, etc. are less important than the protocol, GraphQL schema, etc.

I would love to be able use Python for this, but I think that today it makes sense to use the approach that enables us to explore and iterate quickly on the schema and queries, with a solid GraphQL implementation that won't get in our way. To me that suggests starting with Apollo. I do think that jsonld will be important in the long term, but I don't think we need to tackle that starting out.

[bollwyvl]

isn't about using graphql or schema.org for notebook's themselves

Sure, but we have a lot of shipped stuff that look like complex, potentially evented graph data models! Also if we're talking about putting any of this stuff into nbformat itself, it's worth thinking about how these things might play.

need graphql subscriptions

GraphQL isn't going to solve the conflict-resolution problem, but it's great that it has a spec for delivering changes and multiple implementations. On that note, added subscriptions to contents on that prototype based on this PR:

jsonld will be important in the long term, but I don't think we need to tackle that starting out.

Like it or lump, if we're buying schema.org and web annotation, we're getting JSON-LD.

JSON-LD context really is

Those two contexts are how an SEO consortium and a standards body, respectively, see parts of the world and name things, and do have some incompatibilities (17, but some are spurious).

A key, relevant distinction is that both define Dataset: schema.org defines it's own, while web annotation uses dublin core. ANYHOO... it doesn't really matter, as we will end up needing a jupyter one at some point, but it's worth being on the lookout for things that the data can do for us.

Do we keep a bunch of RDF files in disk? Run a SPARQL server?

Like any other JSON, you can treat JSON-LD as a serialization format, (de-)normalize it, and then store it however you want, or accept its graph nature (and maybe also normalize it, unless your store speaks JSON-LD directly).

Taking the former route, flat files first is great (see FileContentsManager)! However, once you have multiple writers, it's hard to ignore Postgres (BSD-like) with LISTEN for events and the ability to store and query JSON. couchdb (APL) was super fun back in the day, and would probably still work great.

Taking the latter, the most cross-platform graph database I've used is virtuoso (GPL). RedisGraph (APL+CrazyClause), edgedb (APL), and gundb (APL) are new and fast and cool. All of them let you do things that can be pretty tricky with an RDBMS, but no two of those suggestions use the same query language natively. But yeah, the major languages with multiple implementations for arbitrary, potentially circular graph traversal are probably SPARQL and Gremlin.

[xmnlab]

The notebook server and TypeScript client library should be entirely independent of JupyterLab and useful outside of it.

so it should create a separated server for metadata and each user should specify the metadata server manually (like a settings page) ?

[Dessix]

In response to @saulshanabrook regarding subscriptions not being available in Graphene at the moment, I have had some success using GraphQL_WS in conjunction with AIOHTTP and Graphene for subscriptions.

The support from that project seems seriously finicky in some areas, with the Flask implementation seemingly leaking all sessions, but the AIOHTTP variant seems to work acceptable in my testing.

[bollwyvl]

On Fri, Jan 11, 2019, 15:13 Zoey ***@***.***> wrote: In response to @saulshanabrook <https://github.com/saulshanabrook> regarding subscriptions not being available in Graphene at the moment, I have had some success using GraphQL_WS <https://github.com/graphql-python/graphql-ws/> in conjunction with AIOHTTP and Graphene for subscriptions.

Right, my finding was that GraphQL_ws is pretty much good to go. Thanks for the corroboration! The support from that project seems seriously finicky in some areas, with

the Flask implementation seemingly leaking all sessions, but the AIOHTTP variant seems to work acceptable in my testing.

Luckily, tornado's model almost exactly matches other async models, without any of the gevent black (green?) magic. While I'll agree the GraphQL implementation doesn't really matter, I'm just going to highlight the challenges end (science, business, education) users have been having in even installing a sane environment for JupyterLab's hidden node dependency, much less running its sole function (webpack). If a node-based deployment can be gotten to a yarn-like single file, such as the one bundled with lab, and never even touches npm, and is installable with pip, it _could_ be viable for the average user. So back to my original point: starting with a versioned... - GraphQL sdl - serialization json schema - black box conformance suite ...will concentrate the discussion on the types, and not on language/vendor-specifics. If the reference implementation (and test suite) is node-based, so be it!

…

[saulshanabrook]

so it should create a separated server for metadata and each user should specify the metadata server manually (like a settings page) ?

@xmnlab Yeah I think that would be a fine way to do it.

As a default, it might make sense to proxy a metadata server through the notebook server using something like https://github.com/jupyterhub/jupyter-server-proxy and start that with the notebook.

I see the ideal default workflow being:

1. User pip installs some python package for the server
2. User installs the jupyterlab extensions
3. User starts up jupyterlab, this starts up metadata server behind the proxy
4. User opens jupyterlab, metadata client connects to default metadata server behind proxy

I think starting out with the metadata server running separately and having the user input the address in the client makes sense. Then we could always add the proxy later and add a default.

[saulshanabrook]

@xmnlab It looks like with the apollo server you define your mutations and then provide a custom javascript function for the definition. So at first, it seems like we could just story everything in memory if that's easiest in the server (https://blog.apollographql.com/react-graphql-tutorial-mutations-764d7ec23c15#f370)