Add language support for notebook cells

[tgodzik]

Is your feature request related to a problem? Please describe.

Currently, when using notebook cells in VS Code, Metals is unable to provide any language support. Most likely connected with an exception:

WARNING: Notification threw an exception: {
  "jsonrpc": "2.0",
  "method": "metals/didFocusTextDocument",
  "params": [
    "vscode-notebook-cell:Untitled-1.ipynb?jupyter-notebook#ch0000000untitled"
  ]
}
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0(GenericEndpoint.java:67)
	at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.notify(GenericEndpoint.java:152)
	at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleNotification(RemoteEndpoint.java:220)
	at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.consume(RemoteEndpoint.java:187)
	at org.eclipse.lsp4j.jsonrpc.TracingMessageConsumer.consume(TracingMessageConsumer.java:114)
	at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage(StreamMessageProducer.java:194)
	at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen(StreamMessageProducer.java:94)
	at org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run(ConcurrentMessageProcessor.java:113)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0(GenericEndpoint.java:65)
	... 12 more
Caused by: java.nio.file.FileSystemNotFoundException: Provider "vscode-notebook-cell" not installed
	at java.nio.file.Paths.get(Paths.java:147)
	at scala.meta.internal.metals.MetalsEnrichments$XtensionString.toAbsolutePath(MetalsEnrichments.scala:577)
	at scala.meta.internal.metals.MetalsEnrichments$XtensionString.toAbsolutePath(MetalsEnrichments.scala:574)
	at scala.meta.internal.metals.MetalsLanguageServer.didFocus(MetalsLanguageServer.scala:1055)
	... 17 more

Describe the solution you'd like

Support all the standard language features. We would most likely also need to provide some notebook specific features (such as using values from other cells?).

Describe alternatives you've considered

None I can think of.

Additional contex

This came up in scalameta/metals#3213 and is separate to supporting https://code.visualstudio.com/api/extension-guides/notebook

Search terms

notebooks vscode

[Quafadas]

@tgodzik This is something that I'd love to have in metals! Depending on how hard it is, I would consider a tilt at it :-)...

I've done some experiments in the past, which prove that one can indeed run scala in a VSCode notebook, simply by wheeling in the Almond kernel. However, it's a "raw" experience right now.

I hope, that could be improved in a couple of ways, which I believe to be 'independent'.

1. convince the notebooks to recognise the metals LSP server. I did manage to get metals working in a jupyterhub via LSP, so something is at least close already. My (naive) hope is that this might just be a case of trawling some documentation to get a config right. This would be cool in its own right, even if I didn't mange to go further.

2. FTW, what would be amazing, would be if it was possible to execute a notebook cell, in the context of a project - same like kicking up an ammonite / scala console out the build tool. I fear, that this may be a rather difficult problem. My hope would be that it was possible to 'trick' almond into having the project context, without having to change almond itself. I am not yet sure, how... but hope springs eternal.

Is this something you'd devoted any thought to?

If so, is the above rational? On a scale of 1-10, how hard would you perceive these problems to be? Something much above 5, and it's likely I'm out of my skill zone, sadly.

If they aren't too bad, would you be willing to guide an effort along these lines, and / or set out a solution sketch? I'm not familiar with the project, so if you had a hint on where to look, it would be great, or any other thoughts?

[tgodzik]

@Quafadas thanks for showing the interest! This is actually part of GSoC projects this year, but there it's possible it will not be picked. I will let you know once we have the results.

convince the notebooks to recognise the metals LSP server. I did manage to get metals working in a jupyterhub via LSP, so something is at least close already. My (naive) hope is that this might just be a case of trawling some documentation to get a config right. This would be cool in its own right, even if I didn't mange to go further.

It already recognises the notebook cells written in Scala from what I've seen, but there is no support for the particular URI, which we would need to work on (we get the exceptions as shown in the description)

FTW, what would be amazing, would be if it was possible to execute a notebook cell, in the context of a project - same like kicking up an ammonite / scala console out the build tool. I fear, that this may be a rather difficult problem. My hope would be that it was possible to 'trick' almond into having the project context, without having to change almond itself. I am not yet sure, how... but hope springs eternal.

I might be possible to execute almond with the proper classpath and settings to make it work, but I am not familiar with the almond API right now.

If so, is the above rational? On a scale of 1-10, how hard would you perceive these problems to be? Something much above 5, and it's likely I'm out of my skill zone, sadly.

I think it's around 5, but mostly it would be getting all the existing tools to work together, so a bit of things to figure out and change, but nothing that would require large scale rewrites.

If they aren't too bad, would you be willing to guide an effort along these lines, and / or set out a solution sketch? I'm not familiar with the project, so if you had a hint on where to look, it would be great, or any other thoughts?

I will let you know if this project gets to be a part of gsoc, if not we can try and figure it out together 😄

[Quafadas]

This is actually part of GSoC projects this year

Ohhhh... excellent.

Thanks for taking the time to write back. I'll keep my fingers crossed that it gets picked up by that project!

[tgodzik]

This is actually part of GSoC projects this year

Ohhhh... excellent.

Thanks for taking the time to write back. I'll keep my fingers crossed that it gets picked up by that project!

This project hasn't been picked for GSoC, so if you are interested in working on it that would be awesome! I can try and provide some pointers, but generally I think the work you already did could be automated almond setup plus making sure that the notebook file provider works (there might be already an implementation for Java, but not sure.)

[Quafadas]

*This project hasn't been picked for GSoC * Oh no! I’m away right now, so will write back in a week or so. I’d like to have a think before committing to spending a lot of time, and probably do need to figure out just how long it might take. I do think this would be a very cool thing to have though!

…

On Mon, 23 May 2022 at 10:06, Tomasz Godzik ***@***.***> wrote: This is actually part of GSoC projects this year Ohhhh... excellent. Thanks for taking the time to write back. I'll keep my fingers crossed that it gets picked up by that project! This project hasn't been picked for GSoC, so if you are interested in working on it that would be awesome! I can try and provide some pointers, but generally I think the work you already did could be automated almond setup plus making sure that the notebook file provider works (there might be already an implementation for Java, but not sure.) — Reply to this email directly, view it on GitHub <#236 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF57BUDQS3VTDED6VPKREP3VLM36XANCNFSM5GLK7ARQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tgodzik]

*This project hasn't been picked for GSoC * Oh no! I’m away right now, so will write back in a week or so. I’d like to have a think before committing to spending a lot of time, and probably do need to figure out just how long it might take. I do think this would be a very cool thing to have though!
…

I think this one can be done iteratively, and I can fill in any missing pieces if it turns out you don't have time. Also, it's totally fine if you at some point decide that you can't work on it further.

[Quafadas]

@tgodzik So I'm definitely up for this, although I have no experience with this sort of tooling. Should be fun.

Sadly however for "reasons", I need to ban myself before mid-July - I have something else to focus on that I can't drop.

I will spend some time now, trying to get started with changing metals and getting over the "contributor getting started" hump.

Hopefully, I'll be posting up here around mid-July / early August, when I think I can allocate some time to this (if it's still open - which weirdly, I hope it is!).

[Quafadas]

@tgodzik I hope to have some time over the coming weeks to look at this... my plan over the coming days is to look at getting started with changing metals and then go from there...

[tgodzik]

Sure, I also mentioned it as a possible student project at EPFL, but that would be in septemeber and we can remove it from the list if you manage to take a look at it. Similar to GSoC this might also not be picked up at all.

Some pointers:

• getting started guide - https://scalameta.org/metals/docs/contributors/getting-started
• downloading dependencies, could be useful to download almond - https://github.com/scalameta/metals/blob/main/metals/src/main/scala/scala/meta/internal/metals/Embedded.scala#L271

[Quafadas]

@tgodzik So, I have invested a little time and sat down with this, and have (already!) reached the stage at which I'm not sure how to proceed. This is complex for me!

For the sake of transparency, here is my branch. Changes thus far are basically trivial - add an almond version, and follow the pattern set out in the download methods. To be clear, it doesn't work!

[info] 2022.07.10 16:38:20 INFO  Downloading almond (for notebook support)
[error] Exception in thread "main" coursierapi.error.SimpleResolutionError$1: Error downloading sh.almond:scala-kernel_2.13.8:0.13.0
[error]   not found: /Users/simon/.ivy2/local/sh.almond/scala-kernel_2.13.8/0.13.0/ivys/ivy.xml

Because that artefact, doesn't exist.
Branch
https://github.com/Quafadas/metals/tree/jupyter_notebooks

Maven
https://mvnrepository.com/artifact/sh.almond/scala-kernel

Given that maintenance of the almond project itself is somewhat beyond the scope of this issue... that has some implications. Here are the things I'm now unsure about.

1. Which artefact to even download (!)...
2. As almond interacts with the scala kernel, I believe it it publishes against the specific kernel versions rather than the binary versions. The versions of scala Almond supports / publish will not intersect with metals published / supported versions except through chance.
3. What the purpose of the download step is? Is it exactly to allow projects like almond to exist inside metals, without the explicit, and hence transitive dependance?

What (I think) about the above... I hope I'm not heading in the wrong direction

1. I believe the artefact of interest to be https://mvnrepository.com/artifact/sh.almond/scala-kernel against the assumptions that
   1.1 our aim is to instantiate one of these; https://github.com/almond-sh/almond/blob/main/modules/scala/scala-kernel/src/main/scala/almond/ScalaKernel.scala.
   1.1 and hence have metals own an almond kernel... for supported versions?
2. For mismatched almond / metals scala (x.x.x) version problem, I assume I could somehow use the coursier API to check the set of published versions of that scala-kernal artefact, and intersect this with the set of supported metals versions... and only attempt to download the intersection?

If the idea was instead to use metals to construct an almond launcher, which would then be launched as an independent service, then I guess this download step would be done rather differently by using coursier to bootstrap a launcher? I am assuming, that an "independent launcher" strategy, means that it would ultimately be very difficult to get the class paths of our build into the notebook...and hence that it is not the right direction of travel?

I'm sorry this post is so long...

[tgodzik]

Which artefact to even download (!)...
As almond interacts with the scala kernel, I believe it it publishes against the specific kernel versions rather than the binary versions. The versions of scala Almond supports / publish will not intersect with metals published / supported versions except through chance.

That's a separate issue altogether, I can try ask about it. We could just have the latest supported version specified similar to how we deal currently with Ammonite. We can show a warning if a version is not yet supported

What the purpose of the download step is? Is it exactly to allow projects like almond to exist inside metals, without the explicit, and hence transitive dependance?

We don't want to always download all possible dependecies especially if we would need to download it for each Scala version, which would mean downloading every Scala version for the user even if they don't need it. The download step is used to avoid that.

I believe the artefact of interest to be https://mvnrepository.com/artifact/sh.almond/scala-kernel against the assumptions that
1.1 our aim is to instantiate one of these; https://github.com/almond-sh/almond/blob/main/modules/scala/scala-kernel/src/main/scala/almond/ScalaKernel.scala.
1.1 and hence have metals own an almond kernel... for supported versions?

We could use ShellRunner for running the specific kernel. When is the kernel actually needed? Only when running?

For mismatched almond / metals scala (x.x.x) version problem, I assume I could somehow use the coursier API to check the set of published versions of that scala-kernal artefact, and intersect this with the set of supported metals versions... and only attempt to download the intersection?

We can put that information in the build.sbt and show the users proper warnings.

[Quafadas]

This is potentially a very useful note

microsoft/vscode-jupyter#10813

https://jupyter-client.readthedocs.io/en/latest/kernels.html#kernel-specs

microsoft/vscode-jupyter#10151

In summary; this now all looks tractable, if I can figure out how to bootstrap almond out of the coursier API.

The first link says VSCode respects the kernel specs. The second link sets out the kernel specs, and suggests that a JSON file which allows one to, for example set environment variables for the notebook process.

The third link sets out how one may influence the class path through setting environment variables.

[Quafadas]

I made a slight change (pushed to branch above) to the shell runner to allow it to accept extra repositories when downloading dependancies, because almond has a dependancy on JVM repr which appears only to be available on jitpack. Easily solved.

In the metals test project, if I fire up a console and run this script, it downloads the key almond kernel dependancy. The last line even installs the kernel in (apparently)... the right place.

import scala.meta.internal.metals._
import scala.meta.internal.builds.ShellRunner
import coursierapi._
implicit val ec: scala.concurrent.ExecutionContext = scala.concurrent.ExecutionContext.global

val absPath = scala.meta.io.AbsolutePath("/Users/simon/Code/metals")

val aClient = new tests.TestingClient(absPath, scala.meta.internal.metals.Buffers())

val uConf = scala.meta.internal.metals.UserConfiguration()
val cConf = new ClientConfiguration(new MetalsServerConfig())
 
val t = Time.system.currentMillis()

val s = new StatusBar(() => aClient, Time.system, ProgressTicks.braille, cConf)

val sr = new ShellRunner(aClient, () => uConf, Time.system, s)

val playdoh =  scala.meta.io.AbsolutePath("/Users/simon/Code/playdoh")

val almondDep =     Dependency.of(
      "sh.almond",
      s"scala-kernel_2.13.7",
      "0.13.0"
    )
    
val jvmReprRepo =   coursierapi.MavenRepository
    .of("https://maven.imagej.net/content/repositories/public/")

 sr.runJava(almondDep, "almond.ScalaKernel", playdoh, List("--install", "--command", "java -jar /Users/simon/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/sh/almond/scala-kernel_2.13.7/0.13.0/scala-kernel_2.13.7-0.13.0.jar", "--id", "metalsAlmond", "--display-name", "metalsAlmond", "--global", "false", "--jupyter-path", "/Users/simon/Library/Jupyter/kernels/" ), false, extraRepos=Array(jvmReprRepo))

// "--copy-launcher", "true" too see what it's trying to run.

This gets tantalisingly close. It places the kernel in the right place, and it's even found by VSCode! There are two problems;

1. I manually figured out the path to the almond kernel artefact. Is there an obvious way to do that programmatically?

2. Bigger issues, is that the kernel cannot be started outside of that metals context. A manual attempt at doing so

java -jar /Users/simon/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/sh/almond/scala-kernel_2.13.7/0.13.0/scala-kernel_2.13.7-0.13.0.jar
Error: Unable to initialize main class almond.ScalaKernel
Caused by: java.lang.NoClassDefFoundError: scala/Product

So what I think is happening, is that normally, coursier is bundling together all he stuff the JAR needs to launch vanilla almond. That's not gonna work for us. Based on my google-fu,
https://stackoverflow.com/questions/18413014/run-a-jar-file-from-the-command-line-and-specify-classpath

I guess that I now need to tamper with class paths...

I do not know much about class paths. I don't even know if that's embarrassing or not :-(. Can I get the class path of the project metals is working on before using the shell runner? Any ideas?