diff --git a/docs/cli.md b/docs/cli.md index df93409d7d..e06f15c3f8 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -366,7 +366,7 @@ The `clone` command downloads a pipeline from a Git-hosting platform into the *c : Service hub where the project is hosted. Options: `gitlab` or `bitbucket`. `-r` (`master`) -: Revision to clone - It can be a git branch, tag, or revision number. +: Revision of the project to clone (either a git branch, tag or commit SHA number). `-user` : Private repository user name. @@ -415,6 +415,9 @@ The `config` command is used for printing the project's configuration i.e. the ` `-properties` : Print config using Java properties notation. +`-r, -revision` +: Revision of the project (either a git branch, tag or commit SHA number). + `-a, -show-profiles` : Show all configuration profiles. @@ -535,12 +538,18 @@ The `drop` command is used to remove the projects which have been downloaded int **Options** +`-a, -all-revisions` +: For specified project, drop all revisions. + `-f` : Delete the repository without taking care of local changes. `-h, -help` : Print the command usage. +`-r, -revision` +: Revision of the project to drop (either a git branch, tag or commit SHA number). + **Examples** Drop the `nextflow-io/hello` project. @@ -664,7 +673,7 @@ $ nextflow info [options] [project] **Description** -The `info` command prints out the nextflow runtime information about the hardware as well as the software versions of the Nextflow version and build, operating system, and Groovy and Java runtime. It can also be used to display information about a specific project. +The `info` command prints out the nextflow runtime information about the hardware as well as the software versions of the Nextflow version and build, operating system, and Groovy and Java runtime. It can also be used to display information about a specific project; in this case, note how revisions marked as `P` are pulled locally. If no run name or session id is provided, it will clean the latest run. @@ -878,6 +887,9 @@ The `list` commands prints a list of the projects which are already downloaded i **Options** +`-a, -all-revisions` +: For each project, also list revisions. + `-h, -help` : Print the command usage. @@ -1072,7 +1084,7 @@ The `pull` command downloads a pipeline from a Git-hosting platform into the glo : Service hub where the project is hosted. Options: `gitlab` or `bitbucket` `-r, -revision` -: Revision of the project to run (either a git branch, tag or commit hash). +: Revision of the project to pull (either a git branch, tag or commit SHA number). : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined. `-user` @@ -1217,7 +1229,7 @@ The `run` command is used to execute a local pipeline script or remote pipeline : Execute the script using the cached results, useful to continue executions that was stopped by an error. `-r, -revision` -: Revision of the project to run (either a git branch, tag or commit hash). +: Revision of the project to run (either a git branch, tag or commit SHA number). : When passing a git tag or branch, the `workflow.revision` and `workflow.commitId` fields are populated. When passing only the commit hash, `workflow.revision` is not defined. `-stub-run, -stub` @@ -1426,6 +1438,9 @@ The `view` command is used to inspect the pipelines that are already stored in t `-q` : Hide header line. +`-r, -revision` +: Revision of the project (either a git branch, tag or commit SHA number). + **Examples** Viewing the contents of a downloaded pipeline. diff --git a/docs/developer/diagrams/nextflow.scm.mmd b/docs/developer/diagrams/nextflow.scm.mmd index 1c52075372..a512e36a75 100644 --- a/docs/developer/diagrams/nextflow.scm.mmd +++ b/docs/developer/diagrams/nextflow.scm.mmd @@ -8,6 +8,7 @@ classDiagram class AssetManager { project : String + revision : String localPath : File mainScript : String repositoryProvider : RepositoryProvider diff --git a/docs/sharing.md b/docs/sharing.md index 2db5b9b527..64dc771641 100644 --- a/docs/sharing.md +++ b/docs/sharing.md @@ -58,6 +58,15 @@ nextflow run nextflow-io/hello -r v1.1 It will execute two different project revisions corresponding to the Git tag/branch having that names. +:::{versionadded} 24.03.0-edge +::: + +Nextflow downloads and locally maintains each explicitly requested Git branch, tag or commit ID in a separate directory path, thus enabling to run multiple revisions of the same pipeline at the same time. Each downloaded revision is stored in a sister path to the default revision one, featuring an extra suffix string `:`. + +:::{warning} +If you really care about reproducibility of your pipelines, you should explicitly refer to them by tag or commit ID, rather than my branch. This is because the same branch will point to different underlying commits over time, as pipeline development goes on. This caveat is particularly relevant in a scenario where multiple people manage and share the same local collection of pipelines. +::: + ## Commands to manage projects The following commands allows you to perform some basic operations that can be used to manage your projects. @@ -94,13 +103,13 @@ repository : http://github.com/nextflow-io/hello local path : $HOME/.nextflow/assets/nextflow-io/hello main script : main.nf revisions : -* master (default) +P master (default) mybranch - v1.1 [t] +P v1.1 [t] v1.2 [t] ``` -Starting from the top it shows: 1) the project name; 2) the Git repository URL; 3) the local folder where the project has been downloaded; 4) the script that is executed when launched; 5) the list of available revisions i.e. branches and tags. Tags are marked with a `[t]` on the right, the current checked-out revision is marked with a `*` on the left. +Starting from the top it shows: 1) the project name; 2) the Git repository URL; 3) the local path where the default project can be found (alternate revisions are in sister paths with an extra suffix `:`); 4) the script that is executed when launched; 5) the list of available revisions i.e. branches and tags. Tags are marked with a `[t]` on the right, the locally pulled revisions are marked with a `P` on the left. ### Pulling or updating a project diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy index 9c582aaaf6..b7be19d0b6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdClone.groovy @@ -15,6 +15,7 @@ */ package nextflow.cli + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -37,7 +38,7 @@ class CmdClone extends CmdBase implements HubOptions { @Parameter(required=true, description = 'name of the project to clone') List args - @Parameter(names='-r', description = 'Revision to clone - It can be a git branch, tag or revision number') + @Parameter(names='-r', description = 'Revision of the project to clone (either a git branch, tag or commit SHA number)') String revision @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth') @@ -52,11 +53,11 @@ class CmdClone extends CmdBase implements HubOptions { Plugins.init() // the pipeline name String pipeline = args[0] - final manager = new AssetManager(pipeline, this) + final manager = new AssetManager(pipeline, revision, this) // the target directory is the second parameter // otherwise default the current pipeline name - def target = new File(args.size()> 1 ? args[1] : manager.getBaseName()) + def target = new File(args.size()> 1 ? args[1] : manager.getBaseNameWithRevision()) if( target.exists() ) { if( target.isFile() ) throw new AbortOperationException("A file with the same name already exists: $target") @@ -68,9 +69,9 @@ class CmdClone extends CmdBase implements HubOptions { } manager.checkValidRemoteRepo() - print "Cloning ${manager.project}${revision ? ':'+revision:''} ..." - manager.clone(target, revision, deep) + print "Cloning ${manager.getProjectWithRevision()} ..." + manager.clone(target, deep) print "\r" - println "${manager.project} cloned to: $target" + println "${manager.getProjectWithRevision()} cloned to: $target" } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy index 5ff2156b18..851261c67b 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdConfig.groovy @@ -44,6 +44,9 @@ class CmdConfig extends CmdBase { @Parameter(description = 'project name') List args = [] + @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names=['-a','-show-profiles'], description = 'Show all configuration profiles') boolean showAllProfiles @@ -183,7 +186,7 @@ class CmdConfig extends CmdBase { return file.parent ?: Paths.get('/') } - final manager = new AssetManager(path) + final manager = new AssetManager(path, revision) manager.isLocal() ? manager.localPath.toPath() : manager.configFile?.parent } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy index 9d67190a54..fdad006d55 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdDrop.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_DELIM + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -39,6 +41,12 @@ class CmdDrop extends CmdBase { @Parameter(required=true, description = 'name of the project to drop') List args + @Parameter(names=['-r','-revision'], description = 'Revision of the project to drop (either a git branch, tag or commit SHA number)') + String revision + + @Parameter(names=['-a','-all-revisions'], description = 'For specified project, drop all revisions') + Boolean allrevisions + @Parameter(names='-f', description = 'Delete the repository without taking care of local changes') boolean force @@ -48,18 +56,34 @@ class CmdDrop extends CmdBase { @Override void run() { Plugins.init() - def manager = new AssetManager(args[0]) - if( !manager.localPath.exists() ) { - throw new AbortOperationException("No match found for: ${args[0]}") + + List dropList = [] + if ( allrevisions ) { + def referenceManager = new AssetManager(args[0]) + referenceManager.listRevisions().each { + dropList << new AssetManager(it.tokenize(REVISION_DELIM)[0], it.tokenize(REVISION_DELIM)[1]) + } + } else { + dropList << new AssetManager(args[0], revision) } - if( this.force || manager.isClean() ) { - manager.close() - if( !manager.localPath.deleteDir() ) - throw new AbortOperationException("Unable to delete project `${manager.project}` -- Check access permissions for path: ${manager.localPath}") - return + if ( !dropList ) { + throw new AbortOperationException("No revisions found for specified project: ${args[0]}") } - throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it") + dropList.each { manager -> + if( !manager.localPath.exists() ) { + throw new AbortOperationException("No match found for: ${manager.getProjectWithRevision()}") + } + + if( this.force || manager.isClean() ) { + manager.close() + if( !manager.localPath.deleteDir() ) + throw new AbortOperationException("Unable to delete project `${manager.getProjectWithRevision()}` -- Check access permissions for path: ${manager.localPath}") + return + } + + throw new AbortOperationException("Local project repository contains uncommitted changes -- won't drop it") + } } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy index c8c1c31bc0..428c914f9a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdInfo.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_DELIM + import java.lang.management.ManagementFactory import java.nio.file.spi.FileSystemProvider @@ -75,9 +77,16 @@ class CmdInfo extends CmdBase { } Plugins.init() - final manager = new AssetManager(args[0]) - if( !manager.isLocal() ) - throw new AbortOperationException("Unknown project `${args[0]}`") + def manager = new AssetManager(args[0], null) + if( !manager.isLocal() ) { + // if default branch not found locally, use first one from list of local pulls + if ( manager.listRevisions() ) { + manager = new AssetManager(args[0], manager.getPulledRevisions()[0]) + } + else { + throw new AbortOperationException("Unknown project `${args[0]}`") + } + } if( !format || format == 'text' ) { printText(manager,level) @@ -101,7 +110,7 @@ class CmdInfo extends CmdBase { out.println " project name: ${manager.project}" out.println " repository : ${manager.repositoryUrl}" - out.println " local path : ${manager.localPath}" + out.println " local path : ${manager.localPath.toString().tokenize(REVISION_DELIM)[0]}" out.println " main script : ${manager.mainScriptName}" if( manager.homePage && manager.homePage != manager.repositoryUrl ) out.println " home page : ${manager.homePage}" @@ -138,7 +147,7 @@ class CmdInfo extends CmdBase { def result = [:] result.projectName = manager.project result.repository = manager.repositoryUrl - result.localPath = manager.localPath?.toString() + result.localPath = manager.localPath?.toString().tokenize(REVISION_DELIM)[0] result.manifest = manager.manifest.toMap() result.revisions = manager.getBranchesAndTags(checkForUpdates) return result diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy index 5b08be249a..f6c851931a 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdList.groovy @@ -16,6 +16,9 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_DELIM + +import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic import groovy.util.logging.Slf4j @@ -33,6 +36,9 @@ class CmdList extends CmdBase { static final public NAME = 'list' + @Parameter(names=['-a','-all-revisions'], description = 'For each project, also list revisions') + Boolean revisions + @Override final String getName() { NAME } @@ -45,7 +51,15 @@ class CmdList extends CmdBase { return } - all.each { println it } + if (revisions) { + all.collect{ it.tokenize(REVISION_DELIM) } + .groupBy{ it[0] } + .each{ println ' ' + it.value[0][0] ; it.value.each{ y -> println ( y.size()==1 ? ' (default)' : ' ' + y[1] ) } } + } else { + all.collect{ it.replaceAll( /$REVISION_DELIM.*/, '' ) } + .unique() + .each{ println ' ' + it } + } } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy index 9d2fa9a01d..2e41e5f5f4 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdPull.groovy @@ -15,6 +15,9 @@ */ package nextflow.cli + +import static nextflow.scm.AssetManager.REVISION_DELIM + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -40,7 +43,7 @@ class CmdPull extends CmdBase implements HubOptions { @Parameter(names='-all', description = 'Update all downloaded projects', arity = 0) boolean all - @Parameter(names=['-r','-revision'], description = 'Revision of the project to run (either a git branch, tag or commit SHA number)') + @Parameter(names=['-r','-revision'], description = 'Revision of the project to pull (either a git branch, tag or commit SHA number)') String revision @Parameter(names=['-d','-deep'], description = 'Create a shallow clone of the specified depth') @@ -73,10 +76,10 @@ class CmdPull extends CmdBase implements HubOptions { Plugins.init() list.each { - log.info "Checking $it ..." - def manager = new AssetManager(it, this) + log.info "Checking $it${revision ? REVISION_DELIM + revision : ''} ..." + def manager = new AssetManager(it, revision, this) - def result = manager.download(revision,deep) + def result = manager.download(deep) manager.updateModules() def scriptFile = manager.getScriptFile() diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy index e5d18bdb87..7f94169f53 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdRun.groovy @@ -17,6 +17,7 @@ package nextflow.cli import static org.fusesource.jansi.Ansi.* +import static nextflow.scm.AssetManager.REVISION_DELIM import java.nio.file.NoSuchFileException import java.nio.file.Path @@ -569,22 +570,21 @@ class CmdRun extends CmdBase implements HubOptions { /* * try to look for a pipeline in the repository */ - def manager = new AssetManager(pipelineName, this) + def manager = new AssetManager(pipelineName, revision, this) def repo = manager.getProject() boolean checkForUpdate = true if( !manager.isRunnable() || latest ) { if( offline ) - throw new AbortOperationException("Unknown project `$repo` -- NOTE: automatic download from remote repositories is disabled") - log.info "Pulling $repo ..." - def result = manager.download(revision,deep) + throw new AbortOperationException("Unknown project `$repo${revision ? REVISION_DELIM + revision : ''}` -- NOTE: automatic download from remote repositories is disabled") + log.info "Pulling $repo${revision ? REVISION_DELIM + revision : ''} ..." + def result = manager.download(deep) if( result ) log.info " $result" checkForUpdate = false } - // checkout requested revision + // post download operations try { - manager.checkout(revision) manager.updateModules() final scriptFile = manager.getScriptFile(mainScript) if( checkForUpdate && !offline ) @@ -596,7 +596,7 @@ class CmdRun extends CmdBase implements HubOptions { throw e } catch( Exception e ) { - throw new AbortOperationException("Unknown error accessing project `$repo` -- Repository may be corrupted: ${manager.localPath}", e) + throw new AbortOperationException("Unknown error accessing project `$repo${revision ? REVISION_DELIM + revision : ''}` -- Repository may be corrupted: ${manager.localPath}", e) } } diff --git a/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy b/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy index dcdcfcae67..165be1f9b3 100644 --- a/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/cli/CmdView.groovy @@ -16,6 +16,8 @@ package nextflow.cli +import static nextflow.scm.AssetManager.REVISION_DELIM + import com.beust.jcommander.Parameter import com.beust.jcommander.Parameters import groovy.transform.CompileStatic @@ -42,6 +44,9 @@ class CmdView extends CmdBase { @Parameter(description = 'project name', required = true) List args = [] + @Parameter(names=['-r','-revision'], description = 'Revision of the project (either a git branch, tag or commit SHA number)') + String revision + @Parameter(names = '-q', description = 'Hide header line', arity = 0) boolean quiet @@ -51,9 +56,9 @@ class CmdView extends CmdBase { @Override void run() { Plugins.init() - def manager = new AssetManager(args[0]) + def manager = new AssetManager(args[0], revision) if( !manager.isLocal() ) - throw new AbortOperationException("Unknown project name `${args[0]}`") + throw new AbortOperationException("Unknown project `${args[0]}${revision ? REVISION_DELIM + revision : ''}`") if( all ) { if( !quiet ) diff --git a/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy b/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy index c50cef627d..4a2e5a83a8 100644 --- a/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/k8s/K8sDriverLauncher.groovy @@ -273,7 +273,7 @@ class K8sDriverLauncher { if( !interactive && !pipelineName.startsWith('/') && !cmd.remoteProfile && !cmd.runRemoteConfig ) { // -- check and parse project remote config Plugins.init() - final pipelineConfig = new AssetManager(pipelineName, cmd) .getConfigFile() + final pipelineConfig = new AssetManager(pipelineName, cmd.revision, cmd) .getConfigFile() builder.setUserConfigFiles(pipelineConfig) } diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy index dd304ad59e..85d7bb1b34 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/AssetManager.groovy @@ -62,6 +62,8 @@ class AssetManager { @PackageScope static File root = DEFAULT_ROOT + static public final String REVISION_DELIM = ':' + /** * The pipeline name. It must be in the form {@code username/repo} where 'username' * is a valid user name or organisation account, while 'repo' is the repository name @@ -69,8 +71,16 @@ class AssetManager { */ private String project + /** + * The name of the commit/branch/tag as requested via command line + * This is now a first class attribute of a pipeline + */ + private String revision + /** * Directory where the pipeline is cloned (i.e. downloaded) + * + * Schema: $NXF_ASSETS//[:] */ private File localPath @@ -94,37 +104,49 @@ class AssetManager { /** * Create a new asset manager with the specified pipeline name * - * @param pipeline The pipeline to be managed by this manager e.g. {@code nextflow-io/hello} + * @param pipelineName The pipeline to be managed by this manager e.g. {@code nextflow-io/hello} + * @param revision Revision ID for the selected pipeline (git branch, tag or commit SHA number) */ - AssetManager( String pipelineName, HubOptions cliOpts = null) { + AssetManager( String pipelineName, String revision = null, HubOptions cliOpts = null) { assert pipelineName // read the default config file (if available) def config = ProviderConfig.getDefault() // build the object - build(pipelineName, config, cliOpts) + build(pipelineName, revision, config, cliOpts) } - AssetManager( String pipelineName, Map config ) { + AssetManager( String pipelineName, String revision, Map config) { assert pipelineName // build the object - build(pipelineName, config) + build(pipelineName, revision, config) } /** * Build the asset manager internal data structure * * @param pipelineName A project name or a project repository Git URL + * @param revision Revision ID for the selected pipeline (git branch, tag or commit SHA number) * @param config A {@link Map} holding the configuration properties defined in the {@link ProviderConfig#DEFAULT_SCM_FILE} file * @param cliOpts User credentials provided on the command line. See {@link HubOptions} trait * @return The {@link AssetManager} object itself */ @PackageScope - AssetManager build( String pipelineName, Map config = null, HubOptions cliOpts = null ) { + AssetManager build( String pipelineName, String revision = null, Map config = null, HubOptions cliOpts = null ) { + + // if requested revision corresponds to the default branch, then unset it + // this avoids duplication of the default branch + if ( revision ) { + def referenceManager = new AssetManager(pipelineName, null, cliOpts) + if ( revision == referenceManager.getDefaultBranch() ) { + revision = null + } + } this.providerConfigs = ProviderConfig.createFromMap(config) - this.project = resolveName(pipelineName) - this.localPath = checkProjectDir(project) + this.revision = revision + this.project = resolveName(pipelineName, this.revision) + this.localPath = checkProjectDir(project, this.revision) this.hub = checkHubProvider(cliOpts) this.provider = createHubProvider(hub) setupCredentials(cliOpts) @@ -173,16 +195,17 @@ class AssetManager { * and return the directory where the project is stored locally * * @param projectName A project name matching the pattern {@code owner/project} + * @param revision Revision ID for the selected pipeline (git branch, tag or commit SHA number) * @return The project dir {@link File} */ @PackageScope - File checkProjectDir(String projectName) { + File checkProjectDir(String projectName, String revision) { if( !isValidProjectName(projectName)) { throw new IllegalArgumentException("Not a valid project name: $projectName") } - new File(root, project) + new File(root, project + (revision ? REVISION_DELIM + revision : '')) } /** @@ -243,7 +266,7 @@ class AssetManager { * @return The fully qualified project name e.g. {@code cbcrg/foo} */ @PackageScope - String resolveName( String name ) { + String resolveName( String name, String revision = null ) { assert name // @@ -285,7 +308,7 @@ class AssetManager { name = parts[0] } - def qualifiedName = find(name) + def qualifiedName = find(name, revision) if( !qualifiedName ) { return "$DEFAULT_ORGANIZATION/$name".toString() } @@ -300,6 +323,10 @@ class AssetManager { String getProject() { project } + String getRevision() { revision } + + String getProjectWithRevision() { project + ( revision ? REVISION_DELIM + revision : '' ) } + String getHub() { hub } @PackageScope @@ -363,7 +390,7 @@ class AssetManager { return this } - AssetManager checkValidRemoteRepo(String revision=null) { + AssetManager checkValidRemoteRepo() { // Configure the git provider to use the required revision as source for all needed remote resources: // - config if present in repo (nextflow.config by default) // - main script (main.nf by default) @@ -478,10 +505,11 @@ class AssetManager { } - String getBaseName() { + String getBaseNameWithRevision() { def result = project.tokenize('/') if( result.size() > 2 ) throw new IllegalArgumentException("Not a valid project name: $project") - return result.size()==1 ? result[0] : result[1] + result = ( result.size()==1 ? result[0] : result[1] ) + return result + ( revision ? REVISION_DELIM + revision : '' ) } boolean isLocal() { @@ -538,16 +566,48 @@ class AssetManager { return result } - static protected def find( String name ) { + /** + * @return The list of available revisions for a given project name + */ + List listRevisions( String projectName = project ) { + log.debug "Listing revisions for project: $projectName" + + def result = new LinkedList() + if( !root.exists() ) + return result + + list().each { + if( it.tokenize(REVISION_DELIM)[0] == projectName ) { + result << it + } + } + + return result + } + + // Updated for new localPath schema (see localPath declaration at top of this class file) + static protected def find( String name, String revision = null ) { def exact = [] def partial = [] list().each { def items = it.split('/') - if( items[1] == name ) - exact << it - else if( items[1].startsWith(name ) ) - partial << it + /** + * itemsRev[0] is the name of each list'ed project + * itemsRev[1] is the revision + */ + def itemsRev = items[1].tokenize(REVISION_DELIM) + // Check on matching revision: either null or same revision string + if( (!revision && !itemsRev[1]) || (revision && itemsRev[1] == revision) ) { + // Exact name match + if( itemsRev[0] == name ) + // Return item without revision + exact << it.tokenize(REVISION_DELIM)[0] + // Partial name match + else if( itemsRev[0].startsWith(name ) ) + // Return item without revision + partial << it.tokenize(REVISION_DELIM)[0] + } } def list = exact ?: partial @@ -565,19 +625,18 @@ class AssetManager { /** * Download a pipeline from a remote Github repository * - * @param revision The revision to download * @result A message representing the operation result */ - String download(String revision=null, Integer deep=null) { + String download(Integer deep=null) { assert project /* - * if the pipeline already exists locally pull it from the remote repo + * if the pipeline does not exists locally pull it from the remote repo */ if( !localPath.exists() ) { localPath.parentFile.mkdirs() // make sure it contains a valid repository - checkValidRemoteRepo(revision) + checkValidRemoteRepo() final cloneURL = getGitRepositoryUrl() log.debug "Pulling $project -- Using remote clone url: ${cloneURL}" @@ -599,14 +658,14 @@ class AssetManager { // use an explicit checkout command *after* the clone instead of cloning a specific branch // because the clone command does not allow the use of SHA commit id (only branch and tag names) try { git.checkout() .setName(revision) .call() } - catch ( RefNotFoundException e ) { checkoutRemoteBranch(revision) } + catch ( RefNotFoundException e ) { checkoutRemoteBranch() } } // return status message return "downloaded from ${cloneURL}" } - log.debug "Pull pipeline $project -- Using local path: $localPath" + log.debug "Pulling $project -- Using local path: $localPath" // verify that is clean if( !isClean() ) @@ -624,7 +683,7 @@ class AssetManager { * Try to checkout it from a remote branch and return */ catch ( RefNotFoundException e ) { - final ref = checkoutRemoteBranch(revision) + final ref = checkoutRemoteBranch() final commitId = ref?.getObjectId() return commitId ? "checked out at ${commitId.name()}" @@ -662,13 +721,12 @@ class AssetManager { * Clone a pipeline from a remote pipeline repository to the specified folder * * @param directory The folder when the pipeline will be cloned - * @param revision The revision to be cloned. It can be a branch, tag, or git revision number */ - void clone(File directory, String revision = null, Integer deep=null) { + void clone(File directory, Integer deep=null) { def clone = Git.cloneRepository() def uri = getGitRepositoryUrl() - log.debug "Clone project `$project` -- Using remote URI: ${uri} into: $directory" + log.debug "Cloning `${project}` from ${uri} into ${directory}" if( !uri ) throw new AbortOperationException("Cannot find the specified project: $project") @@ -706,6 +764,18 @@ class AssetManager { names.get( head.objectId ) ?: head.objectId.name() } + /** + * @return The names of all locally pulled revisions for a given project + * + * If revision is null, default is assumed + */ + List getPulledRevisions() { + return listRevisions().collect{ + it -> String y = it.tokenize(REVISION_DELIM)[1] + it = ( y != null ? y : getDefaultBranch() ) + } + } + RevisionInfo getCurrentRevisionAndName() { Ref head = git.getRepository().findRef(Constants.HEAD); if( !head ) @@ -733,30 +803,31 @@ class AssetManager { /** * @return A list of existing branches and tags names. For example * 
-     *     * master (default)
-     *       patch-x
-     *       v1.0 (t)
-     *       v1.1 (t)
+     *     * P master (default)
+     *         patch-x
+     *         v1.0 (t)
+     *         v1.1 (t)
      *
* - * The star character on the left highlight the current revision, the string {@code (default)} - * ticks that it is the default working branch (usually the master branch), while the string {@code (t)} - * shows that the revision is a git tag (instead of a branch) + * The character {@code P} on the left indicates the revision is pulled locally, + * the string {@code (default)} ticks that it is the default working branch, + * while the string {@code (t)} shows that the revision is a git tag (instead of a branch) */ @Deprecated List getRevisions(int level) { def current = getCurrentRevision() def master = getDefaultBranch() + def pulled = getPulledRevisions() List branches = getBranchList() .findAll { it.name.startsWith('refs/heads/') || it.name.startsWith('refs/remotes/origin/') } .unique { shortenRefName(it.name) } - .collect { Ref it -> refToString(it,current,master,false,level) } + .collect { Ref it -> refToString(it,current,master,pulled,false,level) } List tags = getTagList() .findAll { it.name.startsWith('refs/tags/') } - .collect { refToString(it,current,master,true,level) } + .collect { refToString(it,current,master,pulled,true,level) } def result = new ArrayList(branches.size() + tags.size()) result.addAll(branches) @@ -777,7 +848,6 @@ class AssetManager { Map getBranchesAndTags(boolean checkForUpdates) { final result = [:] - final current = getCurrentRevision() final master = getDefaultBranch() final branches = [] @@ -794,8 +864,8 @@ class AssetManager { .findAll { it.name.startsWith('refs/tags/') } .each { Ref it -> tags << refToMap(it,remote) } - result.current = current // current branch name result.master = master // master branch name + result.pulled = getPulledRevisions() // list of pulled revisions result.branches = branches // collection of branches result.tags = tags // collect of tags return result @@ -830,11 +900,11 @@ class AssetManager { return human ? obj.name.substring(0,10) : obj.name } - protected String refToString(Ref ref, String current, String master, boolean tag, int level ) { + protected String refToString(Ref ref, String current, String master, List pulled, boolean tag, int level ) { def result = new StringBuilder() def name = shortenRefName(ref.name) - result << (name == current ? '*' : ' ') + result << (name in pulled ? 'P' : ' ') if( level ) { def peel = git.getRepository().peel(ref) @@ -899,39 +969,7 @@ class AssetManager { return result } - /** - * Checkout a specific revision - * @param revision The revision to be checked out - */ - void checkout( String revision = null ) { - assert localPath - - def current = getCurrentRevision() - if( current != defaultBranch ) { - if( !revision ) { - throw new AbortOperationException("Project `$project` is currently stuck on revision: $current -- you need to explicitly specify a revision with the option `-r` in order to use it") - } - } - if( !revision || revision == current ) { - // nothing to do - return - } - - // verify that is clean - if( !isClean() ) - throw new AbortOperationException("Project `$project` contains uncommitted changes -- Cannot switch to revision: $revision") - - try { - git.checkout().setName(revision) .call() - } - catch( RefNotFoundException e ) { - checkoutRemoteBranch(revision) - } - - } - - - protected Ref checkoutRemoteBranch( String revision ) { + protected Ref checkoutRemoteBranch() { try { def fetch = git.fetch() @@ -997,7 +1035,7 @@ class AssetManager { if( provider.hasCredentials() ) update.setCredentialsProvider( provider.getGitCredentials() ) def updatedList = update.call() - log.debug "Update submodules $updatedList" + log.debug "Updating submodules $updatedList" } protected String getRemoteCommitId(RevisionInfo rev) { diff --git a/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy b/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy index 51a787d45c..1d0e7e55b6 100644 --- a/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/scm/GitlabRepositoryProvider.groovy @@ -18,6 +18,7 @@ package nextflow.scm import groovy.util.logging.Slf4j +import static nextflow.Const.DEFAULT_BRANCH /** * Implements a repository provider for GitHub service * @@ -59,8 +60,8 @@ class GitlabRepositoryProvider extends RepositoryProvider { String getDefaultBranch() { def result = invokeAndParseResponse(getEndpointUrl()) ?. default_branch if( !result ) { - log.debug "Unable to fetch repo default branch. Using `master` branch -- See https://gitlab.com/gitlab-com/support-forum/issues/1655#note_26132691" - return 'master' + log.debug "Unable to fetch repo default branch. Using `${DEFAULT_BRANCH}` branch -- See https://gitlab.com/gitlab-com/support-forum/issues/1655#note_26132691" + return DEFAULT_BRANCH } return result } diff --git a/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy b/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy index ab2be742bd..59a12a847f 100644 --- a/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/cli/CmdInfoTest.groovy @@ -45,8 +45,9 @@ class CmdInfoTest extends Specification { def setupSpec() { tempDir = Files.createTempDirectory('test') AssetManager.root = tempDir.toFile() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', revision, [providers: [github: [auth: token]]]) // download the project manager.download() } @@ -70,7 +71,7 @@ class CmdInfoTest extends Specification { screen.contains(" local path : $tempDir/nextflow-io/hello" ) screen.contains(" main script : main.nf") screen.contains(" revisions : ") - screen.contains(" * master (default)") + screen.contains(" P master (default)") } def 'should print json info' () { @@ -90,8 +91,9 @@ class CmdInfoTest extends Specification { json.localPath == "$tempDir/nextflow-io/hello" json.manifest.mainScript == 'main.nf' json.manifest.defaultBranch == 'master' - json.revisions.current == 'master' json.revisions.master == 'master' + json.revisions.pulled.size() == 1 + json.revisions.pulled.any { it == 'master' } json.revisions.branches.size()>1 json.revisions.branches.any { it.name == 'master' } json.revisions.tags.size()>1 @@ -116,8 +118,9 @@ class CmdInfoTest extends Specification { json.localPath == "$tempDir/nextflow-io/hello" json.manifest.mainScript == 'main.nf' json.manifest.defaultBranch == 'master' - json.revisions.current == 'master' json.revisions.master == 'master' + json.revisions.pulled.size() == 1 + json.revisions.pulled.any { it == 'master' } json.revisions.branches.size()>1 json.revisions.branches.any { it.name == 'master' } json.revisions.tags.size()>1 diff --git a/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy b/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy index aa8bf1467c..6531f84c21 100644 --- a/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/scm/AssetManagerTest.groovy @@ -103,6 +103,32 @@ class AssetManagerTest extends Specification { } + def testListRevisions() { + given: + def folder = tempDir.getRoot() + folder.resolve('cbcrg/pipe1').mkdirs() + folder.resolve('cbcrg/pipe2').mkdirs() + folder.resolve('cbcrg/pipe2:v2').mkdirs() + folder.resolve('cbcrg/pipe3:v3').mkdirs() + + def manager = new AssetManager() + + when: + def list = manager.listRevisions('cbcrg/pipe1') + then: + list == ['cbcrg/pipe1'] + + when: + list = manager.listRevisions('cbcrg/pipe3') + then: + list == ['cbcrg/pipe3:v3'] + + when: + list = manager.listRevisions('cbcrg/pipe2') + then: + list == ['cbcrg/pipe2', 'cbcrg/pipe2:v2'] + } + def testResolveName() { @@ -110,6 +136,7 @@ class AssetManagerTest extends Specification { def folder = tempDir.getRoot() folder.resolve('cbcrg/pipe1').mkdirs() folder.resolve('cbcrg/pipe2').mkdirs() + folder.resolve('cbcrg/pipe2:v2').mkdirs() folder.resolve('ncbi/blast').mkdirs() def manager = new AssetManager() @@ -119,6 +146,11 @@ class AssetManagerTest extends Specification { then: result == 'x/y' + when: + result = manager.resolveName('x/y', 'v2') + then: + result == 'x/y' + when: result = manager.resolveName('blast') then: @@ -144,6 +176,11 @@ class AssetManagerTest extends Specification { then: thrown(AbortOperationException) + when: + result = manager.resolveName('pipe2', 'v2') + then: + result == 'cbcrg/pipe2' + when: result = manager.resolveName('../blast/script.nf') then: @@ -162,8 +199,9 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', revision, [providers: [github: [auth: token]]]) when: manager.download() @@ -171,10 +209,10 @@ class AssetManagerTest extends Specification { folder.resolve('nextflow-io/hello/.git').isDirectory() when: - manager.download() + def result = manager.download() then: noExceptionThrown() - + result == "Already-up-to-date" } @@ -184,15 +222,15 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', "v1.2", [providers: [github: [auth: token]]]) when: - manager.download("v1.2") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello:v1.2/.git').isDirectory() when: - manager.download("v1.2") + manager.download() then: noExceptionThrown() } @@ -204,15 +242,15 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', "6b9515aba6c7efc6a9b3f273ce116fc0c224bf68", [providers: [github: [auth: token]]]) when: - manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello:6b9515aba6c7efc6a9b3f273ce116fc0c224bf68/.git').isDirectory() when: - def result = manager.download("6b9515aba6c7efc6a9b3f273ce116fc0c224bf68") + def result = manager.download() then: noExceptionThrown() result == "Already-up-to-date" @@ -227,40 +265,18 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', "mybranch", [providers: [github: [auth: token]]]) when: - manager.download("mybranch") - then: - folder.resolve('nextflow-io/hello/.git').isDirectory() - - when: - manager.download("mybranch") - then: - noExceptionThrown() - } - - // First clone a repo with a tag, then forget to include the -r argument - // when you execute nextflow. - // Note that while the download will work, execution will fail subsequently - // at a separate check - this just tests that we don't fail because of a detached head. - @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) - def testPullTagThenBranch() { - - given: - def folder = tempDir.getRoot() - def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers: [github: [auth: token]]]) - - when: - manager.download("v1.2") + manager.download() then: - folder.resolve('nextflow-io/hello/.git').isDirectory() + folder.resolve('nextflow-io/hello:mybranch/.git').isDirectory() when: - manager.download() + def result = manager.download() then: noExceptionThrown() + result == "Already-up-to-date" } @@ -269,8 +285,9 @@ class AssetManagerTest extends Specification { given: def dir = tempDir.getRoot() + String revision = null def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/hello', [providers:[github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/hello', revision, [providers:[github: [auth: token]]]) when: manager.clone(dir.toFile()) @@ -551,19 +568,14 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) - - when: - manager.download("dev") - then: - folder.resolve('nextflow-io/nf-test-branch/.git').isDirectory() - and: - folder.resolve('nextflow-io/nf-test-branch/workflow.nf').text == "println 'Hello'\n" + def manager = new AssetManager().build('nextflow-io/nf-test-branch', "dev", [providers: [github: [auth: token]]]) when: manager.download() then: - noExceptionThrown() + folder.resolve('nextflow-io/nf-test-branch:dev/.git').isDirectory() + and: + folder.resolve('nextflow-io/nf-test-branch:dev/workflow.nf').text == "println 'Hello'\n" } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) @@ -571,13 +583,12 @@ class AssetManagerTest extends Specification { given: def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) + def manager = new AssetManager().build('nextflow-io/nf-test-branch', 'dev', [providers: [github: [auth: token]]]) expect: - manager.checkValidRemoteRepo('dev') + manager.checkValidRemoteRepo() and: manager.getMainScriptName() == 'workflow.nf' - } @Requires({System.getenv('NXF_GITHUB_ACCESS_TOKEN')}) @@ -586,19 +597,14 @@ class AssetManagerTest extends Specification { given: def folder = tempDir.getRoot() def token = System.getenv('NXF_GITHUB_ACCESS_TOKEN') - def manager = new AssetManager().build('nextflow-io/nf-test-branch', [providers: [github: [auth: token]]]) - - when: - manager.download("v0.1") - then: - folder.resolve('nextflow-io/nf-test-branch/.git').isDirectory() - and: - folder.resolve('nextflow-io/nf-test-branch/workflow.nf').text == "println 'Hello'\n" + def manager = new AssetManager().build('nextflow-io/nf-test-branch', "v0.1", [providers: [github: [auth: token]]]) when: manager.download() then: - noExceptionThrown() + folder.resolve('nextflow-io/nf-test-branch:v0.1/.git').isDirectory() + and: + folder.resolve('nextflow-io/nf-test-branch:v0.1/workflow.nf').text == "println 'Hello'\n" } }