Skip to content

Commit 3a89745

Browse files
edmundmillerEdmund Millermaxulyssemashehu
authored
Use new nf-test features (#6286)
* ci: Attempt to split everything out Don't hate me Adam * ci: Add changed since, sharding, and ci * ci: Add filter to try to get jobs split up * ci: Switch to only-changed * ci: See if follow-dependencies works without "related-tests" * ci: Remove skipped tests Idk what we're going to do about these... * ci: Actually use the nf-test version * ci: module => process * ci: Clean up job names * dummy: Make a change * ci: Skip test.tap * ci: Add fetch-depth * ci: Clean up name * ci: Lint everything * ci: Get the job names clean * ci: Add hide-progress on linting * ci: Add psuedocode for conda-fail.yml https://nfcore.slack.com/archives/CJRH30T6V/p1724406283145319 * test: Snapshot the versions contents, not the hash #6505 * ci: Keep running nf-core lint the way it was Blocked by nf-core/tools#3140 * ci: Move conda skips out * ci: Address a comment * style: Move prettier and editorconfig to pre-commit #4554 (comment) * ci: Add note about nf-core lint pre-commit * chore: Copy over conda skips a004c86 2da71b7 * ci: only-changed => changed-since * Add confirm-pass * ci: dynamically set shards * ci: Run 3 process jobs per CI run as an example * install nf-test * fix install nf-test * fix install nf-test * ci: Remove dynamic number of shards, and combine process and workflow filters * ci: Remove variable Nextflow versions * test: Update bowtie versions * ci: Fix indention * ci: We're not testing multiple python versions * style: Clean up job names * build: Remove invisible characters? * test(bowtie): Bump snapshots * style: Remove stray comment * ci: Add GPU tests in their own workflows These will only get trigger on the paths listed * ci(gpu): Clean up triggers and make a note about how to make the includes smarter * ci: pip install cryptography * ci: Try removing path * ci: Add fetch-depth and remove duplicate checkout * test: Remove the second gpu profile * ci: Add path back in * ci(gpu): Remove changed-since * ci: Fix missed dorny/paths-filter update Co-authored-by: mashehu <[email protected]> * Add changes from codereview Co-authored-by: mashehu <[email protected]> * style: Remove example TODO comments https://github.com/nf-core/modules/pull/6286/files#r1846856369 Co-authored-by: mashehu <[email protected]> * refactor: Try tags for GPU CI askimed/nf-test#272 * style: Set NFT_WORKDIR as an ENV variable * style: Run lsp format on a parabricks module For science * test: Add gpu tags to parabricks * ci: Split tags out of matrix * Revert "style: Run lsp format on a parabricks module" This reverts commit d134e78. * ci: Fix name --------- Co-authored-by: Edmund Miller <[email protected]> Co-authored-by: maxulysse <[email protected]> Co-authored-by: mashehu <[email protected]>
1 parent af84433 commit 3a89745

File tree

17 files changed

+916
-792
lines changed

17 files changed

+916
-792
lines changed

‎.github/actions/nf-test-action/action.yml‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,22 @@
11
name: "nf-test Action"
22
description: "Runs nf-test with common setup steps"
33
inputs:
4-
path:
5-
description: "Path to test"
6-
required: true
74
profile:
85
description: "Profile to use"
96
required: true
7+
shard:
8+
description: "Shard number for this CI job"
9+
required: true
10+
total_shards:
11+
description: "Total number of test shards(NOT the total number of matrix jobs)"
12+
required: true
13+
tags:
14+
description: "Tags to test (`<tags>[,<tags>...]`)"
15+
required: false
1016

1117
runs:
1218
using: "composite"
1319
steps:
14-
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
15-
1620
- uses: actions/setup-java@8df1039502a15bceb9433410b1a100fbe190c53b # v4
1721
with:
1822
distribution: "temurin"
@@ -56,20 +60,31 @@ runs:
5660
if: env.SENTIEON_ENCRYPTION_KEY != '' && env.SENTIEON_LICENSE_MESSAGE != ''
5761
shell: bash
5862
run: |
63+
python -m pip install cryptography
5964
nextflow secrets set SENTIEON_AUTH_DATA $(python3 tests/modules/nf-core/sentieon/license_message.py encrypt --key "$SENTIEON_ENCRYPTION_KEY" --message "$SENTIEON_LICENSE_MESSAGE")
6065
66+
# TODO Skip failing conda tests and document their failures
67+
# https://github.com/nf-core/modules/issues/7017
6168
- name: Run nf-test
6269
shell: bash
6370
env:
6471
SENTIEON_LICSRVR_IP: ${{ env.SENTIEON_LICSRVR_IP }}
6572
SENTIEON_AUTH_MECH: "GitHub Actions - token"
73+
TAGS: ${{ inputs.tags && format('--tag {0}', inputs.tags) || '' }}
74+
NFT_WORKDIR: "~"
6675
run: |
67-
NFT_WORKDIR=~ \
6876
nf-test test \
6977
--profile=${{ inputs.profile }} \
7078
--tap=test.tap \
7179
--verbose \
72-
${{ inputs.path }}
80+
--ci \
81+
--changed-since HEAD^ \
82+
--shard ${{ inputs.shard }}/${{ inputs.total_shards }} \
83+
--filter process,workflow \
84+
--follow-dependencies \
85+
${{ env.TAGS }}
86+
87+
# TODO If no test.tap, then make one to spoof?
7388
- uses: pcolby/tap-summary@0959cbe1d4422e62afc65778cdaea6716c41d936 # v1
7489
if: ${{ inputs.path != '' }}
7590
with:

‎.github/conda_skip.yml‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# TODO What do we do with these?
2+
# I think we can require Docker,modules,Shard 1..Singularity,subworkflows,3 and skip the condas now
3+
exclude:
4+
- path: modules/nf-core/nf-test
5+
- profile: conda
6+
path: modules/nf-core/angsd/gl
7+
- profile: conda
8+
path: modules/nf-core/annotsv/installannotations
9+
- profile: conda
10+
path: modules/nf-core/happy/sompy
11+
- profile: conda
12+
path: modules/nf-core/backsub
13+
- profile: conda
14+
path: modules/nf-core/bakta/bakta
15+
- profile: conda
16+
path: modules/nf-core/bakta/baktadbdownload
17+
- profile: conda
18+
path: modules/nf-core/bases2fastq
19+
- profile: conda
20+
path: modules/nf-core/bcl2fastq
21+
- profile: conda
22+
path: modules/nf-core/bclconvert
23+
- profile: conda
24+
path: modules/nf-core/celesta
25+
- profile: conda
26+
path: modules/nf-core/cellpose
27+
- profile: conda
28+
path: modules/nf-core/cellranger/count
29+
- profile: conda
30+
path: modules/nf-core/cellranger/mkfastq
31+
- profile: conda
32+
path: modules/nf-core/cellranger/mkgtf
33+
- profile: conda
34+
path: modules/nf-core/cellranger/mkref
35+
- profile: conda
36+
path: modules/nf-core/cellranger/mkvdjref
37+
- profile: conda
38+
path: modules/nf-core/cellranger/multi
39+
- profile: conda
40+
path: modules/nf-core/cellranger/vdj
41+
- profile: conda
42+
path: modules/nf-core/checkqc
43+
- profile: conda
44+
path: modules/nf-core/custom/dumpsoftwareversions
45+
- profile: conda
46+
path: modules/nf-core/deepcell/mesmer
47+
- profile: conda
48+
path: modules/nf-core/deepsomatic
49+
- profile: singularity
50+
path: modules/nf-core/deepsomatic
51+
- profile: conda
52+
path: modules/nf-core/deepvariant
53+
- profile: conda
54+
path: modules/nf-core/deepvariant/callvariants
55+
- profile: conda
56+
path: modules/nf-core/deepvariant/makeexamples
57+
- profile: conda
58+
path: modules/nf-core/deepvariant/postprocessvariants
59+
- profile: conda
60+
path: modules/nf-core/deepvariant/rundeepvariant
61+
- profile: conda
62+
path: modules/nf-core/deepvariant/vcfstatsreport
63+
- profile: conda
64+
path: modules/nf-core/doubletdetection
65+
- profile: conda
66+
path: modules/nf-core/ensemblvep/vep
67+
- profile: conda
68+
path: modules/nf-core/fastk/fastk
69+
- profile: conda
70+
path: modules/nf-core/cellrangerarc/mkgtf
71+
- profile: conda
72+
path: modules/nf-core/fastk/histex
73+
- profile: conda
74+
path: modules/nf-core/fastk/merge
75+
- profile: conda
76+
path: modules/nf-core/fcs/fcsadaptor
77+
- profile: conda
78+
path: modules/nf-core/fcs/fcsgx
79+
- profile: conda
80+
path: modules/nf-core/ganon/buildcustom
81+
- profile: conda
82+
path: modules/nf-core/ganon/classify
83+
- profile: conda
84+
path: modules/nf-core/ganon/report
85+
- profile: conda
86+
path: modules/nf-core/ganon/table
87+
- profile: conda
88+
path: modules/nf-core/gatk4/cnnscorevariants
89+
- profile: conda
90+
path: modules/nf-core/gatk4/determinegermlinecontigploidy
91+
- profile: conda
92+
path: modules/nf-core/genescopefk
93+
- profile: conda
94+
path: modules/nf-core/ilastik/multicut
95+
- profile: conda
96+
path: modules/nf-core/ilastik/pixelclassification
97+
- profile: conda
98+
path: modules/nf-core/imputeme/vcftoprs
99+
- profile: conda
100+
path: modules/nf-core/mcstaging/imc2mc
101+
- profile: conda
102+
path: modules/nf-core/mcquant
103+
- profile: conda
104+
path: modules/nf-core/mcstaging/phenoimager2mc
105+
- profile: conda
106+
path: modules/nf-core/merquryfk/katcomp
107+
- profile: conda
108+
path: modules/nf-core/merquryfk/katgc
109+
- profile: conda
110+
path: modules/nf-core/merquryfk/merquryfk
111+
- profile: conda
112+
path: modules/nf-core/merquryfk/ploidyplot
113+
- profile: conda
114+
path: modules/nf-core/molkartgarage/clahe
115+
- profile: conda
116+
path: modules/nf-core/quartonotebook
117+
- profile: conda
118+
path: modules/nf-core/scimap/spatiallda
119+
- profile: conda
120+
path: modules/nf-core/sentieon/bwaindex
121+
- profile: conda
122+
path: modules/nf-core/sentieon/bwamem
123+
- profile: conda
124+
path: modules/nf-core/sentieon/datametrics
125+
- profile: conda
126+
path: modules/nf-core/sentieon/dedup
127+
- profile: conda
128+
path: modules/nf-core/sentieon/qualcal
129+
- profile: conda
130+
path: modules/nf-core/spaceranger/mkgtf
131+
- profile: conda
132+
path: modules/nf-core/spaceranger/mkref
133+
- profile: conda
134+
path: modules/nf-core/spaceranger/count
135+
- profile: conda
136+
path: modules/nf-core/spotiflow
137+
- profile: conda
138+
path: modules/nf-core/svanalyzer/svbenchmark
139+
- profile: conda
140+
path: modules/nf-core/universc
141+
- profile: singularity
142+
path: modules/nf-core/universc
143+
- profile: conda
144+
path: modules/nf-core/vt/decompose
145+
- profile: singularity
146+
path: modules/nf-core/bases2fastq
147+
- profile: conda
148+
path: modules/nf-core/wittyer
149+
- profile: conda
150+
path: modules/nf-core/islandpath
151+
- profile: conda
152+
path: modules/nf-core/scimap/mcmicro
153+
- profile: conda
154+
path: modules/nf-core/parabricks/fq2bammeth
155+
- profile: docker_self_hosted
156+
path: modules/nf-core/parabricks/fq2bammeth
157+
- profile: singularity
158+
path: modules/nf-core/parabricks/fq2bammeth
159+
- profile: conda
160+
path: modules/nf-core/parabricks/fq2bam
161+
- profile: docker_self_hosted
162+
path: modules/nf-core/parabricks/fq2bam
163+
- profile: singularity
164+
path: modules/nf-core/parabricks/fq2bam
165+
- profile: conda
166+
path: subworkflows/nf-core/vcf_annotate_ensemblvep
167+
- profile: conda
168+
path: subworkflows/nf-core/bcl_demultiplex
169+
- profile: conda
170+
path: subworkflows/nf-core/deepvariant
171+
- profile: conda
172+
path: subworkflows/nf-core/fastq_align_bamcmp_bwa
173+
- profile: conda
174+
path: subworkflows/nf-core/fastq_align_bwa
175+
- profile: conda
176+
path: subworkflows/nf-core/fasta_newick_epang_gappa
177+
- profile: conda
178+
path: modules/nf-core/xeniumranger/relabel
179+
- profile: conda
180+
path: modules/nf-core/xeniumranger/rename
181+
- profile: conda
182+
path: modules/nf-core/xeniumranger/resegment
183+
- profile: conda
184+
path: modules/nf-core/xeniumranger/import-segmentation

‎.github/workflows/gpu-tests.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: Run GPU nf-tests
2+
on:
3+
push:
4+
branches:
5+
# https://docs.renovatebot.com/key-concepts/automerge/#branch-vs-pr-automerging
6+
- "renovate/**" # branches Renovate creates
7+
pull_request:
8+
branches: [master]
9+
paths:
10+
- ".github/workflows/gpu-tests.yml"
11+
- "modules/nf-core/parabricks/**"
12+
merge_group:
13+
types: [checks_requested]
14+
branches: [master]
15+
workflow_dispatch:
16+
inputs:
17+
runners:
18+
description: "Runners to test on"
19+
type: choice
20+
options:
21+
- "ubuntu-latest"
22+
- "self-hosted"
23+
default: "self-hosted"
24+
25+
# Cancel if a newer run is started
26+
concurrency:
27+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
28+
cancel-in-progress: true
29+
30+
env:
31+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
32+
NFT_VER: "0.9.2"
33+
NXF_ANSI_LOG: false
34+
NXF_SINGULARITY_CACHEDIR: ${{ github.workspace }}/.singularity
35+
NXF_SINGULARITY_LIBRARYDIR: ${{ github.workspace }}/.singularity
36+
NXF_VER: "24.04.4"
37+
38+
jobs:
39+
nf-test-gpu:
40+
runs-on: "gpu"
41+
name: "GPU | ${{ matrix.tags}} | ${{ matrix.profile }} | ${{ matrix.shard }}"
42+
strategy:
43+
fail-fast: false
44+
matrix:
45+
shard: [1, 2]
46+
profile: [docker_self_hosted, singularity] # conda?
47+
# TODO Pass these in from GitHub PR trigger events
48+
tags:
49+
- parabricks/applybqsr
50+
- parabricks/fq2bam
51+
- parabricks/fq2bammeth
52+
env:
53+
NXF_ANSI_LOG: false
54+
TOTAL_SHARDS: 2
55+
56+
steps:
57+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
58+
with:
59+
fetch-depth: 0
60+
- name: Run nf-test Action
61+
uses: ./.github/actions/nf-test-action
62+
env:
63+
SENTIEON_ENCRYPTION_KEY: ${{ secrets.SENTIEON_ENCRYPTION_KEY }}
64+
SENTIEON_LICENSE_MESSAGE: ${{ secrets.SENTIEON_LICENSE_MESSAGE }}
65+
SENTIEON_LICSRVR_IP: ${{ secrets.SENTIEON_LICSRVR_IP }}
66+
SENTIEON_AUTH_MECH: "GitHub Actions - token"
67+
with:
68+
profile: ${{ matrix.profile }},gpu
69+
shard: ${{ matrix.shard }}
70+
total_shards: ${{ env.TOTAL_SHARDS }}
71+
tags: ${{matrix.tags}},gpu

0 commit comments

Comments
 (0)