Skip to content

Commit fc4efdf

Browse files
authored
Merge pull request #49 from santisq/48-ability-to-import-modules-during-startup-of-powershell-runspace
Adds `-ModuleNames` and `-ModulePaths` parameters to `Invoke-Parallel`: - Introduce `-ModuleNames` to import system-installed modules via `$env:PSModulePath`. - Add `-ModulePaths` to load custom modules from specified directories. - Updates README and docs with examples and usage notes. - Simplify module integration, removing need for manual `Import-Module` in script blocks. - Add Pester tests for new parameters.
2 parents 5a5582b + 9fbb6d8 commit fc4efdf

File tree

14 files changed

+400
-91
lines changed

14 files changed

+400
-91
lines changed

README.md

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010

1111
</div>
1212

13-
`PSParallelPipeline` is a PowerShell module featuring the `Invoke-Parallel` cmdlet, designed to process pipeline input objects in parallel using multithreading. It mirrors the capabilities of [`ForEach-Object -Parallel`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object) from PowerShell 7.0+, bringing this functionality to Windows PowerShell 5.1, surpassing the constraints of [`Start-ThreadJob`](https://learn.microsoft.com/en-us/powershell/module/threadjob/start-threadjob?view=powershell-7.4).
13+
`PSParallelPipeline` is a PowerShell module featuring the `Invoke-Parallel` cmdlet, designed to process pipeline input objects in parallel using multithreading. It mirrors the capabilities of [`ForEach-Object -Parallel`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object) from PowerShell 7.0+, bringing this functionality to Windows PowerShell 5.1, surpassing the constraints of [`Start-ThreadJob`](https://learn.microsoft.com/en-us/powershell/module/threadjob/start-threadjob).
1414

1515
# Why Use This Module?
1616

@@ -33,7 +33,7 @@ Measure-Command {
3333

3434
## Common Parameters Support
3535

36-
Unlike `ForEach-Object -Parallel` (up to v7.5), `Invoke-Parallel` supports [Common Parameters](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_commonparameters?view=powershell-7.4), enhancing control and debugging.
36+
Unlike `ForEach-Object -Parallel` (up to v7.5), `Invoke-Parallel` supports [Common Parameters](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_commonparameters), enhancing control and debugging.
3737

3838
```powershell
3939
# Stops on first error
@@ -60,7 +60,7 @@ Get a single, friendly timeout message instead of multiple errors:
6060
# Invoke-Parallel: Timeout has been reached.
6161
```
6262

63-
## [`$using:`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#the-using-scope-modifier) Scope Support
63+
## [`$using:`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#the-using-scope-modifier) Scope Support
6464

6565
Easily pass variables into parallel scopes with the `$using:` modifier, just like `ForEach-Object -Parallel`:
6666

@@ -70,7 +70,7 @@ $message = 'world!'
7070
# hello world!
7171
```
7272

73-
## `-Variables` and `-Functions` Parameters
73+
## `-Variables`, `-Functions`, `-ModuleNames`, and `-ModulePaths` Parameters
7474

7575
- [`-Variables` Parameter](./docs/en-US/Invoke-Parallel.md#-variables): Pass variables directly to parallel runspaces.
7676

@@ -87,7 +87,22 @@ $message = 'world!'
8787
# hello world!
8888
```
8989
90-
Both parameters are quality-of-life enhancements, especially `-Functions`, which adds locally defined functions to the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate)—a feature absent in `ForEach-Object -Parallel`. This is a far better option than passing function definitions into the parallel scope.
90+
- [`-ModuleNames` Parameter](./docs/en-US/Invoke-Parallel.md#-modulenames): Import system-installed modules into parallel runspaces by name, using modules discoverable via `$env:PSModulePath`.
91+
92+
```powershell
93+
Import-Csv users.csv | Invoke-Parallel { Get-ADUser $_.UserPrincipalName } -ModuleNames ActiveDirectory
94+
# Imports ActiveDirectory module for Get-ADUser
95+
```
96+
97+
- [`-ModulePaths` Parameter](./docs/en-US/Invoke-Parallel.md#-modulepaths): Import custom modules from specified directory paths into parallel runspaces.
98+
99+
```powershell
100+
$moduleDir = Join-Path $PSScriptRoot "CustomModule"
101+
0..10 | Invoke-Parallel { Get-CustomCmdlet } -ModulePaths $moduleDir
102+
# Imports custom module for Get-CustomCmdlet
103+
```
104+
105+
These parameters are a quality-of-life enhancement, especially `-Functions`, which incorporates locally defined functions to the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate)—a feature absent in `ForEach-Object -Parallel` and a far better option than passing function definitions into the parallel scope. The new `-ModuleNames` and `-ModulePaths` parameters simplify module integration by automatically loading system-installed and custom modules, respectively, eliminating the need for manual `Import-Module` calls within the script block.
91106
92107
# Documentation
93108

docs/en-US/Invoke-Parallel.md

Lines changed: 116 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -15,19 +15,24 @@ Executes parallel processing of pipeline input objects using multithreading.
1515

1616
```powershell
1717
Invoke-Parallel
18-
-InputObject <Object>
1918
[-ScriptBlock] <ScriptBlock>
19+
[-InputObject <Object>]
2020
[-ThrottleLimit <Int32>]
21+
[-TimeoutSeconds <Int32>]
2122
[-Variables <Hashtable>]
2223
[-Functions <String[]>]
24+
[-ModuleNames <String[]>]
25+
[-ModulePaths <String[]>]
2326
[-UseNewRunspace]
24-
[-TimeoutSeconds <Int32>]
2527
[<CommonParameters>]
2628
```
2729

2830
## DESCRIPTION
2931

30-
The `Invoke-Parallel` cmdlet enables parallel processing of input objects in PowerShell, including __Windows PowerShell 5.1__, offering functionality similar to `ForEach-Object -Parallel` introduced in PowerShell 7.0. It processes pipeline input across multiple threads, improving performance for tasks that benefit from parallel execution.
32+
The `Invoke-Parallel` cmdlet enables parallel processing of input objects in PowerShell, including
33+
__Windows PowerShell 5.1__ and PowerShell 7+, offering functionality similar to `ForEach-Object -Parallel` introduced in
34+
PowerShell 7.0. It processes pipeline input across multiple threads, improving performance for tasks that benefit from
35+
parallel execution.
3136

3237
## EXAMPLES
3338

@@ -42,7 +47,10 @@ $message = 'Hello world from '
4247
}
4348
```
4449

45-
This example demonstrates parallel execution of a script block with a 3-second delay, appending a unique runspace ID to a message. The [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#the-using-scope-modifier) is used to pass the local variable `$message` into the parallel scope, a supported method for accessing external variables in `Invoke-Parallel`.
50+
This example demonstrates parallel execution of a script block with a 3-second delay, appending a unique runspace ID to
51+
a message. The [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#the-using-scope-modifier)
52+
is used to pass the local variable `$message` into the parallel scope, a supported method for accessing external
53+
variables in `Invoke-Parallel`.
4654

4755
### Example 2: Demonstrates `-Variables` Parameter
4856

@@ -55,7 +63,9 @@ $message = 'Hello world from '
5563
} -Variables @{ message = $message }
5664
```
5765

58-
This example demonstrates the [`-Variables` parameter](#-variables), which passes the local variable `$message` into the parallel scope using a hashtable. The key `message` in the hashtable defines the variable name available within the script block, serving as an alternative to the `$using:` scope modifier.
66+
This example demonstrates the [`-Variables` parameter](#-variables), which passes the local variable `$message` into
67+
the parallel scope using a hashtable. The key `message` in the hashtable defines the variable name available within the
68+
script block, serving as an alternative to the `$using:` scope modifier.
5969

6070
### Example 3: Adding to a thread-safe collection with `$using:`
6171

@@ -65,7 +75,8 @@ Get-Process | Invoke-Parallel { ($using:dict)[$_.Id] = $_ }
6575
$dict[$PID]
6676
```
6777

68-
This example uses a thread-safe dictionary to store process objects by ID, leveraging the `$using:` modifier for variable access.
78+
This example uses a thread-safe dictionary to store process objects by ID, leveraging the `$using:` modifier for
79+
variable access.
6980

7081
### Example 4: Adding to a thread-safe collection with `-Variables`
7182

@@ -85,15 +96,17 @@ function Greet { param($s) "$s hey there!" }
8596
0..10 | Invoke-Parallel { Greet $_ } -Functions Greet
8697
```
8798

88-
This example imports a local function `Greet` into the parallel scope using [`-Functions` parameter](#-functions), allowing its use within the script block.
99+
This example imports a local function `Greet` into the parallel scope using [`-Functions` parameter](#-functions),
100+
allowing its use within the script block.
89101

90102
### Example 6: Setting a timeout with `-TimeoutSeconds`
91103

92104
```powershell
93105
0..10 | Invoke-Parallel { Start-Sleep 1 } -TimeoutSeconds 3
94106
```
95107

96-
This example limits execution to 3 seconds, stopping all running script blocks and ignoring unprocessed input once the timeout is reached.
108+
This example limits execution to 3 seconds, stopping all running script blocks and ignoring unprocessed input once the
109+
timeout is reached.
97110

98111
### Example 7: Creating new runspaces with `-UseNewRunspace`
99112

@@ -117,17 +130,45 @@ This example limits execution to 3 seconds, stopping all running script blocks a
117130
# 9af7c222-061d-4c89-b073-375ee925e538
118131
```
119132

120-
This example contrasts default runspace reuse with the `-UseNewRunspace` switch, showing unique runspace IDs for each invocation in the latter case.
133+
This example contrasts default runspace reuse with the `-UseNewRunspace` switch, showing unique runspace IDs for each
134+
invocation in the latter case.
135+
136+
### Example 8: Using the `-ModuleNames` parameter
137+
138+
```powershell
139+
Import-Csv users.csv | Invoke-Parallel { Get-ADUser $_.UserPrincipalName } -ModuleNames ActiveDirectory
140+
```
141+
142+
This example imports the `ActiveDirectory` module into the parallel scope using `-ModuleNames`, enabling the
143+
`Get-ADUser` cmdlet within the script block.
144+
145+
### Example 9: Using the `-ModulePaths` parameter
146+
147+
```powershell
148+
$moduleDir = Join-Path $PSScriptRoot "CustomModule"
149+
0..10 | Invoke-Parallel { Get-CustomCmdlet } -ModulePaths $moduleDir
150+
```
151+
152+
This example imports a custom module from the specified directory using `-ModulePaths`, allowing the `Get-CustomCmdlet`
153+
function to be used in the parallel script block.
154+
155+
> [!NOTE]
156+
>
157+
> The path must point to a directory containing a valid PowerShell module.
121158
122159
## PARAMETERS
123160

124161
### -Functions
125162

126-
Specifies an array of function names from the local session to include in the runspaces' [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate). This enables their use within the parallel script block.
163+
Specifies an array of function names from the local session to include in the runspaces’
164+
[Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate).
165+
This enables their use within the parallel script block.
127166

128167
> [!TIP]
129168
>
130-
> This parameter is the recommended way to make local functions available in the parallel scope. Alternatively, you can retrieve the function definition as a string (e.g., `$def = ${function:Greet}.ToString()`) and use `$using:` to pass it into the script block, defining it there (e.g., `${function:Greet} = $using:def`).
169+
> This parameter is the recommended way to make local functions available in the parallel scope.
170+
Alternatively, you can retrieve the function definition as a string (e.g., `$def = ${function:Greet}.ToString()`) and
171+
use `$using:` to pass it into the script block, defining it there (e.g., `${function:Greet} = $using:def`).
131172

132173
```yaml
133174
Type: String[]
@@ -175,11 +216,8 @@ Accept wildcard characters: False
175216
176217
### -ThrottleLimit
177218
178-
Sets the maximum number of script blocks executed in parallel across multiple threads. Additional input objects wait until the number of running script blocks falls below this limit.
179-
180-
> [!NOTE]
181-
>
182-
> The default value is `5`.
219+
Sets the maximum number of script blocks executed in parallel across multiple threads. Additional input objects wait
220+
until the number of running script blocks falls below this limit. The default value is `5`.
183221

184222
```yaml
185223
Type: Int32
@@ -195,7 +233,8 @@ Accept wildcard characters: False
195233

196234
### -TimeoutSeconds
197235

198-
Specifies the maximum time (in seconds) to process all input objects. When the timeout is reached, running script blocks are terminated, and remaining input is discarded.
236+
Specifies the maximum time (in seconds) to process all input objects. When the timeout is reached, running script blocks
237+
are terminated, and remaining input is discarded.
199238

200239
> [!NOTE]
201240
>
@@ -231,11 +270,13 @@ Accept wildcard characters: False
231270

232271
### -Variables
233272

234-
Provides a hashtable of variables to make available in the parallel scope. Keys define the variable names within the script block.
273+
Provides a hashtable of variables to make available in the parallel scope. Keys define the variable names within the
274+
script block.
235275

236276
> [!TIP]
237277
>
238-
> Use this parameter as an alternative to the [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#scope-modifiers).
278+
> Use this parameter as an alternative to the
279+
[`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#scope-modifiers).
239280

240281
```yaml
241282
Type: Hashtable
@@ -249,6 +290,54 @@ Accept pipeline input: False
249290
Accept wildcard characters: False
250291
```
251292

293+
### -ModuleNames
294+
295+
Specifies an array of module names to import into the runspaces’
296+
[Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate).
297+
This allows the script block to use cmdlets and functions from the specified modules.
298+
299+
> [!TIP]
300+
>
301+
> Use this parameter to ensure required modules are available in the parallel scope. Module names must be discoverable
302+
via the [`$env:PSModulePath`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_psmodulepath)
303+
environment variable, which lists installed module locations.
304+
305+
```yaml
306+
Type: String[]
307+
Parameter Sets: (All)
308+
Aliases: mn
309+
310+
Required: False
311+
Position: Named
312+
Default value: None
313+
Accept pipeline input: False
314+
Accept wildcard characters: False
315+
```
316+
317+
### -ModulePaths
318+
319+
Specifies an array of file paths to directories containing PowerShell modules (e.g., `.psm1` or `.psd1` files) to import
320+
into the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate).
321+
This enables the script block to use cmdlets and functions from custom or local modules.
322+
323+
> [!NOTE]
324+
>
325+
> Paths must be absolute or relative to the current working directory and must point to valid directories containing
326+
PowerShell modules. If an invalid path (e.g., a file or non-existent directory) is provided, a terminating error is
327+
thrown.
328+
329+
```yaml
330+
Type: String[]
331+
Parameter Sets: (All)
332+
Aliases: mp
333+
334+
Required: False
335+
Position: Named
336+
Default value: None
337+
Accept pipeline input: False
338+
Accept wildcard characters: False
339+
```
340+
252341
### CommonParameters
253342

254343
This cmdlet supports the common parameters. For more information, see [about_CommonParameters](http://go.microsoft.com/fwlink/?LinkID=113216).
@@ -268,14 +357,19 @@ Returns objects produced by the script block.
268357
## NOTES
269358

270359
- `Invoke-Parallel` uses multithreading, which may introduce overhead. For small datasets, sequential processing might be faster.
271-
- Ensure variables or collections passed to the parallel scope are thread-safe (e.g., `[System.Collections.Concurrent.ConcurrentDictionary]`), as shown in Examples 3 and 4.
272-
- By default, runspaces are reused from a pool to optimize resource usage. Using `-UseNewRunspace` increases memory and startup time but ensures isolation.
360+
- Ensure variables or collections passed to the parallel scope are thread-safe (e.g., use
361+
[`ConcurrentDictionary<TKey, TValue>`](https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2) or similar), as shown in
362+
[__Example #3__](#example-3-adding-to-a-thread-safe-collection-with-using) and
363+
[__Example #4__](#example-4-adding-to-a-thread-safe-collection-with--variables),
364+
to avoid race conditions.
365+
- By default, runspaces are reused from a pool to optimize resource usage. Using [`-UseNewRunspace`](#-usenewrunspace) increases memory and
366+
startup time but ensures isolation.
273367

274368
## RELATED LINKS
275369

276370
[__ForEach-Object -Parallel__](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object)
277371

278-
[__Runspaces Overview__](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.runspace?view=powershellsdk-7.4.0)
372+
[__Runspaces Overview__](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.runspace)
279373

280374
[__Managed threading best practices__](https://learn.microsoft.com/en-us/dotnet/standard/threading/managed-threading-best-practices)
281375

module/PSParallelPipeline.psd1

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
RootModule = 'bin/netstandard2.0/PSParallelPipeline.dll'
1212

1313
# Version number of this module.
14-
ModuleVersion = '1.2.3'
14+
ModuleVersion = '1.2.4'
1515

1616
# Supported PSEditions
1717
# CompatiblePSEditions = @()

src/PSParallelPipeline/Commands/InvokeParallelCommand.cs

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,17 @@ public sealed class InvokeParallelCommand : PSCmdlet, IDisposable
4343
[Alias("funcs")]
4444
public string[]? Functions { get; set; }
4545

46+
[Parameter]
47+
[ValidateNotNullOrEmpty]
48+
[ArgumentCompleter(typeof(ModuleCompleter))]
49+
[Alias("mn")]
50+
public string[]? ModuleNames { get; set; }
51+
52+
[Parameter]
53+
[ValidateNotNullOrEmpty]
54+
[Alias("mp")]
55+
public string[]? ModulePaths { get; set; }
56+
4657
[Parameter]
4758
[Alias("unr")]
4859
public SwitchParameter UseNewRunspace { get; set; }
@@ -57,7 +68,9 @@ protected override void BeginProcessing()
5768
InitialSessionState iss = InitialSessionState
5869
.CreateDefault2()
5970
.AddFunctions(Functions, this)
60-
.AddVariables(Variables, this);
71+
.AddVariables(Variables, this)
72+
.ImportModules(ModuleNames)
73+
.ImportModulesFromPath(ModulePaths, this);
6174

6275
PoolSettings poolSettings = new(
6376
ThrottleLimit, UseNewRunspace, iss);
@@ -90,7 +103,7 @@ protected override void ProcessRecord()
90103
}
91104
catch (OperationCanceledException exception)
92105
{
93-
_worker.WaitForCompletion();
106+
CancelAndWait();
94107
exception.WriteTimeoutError(this);
95108
}
96109
}
@@ -116,7 +129,7 @@ protected override void EndProcessing()
116129
}
117130
catch (OperationCanceledException exception)
118131
{
119-
_worker.WaitForCompletion();
132+
CancelAndWait();
120133
exception.WriteTimeoutError(this);
121134
}
122135
}

0 commit comments

Comments
 (0)