fix: fixed race warnings #311

cx-rogerio-dalot · 2025-07-24T14:45:43Z

Closes #

Proposed Changes

Removed concurrency native primitives from global variables
Moved the pipeline to run the scan from cmd/ to the engine itself so it's easier to orchestrate and design the pipeline of the scan.
Fixed race warnings and made the pipeline of the scan truly concurrent.

Checklist

I covered my changes with tests.
I Updated the documentation that is affected by my changes:
- Change in the CLI arguments
- Change in the configuration file

I submit this contribution under the Apache-2.0 license.

* fix: Improve reporting service for different outputs * refactor: applied code review suggestions * refactor: applied code review suggestions * refactor: applied code review suggestions * Update cmd/main.go * ci: windows support fix * ci: revert fix attempt --------- Co-authored-by: Jossef Harush Kadouri <[email protected]>

* Refactor Confluence Plugin * Apply PR suggestions

closes #19

* ignore .vscode * feat(confluence): filter spaces * fix(confluence): use username and token * chore: move httpRequest to lib

@baruchiro

This PR if replacing #46. @baruchiro made a PR with a better mechanism to search for only specified spaces. Now on this PR I'm just fixing a small issue with that implementation that is to replace the --confluence-spaces argument from string to stringArray. This will fix a issue we got when no spaces selected.

closes #34 - create docker image - create makefile - validate docker on pr-validation - upload tar image to Github release - run Kics on pr-validation - fix Kics issues

Implement suggested concurrency logic Also added confluence window constant to avoid duplicated requests Added a time.Sleep since without that the last item wasn't being added to the list of results

Fixes #33

Co-authored-by: Baruch Odem <[email protected]>

I accidently used `docker save` instead of `make save` on #48.

Resolves #31 ### Features - We expect the user to give us a *Personal Access Token*. This token can be retrieved from the browser *Dev Tool*, or by authenticating a *Discord App*. See #31 (comment). - The user must give at least one `--discord-server`, we will not loop over all the user's servers. - The *Server* in Discord called *Guild* in the API. - If the user doesn't give `--discord-channel`, we will scan all the channels in a server. - We will scan all messages until `--discord-duration` or `--discord-messages-count` (the closest one). - **Threads**: Only *bots* can get all the **threads** from a **channel**. As we currently use *Personal Access Token*, we can get the **thread** from the **message** started it. So we will scan the **threads** that *started* in the time/limit arguments (as they are messages that scanned), and for each **thread** we will scan the messages in the time/limit requirements. ### Questions - It is a little confusing if you give multiple **servers** and multiple **channels**, because each **channel** relates to its **server**, and also if you will give **channels** only for one **server**, the other **server** will not be scanned at all. Compared to *Confluence*, which has **servers** and **spaces**, we are not scanning multiple **servers** in one scan. Waits for #52

I noticed in `pr-validation` we are running our linter twice, and not running unit-tests at all.

BREAKING CHANGE: the CLI command and arguments was changed See the discussion: #20 (comment) I think we don't need to support running *2ms* for multiple plugins at once. It is a rare case and it is confusing the command line arguments. Instead, I'm proposing using *SubCommand* for each plugin. --------- Co-authored-by: Jossef Harush Kadouri <[email protected]>

- make functions private - add helpful examples for -h/--help/help Close #43 ``` ❯ go run . 2ms Secrets Detection: A tool to detect secrets in public websites and communication services. Usage: 2ms [command] Plugins confluence Scan Confluence server discord Scan Discord server repository Scan local repository Additional Commands: completion Generate the autocompletion script for the specified shell help Help about any command Flags: -h, --help help for 2ms --log-level string log level (trace, debug, info, warn, error, fatal) (default "info") --tags strings select rules to be applied (default [all]) -v, --version version for 2ms Use "2ms [command] --help" for more information about a command. ``` ``` ❯ go run . help confluence Scan Confluence server for sensitive information Usage: 2ms confluence --url URL [flags] Flags: -h, --help help for confluence --history Scan pages history --spaces stringArray Confluence spaces: The names or IDs of the spaces to scan --token string The Confluence API token for authentication --url string Confluence server URL (example: https://company.atlassian.net/wiki) [required] --username string Confluence user name or email for authentication Global Flags: --log-level string log level (trace, debug, info, warn, error, fatal) (default "info") --tags strings select rules to be applied (default [all]) ``` ``` ❯ go run . help discord Scan Discord server for sensitive information. Usage: 2ms discord --token TOKEN --server SERVER [flags] Flags: --channel stringArray Discord channels IDs to scan. If not provided, all channels will be scanned --duration duration The time interval to scan from the current time. For example, 24h for 24 hours or 7d for 7 days. (default 336h0m0s) -h, --help help for discord --messages-count int The number of messages to scan. If not provided, all messages will be scanned until the fromDate flag value. --server stringArray Discord servers IDs to scan [required] --token string Discord token [required] Global Flags: --log-level string log level (trace, debug, info, warn, error, fatal) (default "info") --tags strings select rules to be applied (default [all]) ``` ``` ❯ go run . help repository Scan local repository for sensitive information Usage: 2ms repository --path PATH [flags] Flags: -h, --help help for repository --path string Local repository path [required] Global Flags: --log-level string log level (trace, debug, info, warn, error, fatal) (default "info") --tags strings select rules to be applied (default [all]) ``` --------- Co-authored-by: Jossef Harush Kadouri <[email protected]>

close #40

Close: #32 ``` ❯ go run . help slack Scan Slack team for sensitive information. Usage: 2ms slack --token TOKEN --team TEAM [flags] Flags: --channel stringArray Slack channels to scan --duration duration Slack backward duration for messages (ex: 24h, 7d, 1M, 1y) (default 336h0m0s) -h, --help help for slack --messages-count int Slack messages count to scan (0 = all messages) --team string Slack team name or ID [required] --token string Slack token [required] Global Flags: --all scan all plugins (default true) --log-level string log level (trace, debug, info, warn, error, fatal) (default "info") --tags strings select rules to be applied (default [all]) ``` Like in Discord, more knowledge is required to integrate this plugin into an E2E system. For example, except for retrieving the token (from **OAuth & Permissions** in the Slack App page), you have to add your app to each **channel** you want to read. ![image](https://github.com/Checkmarx/2ms/assets/17686879/0db4994a-7304-4e70-a268-fff242f6ca35) --------- Co-authored-by: Jossef Harush Kadouri <[email protected]>

- split into 2 depends jobs - publish to Dockerhub For #77 (not closing until testing it)

- get folders and components - handle only `component` and `folder` types - send content to items - avoid reaching the rate limit Note we are handling only `folder`s to find sub-folders or `components`, and reading only the `component` (document) content. ### Rate Limit The rate limit **per minute** described [here](https://paligo.net/docs/apidocs/en/index-en.html#UUID-a5b548af-9a37-d305-f5a8-11142d86fe20). The core lib `time/rate` is limiting the rate **per second**. That's why I'm starting with open seats for all the requests-per-minute, in case there are fewer requests than the limit. After the first too-many-requests error, I'm reducing the available seats to 1, so the frequency of the requests will not reach the rate limit. Close #75

All IDs are now usable for the user

That's because the Docker publish step is still in progress, but I don't want it will block us from creating a new release.

In Jenkins, I sew users hosting their credentials as a `username:token` base64 encoded secret, so I want to support it as an argument.

- refactor: repository plugin initialization - change repository to filesystem - repository plugin - support scanning historical git commits Fixes #66

cx-leonardo-fontes · 2025-09-04T09:25:58Z

engine/engine.go

+
+		ScanConfig: engineConfig.ScanConfig,
+
+		secretsChan:                    make(chan *secrets.Secret, 10),


is there any reason for this to be specifically 10? will all channels from here have the same buffer size? if so, what about a const for this size?

At the time, I just had to put something. I changed all these to runtime.GOMAXPROCS which is roughly the amount of concurrency we have.

cx-leonardo-fontes · 2025-09-04T09:37:37Z

plugins/plugins.go

+
+func NewChannels(opts ...Option) PluginChannels {
+	channels := &Channels{
+		Items:  make(chan ISourceItem, 64),


same question I did in other comment, is there any reason for the channel buffer size to be exactly this value? and the same for the error channel

see my answer above

…ed channels size

Closes # **Proposed Changes** - It introduces a workerpool to optimize the detection which wraps around [pond](https://github.com/alitto/pond) - Initializes with a default number of workers calculated by number of CPUs x2 - Reached that conclusion with the benchmark, see my comment below with the stats. - This workerpool can be used for other purposes in next steps of the pipeline (in future PRs) - On a future PR we won't wait for the detection to end. - Fixed some linter issues  **Checklist** - [X] I covered my changes with tests. - [ ] I Updated the documentation that is affected by my changes: - [ ] Change in the CLI arguments - [ ] Change in the configuration file I submit this contribution under the Apache-2.0 license. --------- Co-authored-by: Rogério Dalot <[email protected]>

The base branch was changed.

github-actions · 2025-09-09T11:01:53Z

KICS version: v1.7.13

	Category	Results
	HIGH	0
	MEDIUM	0
	LOW	0
	INFO	0
	TRACE	0
	TOTAL	0

Metric	Values
Files scanned	13
Files parsed	13
Files failed to scan	0
Total executed queries	53
Queries failed to execute	0
Execution time	1

cx-rogerio-dalot · 2025-09-11T16:30:54Z

messed up the history trying to fix commit history because of .2ms.yml, this PR has moved into #327

kaplanlior and others added 30 commits March 29, 2023 15:16

Update license to Apache 2.0

a6b2edd

refactor: confluence plugin simplification (#37)

23877bf

* Refactor Confluence Plugin * Apply PR suggestions

feat: argument to print compiled version (#36)

c379717

closes #19

docs: fixed param

d4d5314

refactor: http lib (#47)

56c6112

* ignore .vscode * feat(confluence): filter spaces * fix(confluence): use username and token * chore: move httpRequest to lib

ci: create docker image (#48)

dad95a9

closes #34 - create docker image - create makefile - validate docker on pr-validation - upload tar image to Github release - run Kics on pr-validation - fix Kics issues

feat: implement concurrency mechanism (#52)

a4b0494

Implement suggested concurrency logic Also added confluence window constant to avoid duplicated requests Added a time.Sleep since without that the last item wasn't being added to the list of results

fix: remove secret location from report (#58)

1a8c38f

Fixes #33

feat: adds repository plugin (#62)

3d1a36a

Co-authored-by: Baruch Odem <[email protected]>

ci: use make save to publish image.tar (#60)

43bbc7a

I accidently used `docker save` instead of `make save` on #48.

infra: remove duplicate lint and run tests (#67)

0cf24bb

I noticed in `pr-validation` we are running our linter twice, and not running unit-tests at all.

docs: updated README.md with mascot

cf9b39e

feat: output report to file + different formats (#71)

2d4a264

close #40

cd: release docker image to dockerhub on manual version (#79)

b233512

- split into 2 depends jobs - publish to Dockerhub For #77 (not closing until testing it)

ci: validate PR title (#81)

14205d1

docker/login-action v2

24e6fc3

refactor: repository plugin initialization (#86)

aaa7781

refactor: remove unused Source field (#87)

cdc58c0

All IDs are now usable for the user

ci: publish Docker after creating release (#85)

2031125

That's because the Docker publish step is still in progress, but I don't want it will block us from creating a new release.

feat: accept base64 encoded credentials (#84)

9be6489

In Jenkins, I sew users hosting their credentials as a `username:token` base64 encoded secret, so I want to support it as an argument.

feat: support scanning historical git commits (#88)

e5daef2

- refactor: repository plugin initialization - change repository to filesystem - repository plugin - support scanning historical git commits Fixes #66

Fix typo

97b697f

Update base images

9027a74

Fix tests

15f6d96

cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor-2 branch from 1fcb373 to 03320ed Compare September 2, 2025 11:29

cx-rogerio-dalot added 5 commits September 2, 2025 12:35

Remove benchmark from the tests and bump package

fcf4b94

refactor pipeline

80ce7df

Fix tests and minor improvements

e1bdac5

minor improvements

2474276

cleanup

6e797cb

cx-leonardo-fontes reviewed Sep 4, 2025

View reviewed changes

fix flakiness of test and put the amount of concurrency on our buffer…

540f9e4

…ed channels size

cx-leonardo-fontes previously approved these changes Sep 4, 2025

View reviewed changes

Base automatically changed from AST-99998-architecture-refactor to master September 9, 2025 09:57

cx-rogerio-dalot added 2 commits September 9, 2025 11:55

Merge branch 'master' into AST-99998-architecture-refactor-2

7a1bcc2

Fix merge issue

f26de30

cx-rogerio-dalot added 9 commits September 9, 2025 12:04

Fix .2ms.yml

e00ae66

Keep behavior of the flag

6de9d51

Adjust command vars

d8e77b0

Make sure config flag works and is tested

0819a72

Fix cli flags and improve integration tests

72c9754

update .2ms.yml

474469d

yet another update .2ms.yml

f8a936a

yet another update .2ms.yml

f4bbb90

yet another update .2ms.yml

321e9ce

cx-rogerio-dalot force-pushed the AST-99998-architecture-refactor-2 branch 2 times, most recently from 568e87e to 321e9ce Compare September 11, 2025 16:06

cx-rogerio-dalot closed this Sep 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: fixed race warnings #311

fix: fixed race warnings #311

Uh oh!

cx-rogerio-dalot commented Jul 24, 2025 •

edited

Loading

Uh oh!

cx-leonardo-fontes Sep 4, 2025 •

edited

Loading

Uh oh!

cx-rogerio-dalot Sep 4, 2025

Uh oh!

cx-leonardo-fontes Sep 4, 2025

Uh oh!

cx-rogerio-dalot Sep 4, 2025

Uh oh!

github-actions bot commented Sep 9, 2025

Uh oh!

cx-rogerio-dalot commented Sep 11, 2025

Uh oh!

Uh oh!


		ScanConfig: engineConfig.ScanConfig,

		secretsChan: make(chan *secrets.Secret, 10),

fix: fixed race warnings #311

fix: fixed race warnings #311

Uh oh!

Conversation

cx-rogerio-dalot commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cx-leonardo-fontes Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cx-rogerio-dalot Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

cx-leonardo-fontes Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

cx-rogerio-dalot Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 9, 2025

Uh oh!

cx-rogerio-dalot commented Sep 11, 2025

Uh oh!

Uh oh!

cx-rogerio-dalot commented Jul 24, 2025 •

edited

Loading

cx-leonardo-fontes Sep 4, 2025 •

edited

Loading