Skip to content

HADOOP-19575. ABFS: [FNSOverBlob] Add Distinct String In User Agent to Get Telemetry for FNS-Blob #7713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: trunk
Choose a base branch
from
+47 −0
Open

HADOOP-19575. ABFS: [FNSOverBlob] Add Distinct String In User Agent to Get Telemetry for FNS-Blob #7713

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d538ce5
fns-blob unique identifier
manika137 May 26, 2025
5930d41
added test
manika137 May 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ...p-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@
import org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.Permissions;
import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants;
import org.apache.hadoop.fs.azurebfs.constants.AbfsHttpConstants.ApiVersion;
import org.apache.hadoop.fs.azurebfs.constants.AbfsServiceType;
import org.apache.hadoop.fs.azurebfs.constants.FSOperationType;
import org.apache.hadoop.fs.azurebfs.constants.HttpHeaderConfigurations;
import org.apache.hadoop.fs.azurebfs.constants.HttpOperationType;
Expand Down Expand Up @@ -152,6 +153,7 @@ public abstract class AbfsClient implements Closeable {
public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class);
public static final String HUNDRED_CONTINUE_USER_AGENT = SINGLE_WHITE_SPACE + HUNDRED_CONTINUE + SEMICOLON;
public static final String ABFS_CLIENT_TIMER_THREAD_NAME = "abfs-timer-client";
public static final String FNS_BLOB_USER_AGENT_IDENTIFIER = "FNS";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason for just facing FNS as value for FNS Blob user agent? FNS user agent can be there for FNS DFS as well, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are adding the string only for FNS-Blob cases (since we wanted to check their usage only)
Initially we added "FNS-Blob" as string but then decided to shorten it to "FNS" since the endpoint we can check from the URL itself


private final URL baseUrl;
private final SharedKeyCredentials sharedKeyCredentials;
Expand Down Expand Up @@ -1319,6 +1321,12 @@ String initializeUserAgent(final AbfsConfiguration abfsConfiguration,
sb.append(FORWARD_SLASH);
sb.append(abfsConfiguration.getClusterType());

// Add a unique identifier in FNS-Blob user agent string
if (!getIsNamespaceEnabled() && abfsConfiguration.getFsConfiguredServiceType() == AbfsServiceType.BLOB){
sb.append(SEMICOLON).append(SINGLE_WHITE_SPACE);
sb.append(FNS_BLOB_USER_AGENT_IDENTIFIER);
}

sb.append(")");

appendIfNotEmpty(sb, abfsConfiguration.getCustomUserAgentPrefix(), false);
Expand Down
39 changes: 39 additions & 0 deletions ...ls/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ public final class ITestAbfsClient extends AbstractAbfsIntegrationTest {

private static final String ACCOUNT_NAME = "bogusAccountName.dfs.core.windows.net";
private static final String FS_AZURE_USER_AGENT_PREFIX = "Partner Service";
private static final String FNS_BLOB_USER_AGENT_IDENTIFIER = "FNS";
private static final String HUNDRED_CONTINUE_USER_AGENT = SINGLE_WHITE_SPACE + HUNDRED_CONTINUE + SEMICOLON;
private static final String TEST_PATH = "/testfile";
public static final int REDUCED_RETRY_COUNT = 2;
Expand Down Expand Up @@ -355,6 +356,44 @@ public void verifyUserAgentClusterType() throws Exception {
.contains(DEFAULT_VALUE_UNKNOWN);
}

@Test
// Test to verify the unique identifier in user agent string for FNS-Blob accounts
public void verifyUserAgentForFNSBlob() throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add similar test for dfs and validate that no extra string is added.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

Assume.assumeTrue(JDK_HTTP_URL_CONNECTION == httpOperationType);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this assume for connection type?

assumeHnsDisabled();
Assume.assumeTrue(getAbfsServiceType() == AbfsServiceType.BLOB);
final AzureBlobFileSystem fs = getFileSystem(getRawConfiguration());
final AbfsConfiguration configuration = fs.getAbfsStore()
.getAbfsConfiguration();

String userAgentStr = getUserAgentString(configuration, false);
verifyBasicInfo(userAgentStr);
Assertions.assertThat(userAgentStr)
.describedAs(
"User-Agent string for FNS accounts on Blob endpoint should contain "
+ FNS_BLOB_USER_AGENT_IDENTIFIER)
.contains(FNS_BLOB_USER_AGENT_IDENTIFIER);
}

@Test
// Test to verify that the user agent string for non-FNS-Blob accounts
// does not contain the FNS identifier.
public void verifyUserAgentForDFS() throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assumeHnsDisabled() as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For both HNS-DFS and FNS-DFS we would want to check that the string isnt added right

Assume.assumeTrue(JDK_HTTP_URL_CONNECTION == httpOperationType);
Assume.assumeTrue(getAbfsServiceType() == AbfsServiceType.DFS);
final AzureBlobFileSystem fs = getFileSystem(getRawConfiguration());
final AbfsConfiguration configuration = fs.getAbfsStore()
.getAbfsConfiguration();

String userAgentStr = getUserAgentString(configuration, false);
verifyBasicInfo(userAgentStr);
Assertions.assertThat(userAgentStr)
.describedAs(
"User-Agent string for non-FNS-Blob accounts should not contain"
+ FNS_BLOB_USER_AGENT_IDENTIFIER)
.doesNotContain(FNS_BLOB_USER_AGENT_IDENTIFIER);
}

public static AbfsClient createTestClientFromCurrentContext(
AbfsClient baseAbfsClientInstance,
AbfsConfiguration abfsConfig) throws IOException, URISyntaxException {
Expand Down