Skip to content

Use vendored calcite 1.40 #35718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jul 29, 2025
Merged

Use vendored calcite 1.40 #35718

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
31ce80f
Use vendored Calcite 1.40.0
Abacn Jul 28, 2025
a91d106
Fix after upgrading to Calcite 1.40
Abacn Jul 28, 2025
6a0f130
Fix Iceberg SQL test
ahmedabu98 Jul 22, 2025
531b5fc
trigger tests
Abacn Jul 28, 2025
643d5e5
Fix tpcds json dependency change
Abacn Jul 28, 2025
377dff5
Update CHANGES.md
Abacn Jul 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PostCommit_SQL.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run ",
"modification": 2
"modification": 3
}
3 changes: 1 addition & 2 deletions .github/trigger_files/beam_PostCommit_XVR_Direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
{
"https://github.com/apache/beam/pull/32648": "testing Flink 1.19 support",
"modification": 3
"modification": 4
}
6 changes: 5 additions & 1 deletion CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,11 @@
## Breaking Changes

* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).
* Previously deprecated Beam ZetaSQL component has been removed from new releases ([#34423](https://github.com/apache/beam/issues/34423)).
* Previously deprecated Beam ZetaSQL component has been removed ([#34423](https://github.com/apache/beam/issues/34423)).
ZetaSQL users could migrate to Calcite SQL with BigQuery dialect enabled.
* Upgraded Beam vendored Calcite to 1.40.0 for Beam SQL ([#35483](https://github.com/apache/beam/issues/35483)), which
improves support for BigQuery and other SQL dialects. Note: Minor behavior changes are observed such as output
significant digits related to casting.

## Deprecations

Expand Down
9 changes: 4 additions & 5 deletions buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -907,7 +907,7 @@ class BeamModulePlugin implements Plugin<Project> {
threetenbp : "org.threeten:threetenbp:1.6.8",
vendored_grpc_1_69_0 : "org.apache.beam:beam-vendor-grpc-1_69_0:0.1",
vendored_guava_32_1_2_jre : "org.apache.beam:beam-vendor-guava-32_1_2-jre:0.1",
vendored_calcite_1_28_0 : "org.apache.beam:beam-vendor-calcite-1_28_0:0.2",
vendored_calcite_1_40_0 : "org.apache.beam:beam-vendor-calcite-1_40_0:0.1",
woodstox_core_asl : "org.codehaus.woodstox:woodstox-core-asl:4.4.1",
zstd_jni : "com.github.luben:zstd-jni:1.5.6-3",
quickcheck_core : "com.pholser:junit-quickcheck-core:$quickcheck_version",
Expand Down Expand Up @@ -1832,21 +1832,20 @@ class BeamModulePlugin implements Plugin<Project> {
project.ext.includeInJavaBom = configuration.publish
project.ext.exportJavadoc = configuration.exportJavadoc

boolean publishEnabledByCommand = isRelease(project) || project.hasProperty('publishing')
if (forkJavaVersion == '') {
// project needs newer version and not served.
// If not publishing ,disable the project. Otherwise, fail the build
def msg = "project ${project.name} needs newer Java version to compile. Consider set -Pjava${project.javaVersion}Home"
if (publishEnabledByCommand) {
throw new GradleException("Publish enabled but " + msg + ".")
if (isRelease(project)) {
throw new GradleException("Release enabled but " + msg + ".")
} else {
Comment on lines +1839 to 1841
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This hunk introduces a logic change that seems unrelated to the Calcite version upgrade. The condition for throwing a GradleException when a required Java version is missing has been changed. While this change might be desirable, including it in a large dependency upgrade PR makes it harder to review and understand its impact. It would be better to extract this logic change into a separate, focused pull request. This will improve clarity and make the history easier to follow.

logger.config(msg + " if needed.")
project.tasks.each {
it.enabled = false
}
}
}
if (publishEnabledByCommand && configuration.publish) {
if ((isRelease(project) || project.hasProperty('publishing')) && configuration.publish) {
project.apply plugin: "maven-publish"

// plugin to support repository authentication via ~/.m2/settings.xml
Expand Down
14 changes: 9 additions & 5 deletions sdks/java/extensions/sql/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ dependencies {
javacc "net.java.dev.javacc:javacc:4.0"
fmppTask "com.googlecode.fmpp-maven-plugin:fmpp-maven-plugin:1.0"
fmppTask "org.freemarker:freemarker:2.3.31"
fmppTemplates library.java.vendored_calcite_1_28_0
fmppTemplates library.java.vendored_calcite_1_40_0
implementation project(path: ":sdks:java:core", configuration: "shadow")
implementation project(":sdks:java:managed")
implementation project(":sdks:java:io:iceberg")
Expand All @@ -87,7 +87,7 @@ dependencies {
implementation library.java.commons_csv
implementation library.java.jackson_databind
implementation library.java.joda_time
implementation library.java.vendored_calcite_1_28_0
implementation library.java.vendored_calcite_1_40_0
implementation "org.codehaus.janino:janino:3.0.11"
implementation "org.codehaus.janino:commons-compiler:3.0.11"
implementation library.java.jackson_core
Expand Down Expand Up @@ -117,7 +117,7 @@ dependencies {

testImplementation "org.apache.iceberg:iceberg-api:1.6.1"
testImplementation "org.apache.iceberg:iceberg-core:1.6.1"
testImplementation library.java.vendored_calcite_1_28_0
testImplementation library.java.vendored_calcite_1_40_0
testImplementation library.java.vendored_guava_32_1_2_jre
testImplementation library.java.junit
testImplementation library.java.quickcheck_core
Expand Down Expand Up @@ -164,11 +164,15 @@ task copyFmppTemplatesFromCalciteCore(type: Copy) {
into "${project.buildDir}/templates-fmpp"
filter{
line ->
line.replace('import org.apache.calcite.', 'import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.')
line.replace('import org.apache.calcite.', 'import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.')
}
filter{
line ->
line.replace('import static org.apache.calcite.', 'import static org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.')
line.replace('import static org.apache.calcite.', 'import static org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.')
}
filter{
line ->
line.replace('import com.google.common.', 'import org.apache.beam.vendor.calcite.v1_40_0.com.google.common.')
Comment on lines 166 to +175
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For better readability and a minor performance improvement, these three separate filter closures can be combined into a single one.

  filter { line ->
    line.replace('import org.apache.calcite.', 'import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.')
        .replace('import static org.apache.calcite.', 'import static org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.')
        .replace('import com.google.common.', 'import org.apache.beam.vendor.calcite.v1_40_0.com.google.common.')
  }

}
}

Expand Down
4 changes: 2 additions & 2 deletions sdks/java/extensions/sql/jdbc/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ dependencies {
implementation "jline:jline:2.14.6"
permitUnusedDeclared "jline:jline:2.14.6" // BEAM-11761
implementation "sqlline:sqlline:1.4.0"
implementation library.java.vendored_calcite_1_28_0
permitUnusedDeclared library.java.vendored_calcite_1_28_0
implementation library.java.vendored_calcite_1_40_0
permitUnusedDeclared library.java.vendored_calcite_1_40_0
testImplementation project(path: ":sdks:java:core", configuration: "shadow")
testImplementation project(path: ":sdks:java:io:google-cloud-platform")
testImplementation project(path: ":sdks:java:extensions:google-cloud-platform-core")
Expand Down
10 changes: 5 additions & 5 deletions ...ensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,9 +129,9 @@ public void testSelectFromPubsub() throws Exception {

assertThat(
Arrays.asList(
Arrays.asList("2018-07-01 21:25:20", "enroute", "40.702", "-74.001"),
Arrays.asList("2018-07-01 21:26:06", "enroute", "40.703", "-74.002"),
Arrays.asList("2018-07-02 13:26:06", "enroute", "30.0", "-72.32324")),
Arrays.asList("2018-07-01 21:25:20.000000", "enroute", "40.702", "-74.001"),
Arrays.asList("2018-07-01 21:26:06.000000", "enroute", "40.703", "-74.002"),
Arrays.asList("2018-07-02 13:26:06.000000", "enroute", "30.0", "-72.32324")),
everyItem(IsIn.isOneOf(expectedResult.get(30, TimeUnit.SECONDS).toArray())));
}

Expand Down Expand Up @@ -170,8 +170,8 @@ public void testFilterForSouthManhattan() throws Exception {

assertThat(
Arrays.asList(
Arrays.asList("2018-07-01 21:25:20", "enroute", "40.701", "-74.001"),
Arrays.asList("2018-07-01 21:26:06", "enroute", "40.702", "-74.002")),
Arrays.asList("2018-07-01 21:25:20.000000", "enroute", "40.701", "-74.001"),
Arrays.asList("2018-07-01 21:26:06.000000", "enroute", "40.702", "-74.002")),
everyItem(IsIn.isOneOf(expectedResult.get(30, TimeUnit.SECONDS).toArray())));
}

Expand Down
13 changes: 7 additions & 6 deletions ...sions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,8 @@ public void testSqlLine_fixedWindow() throws Exception {
List<List<String>> lines = toLines(byteArrayOutputStream);
assertThat(
Arrays.asList(
Arrays.asList("2018-07-01 21:26:06", "1"), Arrays.asList("2018-07-01 21:26:07", "1")),
Arrays.asList("2018-07-01 21:26:06.000000", "1"),
Arrays.asList("2018-07-01 21:26:07.000000", "1")),
everyItem(is(oneOf(lines.toArray()))));
}

Expand All @@ -190,11 +191,11 @@ public void testSqlLine_slidingWindow() throws Exception {
List<List<String>> lines = toLines(byteArrayOutputStream);
assertThat(
Arrays.asList(
Arrays.asList("2018-07-01 21:26:07", "1"),
Arrays.asList("2018-07-01 21:26:08", "2"),
Arrays.asList("2018-07-01 21:26:09", "2"),
Arrays.asList("2018-07-01 21:26:10", "2"),
Arrays.asList("2018-07-01 21:26:11", "1")),
Arrays.asList("2018-07-01 21:26:07.000000", "1"),
Arrays.asList("2018-07-01 21:26:08.000000", "2"),
Arrays.asList("2018-07-01 21:26:09.000000", "2"),
Arrays.asList("2018-07-01 21:26:10.000000", "2"),
Arrays.asList("2018-07-01 21:26:11.000000", "1")),
everyItem(is(oneOf(lines.toArray()))));
}
}
6 changes: 3 additions & 3 deletions .../java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryIOPushDownIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,9 @@
import org.apache.beam.sdk.testutils.metrics.TimeMonitor;
import org.apache.beam.sdk.testutils.publishing.InfluxDBSettings;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.plan.RelOptRule;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.tools.RuleSet;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.tools.RuleSets;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.plan.RelOptRule;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.tools.RuleSet;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.tools.RuleSets;
import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
import org.junit.Before;
import org.junit.BeforeClass;
Expand Down
42 changes: 37 additions & 5 deletions sdks/java/extensions/sql/src/main/codegen/config.fmpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,11 @@ data: {

# List of import statements.
imports: [
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.schema.ColumnStrategy"
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlCreate"
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlDrop"
"org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.type.SqlTypeName"
"org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.schema.ColumnStrategy"
"org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlCreate"
"org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlDrop"
"org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.type.SqlTypeName"
"org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.util.NlsString"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlCreateCatalog"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlCreateDatabase"
"org.apache.beam.sdk.extensions.sql.impl.parser.SqlCreateExternalTable"
Expand Down Expand Up @@ -64,6 +65,8 @@ data: {
"AFTER"
"ALWAYS"
"APPLY"
"ARRAY_AGG"
"ARRAY_CONCAT_AGG"
"ASC"
"ASSERTION"
"ASSIGNMENT"
Expand Down Expand Up @@ -102,12 +105,19 @@ data: {
"CONSTRAINTS"
"CONSTRAINT_SCHEMA"
"CONSTRUCTOR"
"CONTAINS_SUBSTR"
"CONTINUE"
"CURSOR_NAME"
"DATA"
"DATABASE"
"DATE_DIFF"
"DATE_TRUNC"
"DATETIME_DIFF"
"DATETIME_INTERVAL_CODE"
"DATETIME_INTERVAL_PRECISION"
"DATETIME_TRUNC"
"DAYOFWEEK"
"DAYOFYEAR"
"DAYS"
"DECADE"
"DEFAULTS"
Expand All @@ -126,6 +136,7 @@ data: {
"DOMAIN"
"DOW"
"DOY"
"DOT_FORMAT"
"DYNAMIC_FUNCTION"
"DYNAMIC_FUNCTION_CODE"
"ENCODING"
Expand All @@ -148,13 +159,16 @@ data: {
"GO"
"GOTO"
"GRANTED"
"GROUP_CONCAT"
"HIERARCHY"
"HOP"
"HOURS"
"IGNORE"
"ILIKE"
"IMMEDIATE"
"IMMEDIATELY"
"IMPLEMENTATION"
"INCLUDE"
"INCLUDING"
"INCREMENT"
"INITIALLY"
Expand Down Expand Up @@ -222,6 +236,7 @@ data: {
"PASSTHROUGH"
"PAST"
"PATH"
"PIVOT"
"PLACING"
"PLAN"
"PLI"
Expand All @@ -231,6 +246,7 @@ data: {
"PRIVILEGES"
"PUBLIC"
"QUARTER"
"QUARTERS"
"READ"
"RELATIVE"
"REPEATABLE"
Expand All @@ -243,6 +259,7 @@ data: {
"RETURNED_OCTET_LENGTH"
"RETURNED_SQLSTATE"
"RETURNING"
"RLIKE"
"ROLE"
"ROUTINE"
"ROUTINE_CATALOG"
Expand All @@ -260,6 +277,7 @@ data: {
"SECTION"
"SECURITY"
"SELF"
"SEPARATOR"
"SEQUENCE"
"SERIALIZABLE"
"SERVER"
Expand Down Expand Up @@ -322,15 +340,20 @@ data: {
"SQL_VARCHAR"
"STATE"
"STATEMENT"
"STRING_AGG"
"STRUCTURE"
"STYLE"
"SUBCLASS_ORIGIN"
"SUBSTITUTE"
"TABLE_NAME"
"TEMPORARY"
"TIES"
"TIME_DIFF"
"TIME_TRUNC"
"TIMESTAMPADD"
"TIMESTAMPDIFF"
"TIMESTAMP_DIFF"
"TIMESTAMP_TRUNC"
"TOP_LEVEL_COUNT"
"TRANSACTION"
"TRANSACTIONS_ACTIVE"
Expand All @@ -347,6 +370,7 @@ data: {
"UNCOMMITTED"
"UNCONDITIONAL"
"UNDER"
"UNPIVOT"
"UNNAMED"
"USAGE"
"USER_DEFINED_TYPE_CATALOG"
Expand All @@ -359,6 +383,7 @@ data: {
"VERSION"
"VIEW"
"WEEK"
"WEEKS"
"WORK"
"WRAPPER"
"WRITE"
Expand Down Expand Up @@ -445,6 +470,12 @@ data: {
"SqlDropCatalog"
]

# List of methods for parsing extensions to "TRUNCATE" calls.
# Each must accept arguments "(SqlParserPos pos)".
# Example: "SqlTruncate".
truncateStatementParserMethods: [
]

# Binary operators tokens
binaryOperatorsTokens: [
]
Expand All @@ -461,11 +492,12 @@ data: {
"parserImpls.ftl"
]

setOptionParserMethod: "SqlSetOption"
includePosixOperators: false
includeCompoundIdentifier: true
includeBraces: true
includeAdditionalDeclarations: false

includeParsingStringLiteralAsArrayLiteral: false
}
}

Expand Down
14 changes: 7 additions & 7 deletions ...nsions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@
import java.util.Collections;
import java.util.List;
import org.apache.beam.sdk.extensions.sql.impl.TableName;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlAsOperator;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlCall;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlIdentifier;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlJoin;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlNode;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlSelect;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.sql.SqlSetOperator;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlAsOperator;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlCall;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlIdentifier;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlJoin;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlNode;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlSelect;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.sql.SqlSetOperator;
import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList;
import org.checkerframework.checker.nullness.qual.Nullable;

Expand Down
16 changes: 8 additions & 8 deletions ...tensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamCalciteSchema.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,13 @@
import org.apache.beam.sdk.extensions.sql.meta.catalog.Catalog;
import org.apache.beam.sdk.extensions.sql.meta.catalog.CatalogManager;
import org.apache.beam.sdk.extensions.sql.meta.provider.TableProvider;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.linq4j.tree.Expression;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.rel.type.RelProtoDataType;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.schema.Function;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.schema.Schema;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.schema.SchemaPlus;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.schema.SchemaVersion;
import org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.schema.Schemas;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.linq4j.tree.Expression;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.rel.type.RelProtoDataType;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.schema.Function;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.schema.Schema;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.schema.SchemaPlus;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.schema.SchemaVersion;
import org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.schema.Schemas;
import org.checkerframework.checker.nullness.qual.Nullable;

/** Adapter from {@link TableProvider} to {@link Schema}. */
Expand Down Expand Up @@ -120,7 +120,7 @@ public Set<String> getTypeNames() {
}

@Override
public org.apache.beam.vendor.calcite.v1_28_0.org.apache.calcite.schema.Table getTable(
public org.apache.beam.vendor.calcite.v1_40_0.org.apache.calcite.schema.Table getTable(
String name) {
Table table = resolveMetastore().getTable(name);
if (table == null) {
Expand Down
Loading
Loading