Skip to content

Metastore 1 2 43 #101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 47 commits into
base: master
Choose a base branch
from
Open

Metastore 1 2 43 #101

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
31db9db
separate stats
Oct 10, 2019
53928c9
separate stats
Oct 10, 2019
12c59ef
mm
Oct 11, 2019
60c28b0
hiveql
Oct 11, 2019
0677ace
fix errors
Oct 11, 2019
8937686
fix comp
Oct 11, 2019
5d6bfac
fix compile errors
Oct 11, 2019
63fff70
intial commit
Oct 11, 2019
d9d717c
changes for separating stats
Oct 14, 2019
dc53731
fix 2 scenario
Oct 14, 2019
7a3266e
separate cluster
Oct 15, 2019
ab2d11c
increment version
Oct 15, 2019
6c7fa67
Updates for standing up Stats cluster
Oct 15, 2019
f9f1445
update lock
Oct 15, 2019
db7e463
mergine
Oct 15, 2019
5a3b71c
Refactoring
Oct 15, 2019
310f65e
update lock query
Oct 15, 2019
97da446
merge with remoteMerge branch 'separate-stats' of https://github.com/…
Oct 15, 2019
a0ec3f1
Refactoring
Oct 15, 2019
1cdf11a
Refactoring
Oct 15, 2019
ba64a41
change age from hours to minutes
Oct 22, 2019
1431d2f
merge branch with remoteMerge branch 'separate-stats' of https://gith…
Oct 22, 2019
b6a6399
fix stats issue in qa
Oct 22, 2019
dc005a8
change version
Oct 22, 2019
660ce3c
fix group logic for autoscaling
Oct 22, 2019
7c869d0
fix group logic for autoscaling
Oct 22, 2019
83d058a
Updated the logic to start stats cluster
Oct 22, 2019
0bcbab6
fix sql
Oct 23, 2019
e340626
increment version
Oct 23, 2019
f4fcd2b
pk fix
Oct 23, 2019
a448de3
notification email subject updates and cleaning up notifications
Oct 24, 2019
8a10790
Sending notification when no partitions available to backload while b…
Oct 25, 2019
f6c22cc
Refactoring
Oct 25, 2019
410776f
Refactoring + stats for date range partition
Oct 28, 2019
d157196
refactoring
Oct 28, 2019
705623a
Updated constructing partition string
Nov 1, 2019
6c1f1ed
Updated constructing partition string
Nov 4, 2019
ef85c5e
Updated for cascade option
Nov 5, 2019
3b452d4
Merge pull request #94 from FINRAOS/separate-stats
VekasS Nov 8, 2019
d8e3292
1.5.0 (#95)
kusid Nov 8, 2019
958e293
Merge branch 'master' into develop
kusid Nov 8, 2019
adea736
Updated OSS version
Nov 8, 2019
6eebc11
1.5.0 (#97)
kusid Nov 9, 2019
b69624a
Merge branch 'master' into develop
kusid Nov 9, 2019
2e9e580
Avoid adding duplicate date range
Jan 2, 2020
07e25d4
Avoid adding duplicate date range
Jan 2, 2020
ec3ba42
Filter not available partitions while adding stats notifications
Jan 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion mdl/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

<artifactId>herd-mdl</artifactId>
<groupId>org.finra.herd-mdl</groupId>
<version>1.4.0</version>
<version>1.5.0</version>

<properties>

Expand Down
2 changes: 1 addition & 1 deletion metastor/managedObjectLoader/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
<parent>
<artifactId>metastore</artifactId>
<groupId>org.finra.herd-mdl.metastore</groupId>
<version>1.2.38</version>
<version>1.2.43</version>
</parent>

<modelVersion>4.0.0</modelVersion>
Expand Down
416 changes: 222 additions & 194 deletions ...agedObjectLoader/src/main/java/org/finra/herd/metastore/managed/datamgmt/DataMgmtSvc.java
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 1 addition & 7 deletions .../src/main/java/org/finra/herd/metastore/managed/jobProcessor/BackLoadObjectProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,15 +133,9 @@ private void identifyPartitionsAndBackLoad( JobDefinition od, JobSubmitterInfo j
if ( !ts.isEmpty() ) {

if ( jsi.isPartitionDateType() ) {
if ( Strings.isNullOrEmpty( startDate.get() ) ) {
startDate.set( ts.first() );
}

// To include skipped partition dates - using end date of the previous chunk
String endDate = ts.last();
jsi.setPartitionValues( String.format( "%s%s%s", endDate, JobProcessorConstants.DOUBLE_UNDERSCORE, startDate ) );
jsi.setPartitionValues( String.format( "%s%s%s", ts.last(), JobProcessorConstants.DOUBLE_UNDERSCORE, ts.first() ) );
addPartitions( jsi );
startDate.set( endDate );
} else {
jsi.setPartitionValues( delimitedPartitionValues( ts.descendingSet() ) );
addPartitions( jsi );
Expand Down
3 changes: 3 additions & 0 deletions ...tLoader/src/main/java/org/finra/herd/metastore/managed/jobProcessor/HiveHqlGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,9 @@ protected void addAnalyzeStats( JobDefinition jd, List<String> partitions ) {
if ( partitions.size() == 1 ) {
submitStatsJob( jd, jd.partitionValuesForStats(partitions.get( 0 )) );
} else {
//Filter not available Partitions
dataMgmtSvc.filterPartitionsAsPerAvailability( jd, partitions );

partitions.stream()
.forEach( s -> submitStatsJob( jd, s ) );
}
Expand Down
2 changes: 1 addition & 1 deletion metastor/metastoreOperations/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<artifactId>metastore</artifactId>
<groupId>org.finra.herd-mdl.metastore</groupId>
<version>1.2.38</version>
<version>1.2.43</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion metastor/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@

<artifactId>metastore</artifactId>
<groupId>org.finra.herd-mdl.metastore</groupId>
<version>1.2.38</version>
<version>1.2.43</version>
<packaging>pom</packaging>

<properties>
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
<groupId>org.finra.herd-mdl</groupId>
<artifactId>herd-mdl</artifactId>
<packaging>pom</packaging>
<version>1.4.0</version>
<version>1.5.0</version>

<description>This is the base pom for the herd-mdl project.</description>

Expand Down