Skip to content

Commit f071c9b

Browse files
mulugetamdblock
mulugetam
and
dblock
authored
Add experimental support for zstd compression. (#3577)
* Add experimental support for zstd and lz4 (native) compression. Add experimental support for zstd (with and without dictionary support) and lz4 (native) compressions as discussed in: #3354. Users would be able to set the index.codec setting with the values "lz4_native" (for lz4 native), "zstd" (for zstd with dictionary), and "zstd_no_dict" (for zstd without a dictionary). Signed-off-by: Mulugeta Mammo <[email protected]> * Fix license issues, add tests for zstd, zstd_no_dict, and lz4_native compressions. Signed-off-by: Mulugeta Mammo <[email protected]> * Fix DCO and and issues with CodecTests.java. Signed-off-by: Mulugeta Mammo <[email protected]> * Fix forbidden api violation error for lz4-java. Signed-off-by: Mulugeta Mammo <[email protected]> * Fix license headers. Remove and fix unnecessary fields. Signed-off-by: Mulugeta Mammo <[email protected]> * Fix magic numbers. Use more restrictive access modifiers. Signed-off-by: Mulugeta Mammo <[email protected]> * Use protected access modifier for Zstd and LZ4 compression mode classes. Signed-off-by: Mulugeta Mammo <[email protected]> * Allow negative compression levels for zstd. Use more restrictive access modifiers. Signed-off-by: Mulugeta Mammo <[email protected]> * Use a more restrictive permission for loading zstd-jni and lz4-java libraries. Signed-off-by: Mulugeta Mammo <[email protected]> * Rename a file (follow a consistent version naming convention). Signed-off-by: Mulugeta Mammo <[email protected]> * Refactor and create a new custom-codecs sandbox module. Signed-off-by: Mulugeta Mammo <[email protected]> * Remove blank lines. Signed-off-by: Mulugeta Mammo <[email protected]> * Restore Lucene92CustomCodec to extend from FilterCodec. Signed-off-by: Mulugeta Mammo <[email protected]> * Make use of the compressionLevel argument. Signed-off-by: Mulugeta Mammo <[email protected]> * Make Lucene92CustomCodec abstract and use a package-private access modifier. Signed-off-by: Mulugeta Mammo <[email protected]> * Fix missing JavaDoc issues. Remove unused field in PerFieldMappingPostingFormatCodec. Signed-off-by: Mulugeta Mammo <[email protected]> * Fix lint errors. Signed-off-by: Mulugeta Mammo <[email protected]> * Fix the description for the custom-codecs plugin. Signed-off-by: Mulugeta Mammo <[email protected]> * Fix wildcard import and improve documentation. Signed-off-by: Mulugeta Mammo <[email protected]> * Access control exception fixed. Removed lz4-java support for now. - PRs were made to zstd-jni and lz4-java to use AccessController.doPrivileged. - The zstd-jni PR is merged since version 1.5.4-1. - The lz4-java support temporarily removed until the PR gets merged. Signed-off-by: Mulugeta Mammo <[email protected]> * Upgrade plugin to use Lucene95Codec. Rename files accordingly. - Upgrade plugin to use Lucene95Codec. Rename files accordingly. - Fix lint issue with plugin-security. - Remove thridPartyAudit that was there for supporting lz4-java. Signed-off-by: Mulugeta Mammo <[email protected]> * Add test cases for compression/decompression. Other minor changes. - add test cases for compression/decompression. - rename package. - add a CHANGELOG entry. - add more checks for signed integer arithmetic. Signed-off-by: Mulugeta Mammo <[email protected]> * Remove ES grant in plugin-security.policy. Fix minor javadoc issue. - Remove ES grant in plugin-security.policy file. - Replace @link and @see to fix javadoc error. Signed-off-by: Mulugeta Mammo <[email protected]> * Upgrade jettison version to 1.5.4. Signed-off-by: Mulugeta Mammo <[email protected]> * Update SHA for jettison 1.5.4. Signed-off-by: Mulugeta Mammo <[email protected]> --------- Signed-off-by: Mulugeta Mammo <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>
1 parent 95c6ed9 commit f071c9b

21 files changed

+1129
-0
lines changed

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
8989
- Enable sort optimization for all NumericTypes ([#6464](https://github.com/opensearch-project/OpenSearch/pull/6464)
9090
- Remove 'cluster_manager' role attachment when using 'node.master' deprecated setting ([#6331](https://github.com/opensearch-project/OpenSearch/pull/6331))
9191
- Add new cluster settings to ignore weighted round-robin routing and fallback to default behaviour. ([#6834](https://github.com/opensearch-project/OpenSearch/pull/6834))
92+
- Add experimental support for ZSTD compression. ([#3577](https://github.com/opensearch-project/OpenSearch/pull/3577))
9293

9394
### Dependencies
9495
- Bump `org.apache.logging.log4j:log4j-core` from 2.18.0 to 2.20.0 ([#6490](https://github.com/opensearch-project/OpenSearch/pull/6490))
@@ -104,6 +105,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
104105
- Bump `net.minidev:json-smart` from 2.4.7 to 2.4.10
105106
- Bump `org.apache.maven:maven-model` from 3.6.2 to 3.9.1
106107
- Bump `org.codehaus.jettison:jettison` from 1.5.3 to 1.5.4 ([#6878](https://github.com/opensearch-project/OpenSearch/pull/6878))
108+
- Add `com.github.luben:zstd-jni:1.5.4-1` ([#3577](https://github.com/opensearch-project/OpenSearch/pull/3577))
107109

108110
### Changed
109111
- Require MediaType in Strings.toString API ([#6009](https://github.com/opensearch-project/OpenSearch/pull/6009))

‎sandbox/modules/custom-codecs/build.gradle‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*
8+
* Modifications Copyright OpenSearch Contributors. See
9+
* GitHub history for details.
10+
*/
11+
12+
apply plugin: 'opensearch.opensearchplugin'
13+
apply plugin: 'opensearch.yaml-rest-test'
14+
15+
opensearchplugin {
16+
name 'custom-codecs'
17+
description 'A plugin that implements custom compression codecs.'
18+
classname 'org.opensearch.index.codec.customcodecs.CustomCodecPlugin'
19+
licenseFile rootProject.file('licenses/APACHE-LICENSE-2.0.txt')
20+
noticeFile rootProject.file('NOTICE.txt')
21+
}
22+
23+
dependencies {
24+
api "com.github.luben:zstd-jni:1.5.4-1"
25+
}
26+
27+
yamlRestTest.enabled = false;
28+
testingConventions.enabled = false;

‎sandbox/modules/custom-codecs/licenses/zstd-jni-1.5.4-1.jar.sha1‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
291ccaacc039e41932de877303edb6af98a91c24

‎sandbox/modules/custom-codecs/licenses/zstd-jni-LICENSE.txt‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
-----------------------------------------------------------------------------
2+
** Beginning of "BSD License" text. **
3+
4+
Zstd-jni: JNI bindings to Zstd Library
5+
6+
Copyright (c) 2015-present, Luben Karavelov/ All rights reserved.
7+
8+
BSD License
9+
10+
Redistribution and use in source and binary forms, with or without modification,
11+
are permitted provided that the following conditions are met:
12+
13+
1. Redistributions of source code must retain the above copyright notice, this
14+
list of conditions and the following disclaimer.
15+
16+
2. Redistributions in binary form must reproduce the above copyright notice, this
17+
list of conditions and the following disclaimer in the documentation and/or
18+
other materials provided with the distribution.
19+
20+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
21+
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
22+
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
23+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
24+
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
25+
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
26+
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
27+
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28+
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
29+
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

‎sandbox/modules/custom-codecs/licenses/zstd-jni-NOTICE.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
The code for the JNI bindings to Zstd library was originally authored by Luben Karavelov

‎sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/CustomCodecPlugin.java‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.index.codec.customcodecs;
10+
11+
import org.opensearch.plugins.Plugin;
12+
import org.opensearch.plugins.EnginePlugin;
13+
import org.opensearch.index.codec.CodecServiceFactory;
14+
import org.opensearch.index.IndexSettings;
15+
16+
import java.util.Optional;
17+
18+
/**
19+
* A plugin that implements custom codecs. Supports these codecs:
20+
* <ul>
21+
* <li>ZSTD
22+
* <li>ZSTDNODICT
23+
* </ul>
24+
*
25+
* @opensearch.internal
26+
*/
27+
public final class CustomCodecPlugin extends Plugin implements EnginePlugin {
28+
29+
/** Creates a new instance */
30+
public CustomCodecPlugin() {}
31+
32+
/**
33+
* @param indexSettings is the default indexSettings
34+
* @return the engine factory
35+
*/
36+
@Override
37+
public Optional<CodecServiceFactory> getCustomCodecServiceFactory(final IndexSettings indexSettings) {
38+
return Optional.of(new CustomCodecServiceFactory());
39+
}
40+
}

‎sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/CustomCodecService.java‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.index.codec.customcodecs;
10+
11+
import org.apache.logging.log4j.Logger;
12+
import org.apache.lucene.codecs.Codec;
13+
import org.opensearch.common.collect.MapBuilder;
14+
import org.opensearch.index.codec.CodecService;
15+
import org.opensearch.index.mapper.MapperService;
16+
17+
import java.util.ArrayList;
18+
import java.util.Arrays;
19+
import java.util.Map;
20+
21+
/**
22+
* CustomCodecService provides ZSTD and ZSTDNODICT compression codecs.
23+
*/
24+
public class CustomCodecService extends CodecService {
25+
private final Map<String, Codec> codecs;
26+
27+
/**
28+
* Creates a new CustomCodecService.
29+
*
30+
* @param mapperService A mapper service.
31+
* @param logger A logger.
32+
*/
33+
public CustomCodecService(MapperService mapperService, Logger logger) {
34+
super(mapperService, logger);
35+
final MapBuilder<String, Codec> codecs = MapBuilder.<String, Codec>newMapBuilder();
36+
if (mapperService == null) {
37+
codecs.put(Lucene95CustomCodec.Mode.ZSTD.name(), new ZstdCodec());
38+
codecs.put(Lucene95CustomCodec.Mode.ZSTDNODICT.name(), new ZstdNoDictCodec());
39+
} else {
40+
codecs.put(
41+
Lucene95CustomCodec.Mode.ZSTD.name(),
42+
new PerFieldMappingPostingFormatCodec(Lucene95CustomCodec.Mode.ZSTD, mapperService)
43+
);
44+
codecs.put(
45+
Lucene95CustomCodec.Mode.ZSTDNODICT.name(),
46+
new PerFieldMappingPostingFormatCodec(Lucene95CustomCodec.Mode.ZSTDNODICT, mapperService)
47+
);
48+
}
49+
this.codecs = codecs.immutableMap();
50+
}
51+
52+
@Override
53+
public Codec codec(String name) {
54+
Codec codec = codecs.get(name);
55+
if (codec == null) {
56+
return super.codec(name);
57+
}
58+
return codec;
59+
}
60+
61+
@Override
62+
public String[] availableCodecs() {
63+
ArrayList<String> ac = new ArrayList<String>(Arrays.asList(super.availableCodecs()));
64+
ac.addAll(codecs.keySet());
65+
return ac.toArray(new String[0]);
66+
}
67+
}

‎sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/CustomCodecServiceFactory.java‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.index.codec.customcodecs;
10+
11+
import org.opensearch.index.codec.CodecService;
12+
import org.opensearch.index.codec.CodecServiceConfig;
13+
import org.opensearch.index.codec.CodecServiceFactory;
14+
15+
/**
16+
* A factory for creating new {@link CodecService} instance
17+
*/
18+
public class CustomCodecServiceFactory implements CodecServiceFactory {
19+
20+
/** Creates a new instance. */
21+
public CustomCodecServiceFactory() {}
22+
23+
@Override
24+
public CodecService createCodecService(CodecServiceConfig config) {
25+
return new CustomCodecService(config.getMapperService(), config.getLogger());
26+
}
27+
}

‎sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/Lucene95CustomCodec.java‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.index.codec.customcodecs;
10+
11+
import org.apache.lucene.codecs.StoredFieldsFormat;
12+
import org.apache.lucene.codecs.FilterCodec;
13+
import org.apache.lucene.codecs.lucene95.Lucene95Codec;
14+
15+
abstract class Lucene95CustomCodec extends FilterCodec {
16+
public static final int DEFAULT_COMPRESSION_LEVEL = 6;
17+
18+
/** Each mode represents a compression algorithm. */
19+
public enum Mode {
20+
ZSTD,
21+
ZSTDNODICT
22+
}
23+
24+
private final StoredFieldsFormat storedFieldsFormat;
25+
26+
/** new codec for a given compression algorithm and default compression level */
27+
public Lucene95CustomCodec(Mode mode) {
28+
this(mode, DEFAULT_COMPRESSION_LEVEL);
29+
}
30+
31+
public Lucene95CustomCodec(Mode mode, int compressionLevel) {
32+
super(mode.name(), new Lucene95Codec());
33+
this.storedFieldsFormat = new Lucene95CustomStoredFieldsFormat(mode, compressionLevel);
34+
}
35+
36+
@Override
37+
public StoredFieldsFormat storedFieldsFormat() {
38+
return storedFieldsFormat;
39+
}
40+
41+
@Override
42+
public String toString() {
43+
return getClass().getSimpleName();
44+
}
45+
}

‎sandbox/modules/custom-codecs/src/main/java/org/opensearch/index/codec/customcodecs/Lucene95CustomStoredFieldsFormat.java‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
/*
2+
* SPDX-License-Identifier: Apache-2.0
3+
*
4+
* The OpenSearch Contributors require contributions made to
5+
* this file be licensed under the Apache-2.0 license or a
6+
* compatible open source license.
7+
*/
8+
9+
package org.opensearch.index.codec.customcodecs;
10+
11+
import java.io.IOException;
12+
import java.util.Objects;
13+
import org.apache.lucene.codecs.StoredFieldsFormat;
14+
import org.apache.lucene.codecs.StoredFieldsReader;
15+
import org.apache.lucene.codecs.StoredFieldsWriter;
16+
import org.apache.lucene.codecs.compressing.CompressionMode;
17+
import org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsFormat;
18+
import org.apache.lucene.index.FieldInfos;
19+
import org.apache.lucene.index.SegmentInfo;
20+
import org.apache.lucene.store.Directory;
21+
import org.apache.lucene.store.IOContext;
22+
23+
/** Stored field format used by pluggable codec */
24+
public class Lucene95CustomStoredFieldsFormat extends StoredFieldsFormat {
25+
26+
/** A key that we use to map to a mode */
27+
public static final String MODE_KEY = Lucene95CustomStoredFieldsFormat.class.getSimpleName() + ".mode";
28+
29+
private static final int ZSTD_BLOCK_LENGTH = 10 * 48 * 1024;
30+
private static final int ZSTD_MAX_DOCS_PER_BLOCK = 4096;
31+
private static final int ZSTD_BLOCK_SHIFT = 10;
32+
33+
private final CompressionMode zstdCompressionMode;
34+
private final CompressionMode zstdNoDictCompressionMode;
35+
36+
private final Lucene95CustomCodec.Mode mode;
37+
38+
/** default constructor */
39+
public Lucene95CustomStoredFieldsFormat() {
40+
this(Lucene95CustomCodec.Mode.ZSTD, Lucene95CustomCodec.DEFAULT_COMPRESSION_LEVEL);
41+
}
42+
43+
/**
44+
* Creates a new instance.
45+
*
46+
* @param mode The mode represents ZSTD or ZSTDNODICT
47+
*/
48+
public Lucene95CustomStoredFieldsFormat(Lucene95CustomCodec.Mode mode) {
49+
this(mode, Lucene95CustomCodec.DEFAULT_COMPRESSION_LEVEL);
50+
}
51+
52+
/**
53+
* Creates a new instance with the specified mode and compression level.
54+
*
55+
* @param mode The mode represents ZSTD or ZSTDNODICT
56+
* @param compressionLevel The compression level for the mode.
57+
*/
58+
public Lucene95CustomStoredFieldsFormat(Lucene95CustomCodec.Mode mode, int compressionLevel) {
59+
this.mode = Objects.requireNonNull(mode);
60+
zstdCompressionMode = new ZstdCompressionMode(compressionLevel);
61+
zstdNoDictCompressionMode = new ZstdNoDictCompressionMode(compressionLevel);
62+
}
63+
64+
@Override
65+
public StoredFieldsReader fieldsReader(Directory directory, SegmentInfo si, FieldInfos fn, IOContext context) throws IOException {
66+
String value = si.getAttribute(MODE_KEY);
67+
if (value == null) {
68+
throw new IllegalStateException("missing value for " + MODE_KEY + " for segment: " + si.name);
69+
}
70+
Lucene95CustomCodec.Mode mode = Lucene95CustomCodec.Mode.valueOf(value);
71+
return impl(mode).fieldsReader(directory, si, fn, context);
72+
}
73+
74+
@Override
75+
public StoredFieldsWriter fieldsWriter(Directory directory, SegmentInfo si, IOContext context) throws IOException {
76+
String previous = si.putAttribute(MODE_KEY, mode.name());
77+
if (previous != null && previous.equals(mode.name()) == false) {
78+
throw new IllegalStateException(
79+
"found existing value for " + MODE_KEY + " for segment: " + si.name + " old = " + previous + ", new = " + mode.name()
80+
);
81+
}
82+
return impl(mode).fieldsWriter(directory, si, context);
83+
}
84+
85+
private StoredFieldsFormat impl(Lucene95CustomCodec.Mode mode) {
86+
switch (mode) {
87+
case ZSTD:
88+
return new Lucene90CompressingStoredFieldsFormat(
89+
"CustomStoredFieldsZstd",
90+
zstdCompressionMode,
91+
ZSTD_BLOCK_LENGTH,
92+
ZSTD_MAX_DOCS_PER_BLOCK,
93+
ZSTD_BLOCK_SHIFT
94+
);
95+
case ZSTDNODICT:
96+
return new Lucene90CompressingStoredFieldsFormat(
97+
"CustomStoredFieldsZstdNoDict",
98+
zstdNoDictCompressionMode,
99+
ZSTD_BLOCK_LENGTH,
100+
ZSTD_MAX_DOCS_PER_BLOCK,
101+
ZSTD_BLOCK_SHIFT
102+
);
103+
default:
104+
throw new AssertionError();
105+
}
106+
}
107+
}

0 commit comments

Comments
 (0)