2022-05-13 Databend Storage Work Reports

[Xuanwo]

Hi, every databenders! Please allow me to present the recent work and some plans for the community members.

In recent weeks, we are focusing on implementing the following shining features:

• Feature: multi-catalog #4947

the original idea is

• Hive as an External Data Source #4826

later, we found that to be more user friendly and extensible, databend need to support multiple catalogs.

currently:

• catalog of hive meta store can be mapped into databend catalog
• Provides a skeleton(or naive:) implementation of hive database and hive table engine
  • which can fetch and translate meta data from hive meta store on demond

This PR is the first working demo, and there are many works to do. Welcome to taking part in this feature via commenting here!

@Xuanwo is working on integrating new OpenDAL features into databend:

• Tracking issues of adopting features added in OpenDAL v0.5 #4736
• Tracking issues of adopting features added in OpenDAL v0.6 #5215

Including:

• Support copying from azblob and hdfs (use as an external stage)
• Support read compressed files like xxx.csv.gz and yyy.csv.zstd

However, @Xuanwo is blocked by config backward compatibility problems, so he proposes a new RFC to address that:

• RFC: Config Backward Compatibility #5324

---

This is the work report of Databend Storage. Any comments are welcome!