refactor: logs

.typos.toml

@@ -15,5 +15,6 @@ flate = "flate"
 [files]
 extend-exclude = [
   "pnpm-lock.yaml",
-  "*/**/df-functions.md"
+  "*/**/df-functions.md",
+  "**/*.svg"
 ]

docs/faq-and-others/faq.md

@@ -219,7 +219,7 @@ Learn more about indexing: [Index Management](/user-guide/manage-data/data-index
 
 **Real-Time Processing**:
 - **[Flow Engine](/user-guide/flow-computation/overview.md)**: Real-time stream processing system that enables continuous, incremental computation on streaming data with automatic result table updates
-- **[Pipeline](/user-guide/logs/pipeline-config.md)**: Data parsing and transformation mechanism for processing incoming data in real-time, with configurable processors for field extraction and data type conversion across multiple data formats
+- **[Pipeline](/reference/pipeline/pipeline-config.md)**: Data parsing and transformation mechanism for processing incoming data in real-time, with configurable processors for field extraction and data type conversion across multiple data formats
 - **Output Tables**: Persist processed results for analysis
 
 

docs/getting-started/quick-start.md

@@ -237,7 +237,7 @@ ORDER BY
 +---------------------+-------+------------------+-----------+--------------------+
 ```
 
-The `@@` operator is used for [term searching](/user-guide/logs/query-logs.md).
+The `@@` operator is used for [term searching](/user-guide/logs/fulltext-search.md).
 
 ### Range query
 

docs/greptimecloud/integrations/fluent-bit.md

@@ -28,7 +28,7 @@ Fluent Bit can be configured to send logs to GreptimeCloud using the HTTP protoc
     http_Passwd      <password>
 ```
 
-In this example, the `http` output plugin is used to send logs to GreptimeCloud. For more information, and extra options, refer to the [Logs HTTP API](https://docs.greptime.com/user-guide/logs/write-logs#http-api) guide.
+In this example, the `http` output plugin is used to send logs to GreptimeCloud. For more information, and extra options, refer to the [Logs HTTP API](https://docs.greptime.com/reference/pipeline/write-log-api/#http-api) guide.
 
 ## Prometheus Remote Write
 

docs/greptimecloud/integrations/kafka.md

@@ -13,7 +13,7 @@ Here we are using Vector as the tool to transport data from Kafka to GreptimeDB.
 ## Logs
 
 A sample configuration. Note that you will need to [create your
-pipeline](https://docs.greptime.com/user-guide/logs/pipeline-config/) for log
+pipeline](https://docs.greptime.com/user-guide/logs/use-custom-pipelines/) for log
 parsing.
 
 ```toml

docs/reference/pipeline/built-in-pipelines.md

@@ -0,0 +1,176 @@
+---
+keywords: [built-in pipelines, greptime_identity, JSON logs, log processing, time index, pipeline, GreptimeDB]
+description: Learn about GreptimeDB's built-in pipelines, including the greptime_identity pipeline for processing JSON logs with automatic schema creation, type conversion, and time index configuration.
+---
+
+# Built-in Pipelines
+
+GreptimeDB offers built-in pipelines for common log formats, allowing you to use them directly without creating new pipelines.
+
+Note that the built-in pipelines are not editable.
+Additionally, the "greptime_" prefix of the pipeline name is reserved.
+
+## `greptime_identity`
+
+The `greptime_identity` pipeline is designed for writing JSON logs and automatically creates columns for each field in the JSON log.
+
+- The first-level keys in the JSON log are used as column names.
+- An error is returned if the same field has different types.
+- Fields with `null` values are ignored.
+- If time index is not specified, an additional column, `greptime_timestamp`, is added to the table as the time index to indicate when the log was written.
+
+### Type conversion rules
+
+- `string` -> `string`
+- `number` -> `int64` or `float64`
+- `boolean` -> `bool`
+- `null` -> ignore
+- `array` -> `json`
+- `object` -> `json`
+
+
+For example, if we have the following json data:
+
+```json
+[
+    {"name": "Alice", "age": 20, "is_student": true, "score": 90.5,"object": {"a":1,"b":2}},
+    {"age": 21, "is_student": false, "score": 85.5, "company": "A" ,"whatever": null},
+    {"name": "Charlie", "age": 22, "is_student": true, "score": 95.5,"array":[1,2,3]}
+]
+```
+
+We'll merge the schema for each row of this batch to get the final schema. The table schema will be:
+
+```sql
+mysql> desc pipeline_logs;
++--------------------+---------------------+------+------+---------+---------------+
+| Column             | Type                | Key  | Null | Default | Semantic Type |
++--------------------+---------------------+------+------+---------+---------------+
+| age                | Int64               |      | YES  |         | FIELD         |
+| is_student         | Boolean             |      | YES  |         | FIELD         |
+| name               | String              |      | YES  |         | FIELD         |
+| object             | Json                |      | YES  |         | FIELD         |
+| score              | Float64             |      | YES  |         | FIELD         |
+| company            | String              |      | YES  |         | FIELD         |
+| array              | Json                |      | YES  |         | FIELD         |
+| greptime_timestamp | TimestampNanosecond | PRI  | NO   |         | TIMESTAMP     |
++--------------------+---------------------+------+------+---------+---------------+
+8 rows in set (0.00 sec)
+```
+
+The data will be stored in the table as follows:
+
+```sql
+mysql> select * from pipeline_logs;
++------+------------+---------+---------------+-------+---------+---------+----------------------------+
+| age  | is_student | name    | object        | score | company | array   | greptime_timestamp         |
++------+------------+---------+---------------+-------+---------+---------+----------------------------+
+|   22 |          1 | Charlie | NULL          |  95.5 | NULL    | [1,2,3] | 2024-10-18 09:35:48.333020 |
+|   21 |          0 | NULL    | NULL          |  85.5 | A       | NULL    | 2024-10-18 09:35:48.333020 |
+|   20 |          1 | Alice   | {"a":1,"b":2} |  90.5 | NULL    | NULL    | 2024-10-18 09:35:48.333020 |
++------+------------+---------+---------------+-------+---------+---------+----------------------------+
+3 rows in set (0.01 sec)
+```
+
+### Specify time index
+
+A time index is necessary in GreptimeDB. Since the `greptime_identity` pipeline does not require a YAML configuration, you must set the time index in the query parameters if you want to use the timestamp from the log data instead of the automatically generated timestamp when the data arrives.
+
+Example of Incoming Log Data:
+```JSON
+[
+    {"action": "login", "ts": 1742814853}
+]
+```
+
+To instruct the server to use ts as the time index, set the following query parameter in the HTTP header:
+```shell
+curl -X "POST" "http://localhost:4000/v1/ingest?db=public&table=pipeline_logs&pipeline_name=greptime_identity&custom_time_index=ts;epoch;s" \
+     -H "Content-Type: application/json" \
+     -H "Authorization: Basic {{authentication}}" \
+     -d $'[{"action": "login", "ts": 1742814853}]'
+```
+
+The `custom_time_index` parameter accepts two formats, depending on the input data format:
+- Epoch number format: `<field_name>;epoch;<resolution>`
+  - The field can be an integer or a string.
+  - The resolution must be one of: `s`, `ms`, `us`, or `ns`.
+- Date string format: `<field_name>;datestr;<format>`
+  - For example, if the input data contains a timestamp like `2025-03-24 19:31:37+08:00`, the corresponding format should be `%Y-%m-%d %H:%M:%S%:z`.
+
+With the configuration above, the resulting table will correctly use the specified log data field as the time index.
+```sql
+DESC pipeline_logs;
+```
+```sql
++--------+-----------------+------+------+---------+---------------+
+| Column | Type            | Key  | Null | Default | Semantic Type |
++--------+-----------------+------+------+---------+---------------+
+| ts     | TimestampSecond | PRI  | NO   |         | TIMESTAMP     |
+| action | String          |      | YES  |         | FIELD         |
++--------+-----------------+------+------+---------+---------------+
+2 rows in set (0.02 sec)
+```
+
+Here are some example of using `custom_time_index` assuming the time variable is named `input_ts`:
+- 1742814853: `custom_time_index=input_ts;epoch;s`
+- 1752749137000: `custom_time_index=input_ts;epoch;ms`
+- "2025-07-17T10:00:00+0800": `custom_time_index=input_ts;datestr;%Y-%m-%dT%H:%M:%S%z`
+- "2025-06-27T15:02:23.082253908Z": `custom_time_index=input_ts;datestr;%Y-%m-%dT%H:%M:%S%.9f%#z`
+
+
+### Flatten JSON objects
+
+If flattening a JSON object into a single-level structure is needed, add the `x-greptime-pipeline-params` header to the request and set `flatten_json_object` to `true`.
+
+Here is a sample request:
+
+```shell
+curl -X "POST" "http://localhost:4000/v1/ingest?db=<db-name>&table=<table-name>&pipeline_name=greptime_identity&version=<pipeline-version>" \
+     -H "Content-Type: application/x-ndjson" \
+     -H "Authorization: Basic {{authentication}}" \
+     -H "x-greptime-pipeline-params: flatten_json_object=true" \
+     -d "$<log-items>"
+```
+
+With this configuration, GreptimeDB will automatically flatten each field of the JSON object into separate columns. For example:
+
+```JSON
+{
+    "a": {
+        "b": {
+            "c": [1, 2, 3]
+        }
+    },
+    "d": [
+        "foo",
+        "bar"
+    ],
+    "e": {
+        "f": [7, 8, 9],
+        "g": {
+            "h": 123,
+            "i": "hello",
+            "j": {
+                "k": true
+            }
+        }
+    }
+}
+```
+
+Will be flattened to:
+
+```json
+{
+    "a.b.c": [1,2,3],
+    "d": ["foo","bar"],
+    "e.f": [7,8,9],
+    "e.g.h": 123,
+    "e.g.i": "hello",
+    "e.g.j.k": true
+}
+```
+
+
+

docs/user-guide/logs/pipeline-config.md → docs/reference/pipeline/pipeline-config.md

@@ -51,10 +51,10 @@ The above plain text data will be converted to the following equivalent form:
 
 In other words, when the input is in plain text format, you need to use `message` to refer to the content of each line when writing `Processor` and `Transform` configurations.
 
-## Overall structure
+## Pipeline Configuration Structure
 
 Pipeline consists of four parts: Processors, Dispatcher, Transform, and Table suffix.
-Processors pre-processes input log data.
+Processors pre-process input log data.
 Dispatcher forwards pipeline execution context onto different subsequent pipeline.
 Transform decides the final datatype and table structure in the database.
 Table suffix allows storing the data into different tables.
@@ -827,6 +827,8 @@ Some notes regarding the `vrl` processor:
 2. The returning value of the vrl script should not contain any regex-type variables. They can be used in the script, but have to be `del`ed before returning.
 3. Due to type conversion between pipeline's value type and vrl's, the value type that comes out of the vrl script will be the ones with max capacity, meaning `i64`, `f64`, and `Timestamp::nanoseconds`.
 
+You can use `vrl` processor to set [table options](./write-log-api.md#set-table-options) while writing logs.
+
 ### `filter`
 
 The `filter` processor can filter out unneeded lines when the condition is meet.
@@ -1013,7 +1015,7 @@ Specify which field uses the inverted index. Refer to the [Transform Example](#t
 
 #### The Fulltext Index
 
-Specify which field will be used for full-text search using `index: fulltext`. This index greatly improves the performance of [log search](./query-logs.md). Refer to the [Transform Example](#transform-example) below for syntax.
+Specify which field will be used for full-text search using `index: fulltext`. This index greatly improves the performance of [log search](/user-guide/logs/fulltext-search.md). Refer to the [Transform Example](#transform-example) below for syntax.
 
 #### The Skipping Index
 
@@ -1159,4 +1161,4 @@ table_suffix: _${type}
 These three lines of input log will be inserted into three tables:
 1. `persist_app_db`
 2. `persist_app_http`
-3. `persist_app`, for it doesn't have a `type` field, thus the default table name will be used.
+3. `persist_app`, for it doesn't have a `type` field, thus the default table name will be used.