Skip to content

Add structured JSON parallel translation feature #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 9 commits into
base: master
Choose a base branch
from
Draft

Add structured JSON parallel translation feature #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
0b4b998
add feat
kargnas Aug 23, 2025
f0ef786
Fix performance issues, magic numbers, and Korean comments
claude[bot] Aug 23, 2025
4adb6d6
fix bug
kargnas Aug 23, 2025
8779662
Merge branch 'feat/json-parallel-structred' of github.com:kargnas/lar…
kargnas Aug 23, 2025
4bfc7d7
fix bug
kargnas Aug 23, 2025
2c3a916
Timeout is too short
kargnas Aug 24, 2025
12a134b
change default config
kargnas Aug 24, 2025
62d7183
compatiable with claude 4
kargnas Aug 24, 2025
83ebad8
change default. to keep the usual format. especially for json
kargnas Aug 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions config/ai-translator.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

'ai' => [
'provider' => 'anthropic',
'model' => 'claude-3-5-sonnet-latest', // Best result. Recommend for production.
'model' => 'claude-sonnet-4-20250514', // Best result. Recommend for production.
'api_key' => env('ANTHROPIC_API_KEY'),

// claude-3-haiku
Expand All @@ -35,8 +35,8 @@
// Additional options
// 'retries' => 5,
// 'max_tokens' => 4096,
// 'use_extended_thinking' => false, // Extended Thinking 기능 사용 여부 (claude-3-7-sonnet-latest 모델만 지원)
// 'disable_stream' => true, // Disable streaming mode for better error messages
'use_extended_thinking' => true, // Extended Thinking 기능 사용 여부 (claude-3-7-sonnet-latest 모델만 지원)
Copy link

@greptile-apps greptile-apps bot Oct 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: use_extended_thinking을 사용하려면 claude-3-7-sonnet 모델이 필요하지만, 12번 라인에서 claude-sonnet-4-20250514로 설정되어 있습니다. 이 옵션은 현재 모델에서는 작동하지 않습니다.

Suggested change
'use_extended_thinking' => true, // Extended Thinking 기능 사용 여부 (claude-3-7-sonnet-latest 모델만 지원)
// 'use_extended_thinking' => true, // Extended Thinking 기능 사용 여부 (claude-3-7-sonnet-latest 모델만 지원)

'disable_stream' => true, // Disable streaming mode for better error messages

// 'prompt_custom_system_file_path' => null, // Full path to your own custom prompt-system.txt - i.e. resource_path('prompt-system.txt')
// 'prompt_custom_user_file_path' => null, // Full path to your own custom prompt-user.txt - i.e. resource_path('prompt-user.txt')
Expand All @@ -47,7 +47,7 @@
// 'skip_files' => [],

// If set to true, translations will be saved as flat arrays using dot notation keys. If set to false, translations will be saved as multi-dimensional arrays.
'dot_notation' => true,
'dot_notation' => false,

// You can add additional custom locale names here.
// Example: 'en_us', 'en-us', 'en_US', 'en-US'
Expand Down
131 changes: 98 additions & 33 deletions src/AI/AIProvider.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ class AIProvider

protected int $totalTokens = 0;

protected int $cacheCreationTokens = 0;

protected int $cacheReadTokens = 0;

// Callback properties
protected $onTranslated = null;

Expand Down Expand Up @@ -172,34 +176,74 @@ protected function getSystemPrompt($replaces = [])
Log::debug("AIProvider: Using translation context - {$contextFileCount} files, {$contextItemCount} items");

$translationContext = collect($this->globalTranslationContext)->map(function ($translations, $file) {
// Remove .php extension from filename
$rootKey = pathinfo($file, PATHINFO_FILENAME);
$itemCount = count($translations);

Log::debug("AIProvider: Including context file - {$rootKey}: {$itemCount} items");
// Handle both PHP and JSON context structures
$isJsonContext = isset($translations['source']) && isset($translations['target']);

$translationsText = collect($translations)->map(function ($item, $key) use ($rootKey) {
$sourceText = $item['source'] ?? '';
if ($isJsonContext) {
// JSON context structure from JSONTranslationContextProvider
$sourceStrings = $translations['source'];
$targetStrings = $translations['target'];

if (empty($sourceText)) {
if (empty($sourceStrings)) {
return null;
}

$text = "`{$rootKey}.{$key}`: src=\"\"\"{$sourceText}\"\"\"";
$itemCount = count($sourceStrings);
Log::debug("AIProvider: Including context file - {$file}: {$itemCount} items");

// Check reference information
$referenceKey = $key;
foreach ($this->references as $locale => $strings) {
if (isset($strings[$referenceKey]) && ! empty($strings[$referenceKey])) {
$text .= "\n {$locale}=\"\"\"{$strings[$referenceKey]}\"\"\"";
$translationsText = collect($sourceStrings)->map(function ($sourceText, $key) use ($targetStrings) {
if (empty($sourceText)) {
return null;
}
}

return $text;
})->filter()->implode("\n");
$text = " <item>\n";
$text .= " <key>{$key}</key>\n";
$text .= " <source><![CDATA[{$sourceText}]]></source>\n";

if (isset($targetStrings[$key]) && ! empty($targetStrings[$key])) {
$text .= " <target><![CDATA[{$targetStrings[$key]}]]></target>\n";
}

$text .= ' </item>';

return $text;
})->filter()->implode("\n");

return empty($translationsText) ? '' : " <file name=\"{$file}\">\n{$translationsText}\n </file>";
} else {
// PHP context structure from TranslationContextProvider
$rootKey = pathinfo($file, PATHINFO_FILENAME);
$itemCount = count($translations);
Log::debug("AIProvider: Including context file - {$rootKey}: {$itemCount} items");

$translationsText = collect($translations)->map(function ($item, $key) use ($rootKey) {
$sourceText = $item['source'] ?? '';

if (empty($sourceText)) {
return null;
}

$text = " <item>\n";
$text .= " <key>{$rootKey}.{$key}</key>\n";
$text .= " <source><![CDATA[{$sourceText}]]></source>\n";

if (isset($item['target']) && ! empty($item['target'])) {
$text .= " <target><![CDATA[{$item['target']}]]></target>\n";
}

$text .= ' </item>';

return empty($translationsText) ? '' : "## `{$rootKey}`\n{$translationsText}";
})->filter()->implode("\n\n");
return $text;
})->filter()->implode("\n");

return empty($translationsText) ? '' : " <file name=\"{$rootKey}\">\n{$translationsText}\n </file>";
}
})->filter()->implode("\n");

// Wrap in global_context XML tags
if (! empty($translationContext)) {
$translationContext = "<global_context>\n{$translationContext}\n</global_context>";
}

$contextLength = strlen($translationContext);
Log::debug("AIProvider: Generated context size - {$contextLength} bytes");
Expand Down Expand Up @@ -241,16 +285,21 @@ protected function getUserPrompt($replaces = [])
'parentKey' => pathinfo($this->filename, PATHINFO_FILENAME),
'keys' => collect($this->strings)->keys()->implode('`, `'),
'strings' => collect($this->strings)->map(function ($string, $key) {
if (is_string($string)) {
return " - `{$key}`: \"\"\"{$string}\"\"\"";
} else {
$text = " - `{$key}`: \"\"\"{$string['text']}\"\"\"";
$text = " <string>\n";
$text .= " <key>{$key}</key>\n";

if (\is_array($string)) {
$text .= " <source><![CDATA[{$string['text']}]]></source>\n";
if (isset($string['context'])) {
$text .= "\n - Context: \"\"\"{$string['context']}\"\"\"";
$text .= " <context><![CDATA[{$string['context']}]]></context>\n";
}

return $text;
} else {
$text .= " <source><![CDATA[{$string}]]></source>\n";
}

$text .= ' </string>';

return $text;
})->implode("\n"),
]);

Expand Down Expand Up @@ -521,9 +570,6 @@ protected function getTranslatedObjectsFromAnthropic(): array
// Prepare request data
$requestData = [
'model' => $this->configModel,
'messages' => [
['role' => 'user', 'content' => $this->getUserPrompt()],
],
'system' => [
[
'type' => 'text',
Expand All @@ -533,19 +579,22 @@ protected function getTranslatedObjectsFromAnthropic(): array
],
],
],
'messages' => [
['role' => 'user', 'content' => $this->getUserPrompt()],
],
];

$defaultMaxTokens = 4096;

if (preg_match('/^claude\-3\-5\-/', $this->configModel)) {
$defaultMaxTokens = 8192;
} elseif (preg_match('/^claude\-3\-7\-/', $this->configModel)) {
} elseif (preg_match('/^claude.*(3\-7|4)/', $this->configModel)) {
// @TODO: if add betas=["output-128k-2025-02-19"], then 128000
$defaultMaxTokens = 64000;
}

// Set up Extended Thinking
if ($useExtendedThinking && preg_match('/^claude\-3\-7\-/', $this->configModel)) {
if ($useExtendedThinking && preg_match('/^claude.*(3\-7|4)/', $this->configModel)) {
$requestData['thinking'] = [
'type' => 'enabled',
'budget_tokens' => 10000,
Expand Down Expand Up @@ -745,6 +794,14 @@ function ($chunk, $data) use (&$responseText, $responseParser, &$inThinkingBlock
$this->totalTokens = $this->inputTokens + $this->outputTokens;
}

// 캐시 토큰 추적
if (isset($response['cache_creation_input_tokens'])) {
$this->cacheCreationTokens = (int) $response['cache_creation_input_tokens'];
}
if (isset($response['cache_read_input_tokens'])) {
$this->cacheReadTokens = (int) $response['cache_read_input_tokens'];
}

$responseText = $response['content'][0]['text'];
$responseParser->parse($responseText);

Expand Down Expand Up @@ -814,8 +871,8 @@ public function getTokenUsage(): array
return [
'input_tokens' => $this->inputTokens,
'output_tokens' => $this->outputTokens,
'cache_creation_input_tokens' => null,
'cache_read_input_tokens' => null,
'cache_creation_input_tokens' => $this->cacheCreationTokens,
'cache_read_input_tokens' => $this->cacheReadTokens,
'total_tokens' => $this->totalTokens,
];
}
Expand Down Expand Up @@ -863,6 +920,14 @@ protected function trackTokenUsage(array $data): void
$this->extractTokensFromUsage($data['message']['usage']);
}

// 캐시 토큰 정보 추출
if (isset($data['message']['cache_creation_input_tokens'])) {
$this->cacheCreationTokens = (int) $data['message']['cache_creation_input_tokens'];
}
if (isset($data['message']['cache_read_input_tokens'])) {
$this->cacheReadTokens = (int) $data['message']['cache_read_input_tokens'];
}

// 유형 3: message.content_policy.input_tokens, output_tokens가 있는 경우
if (isset($data['message']['content_policy'])) {
if (isset($data['message']['content_policy']['input_tokens'])) {
Expand Down
9 changes: 7 additions & 2 deletions src/AI/Clients/AnthropicClient.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,13 @@ public function messages()
*/
public function request(string $method, string $endpoint, array $data = []): array
{
$timeout = 1800; // 30 minutes

$response = Http::withHeaders([
'x-api-key' => $this->apiKey,
'anthropic-version' => $this->apiVersion,
'content-type' => 'application/json',
])->$method("{$this->baseUrl}/{$endpoint}", $data);
])->timeout($timeout)->$method("{$this->baseUrl}/{$endpoint}", $data);

if (! $response->successful()) {
$statusCode = $response->status();
Expand Down Expand Up @@ -283,6 +285,9 @@ public function requestStream(string $method, string $endpoint, array $data, cal
'accept: application/json',
];

// Set timeout to 30 minutes for streaming
$timeout = 1800; // 30 minutes
Copy link

@greptile-apps greptile-apps bot Oct 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: 일관성을 위해 중복된 주석과 변수 선언을 제거하는 것을 고려하세요. 44번 라인과 289번 라인이 동일한 값입니다


// Initialize cURL
$ch = curl_init();

Expand All @@ -291,7 +296,7 @@ public function requestStream(string $method, string $endpoint, array $data, cal
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, strtoupper($method));
curl_setopt($ch, CURLOPT_TIMEOUT, 300);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);

if (strtoupper($method) !== 'GET') {
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data));
Expand Down
12 changes: 11 additions & 1 deletion src/AI/Clients/GeminiClient.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,17 @@ class GeminiClient
public function __construct(string $apiKey)
{
$this->apiKey = $apiKey;
$this->client = \Gemini::client($apiKey);

// Set timeout to 30 minutes
$timeout = 1800; // 30 minutes

// Create client with timeout configuration
$this->client = \Gemini::factory()
->withApiKey($apiKey)
->withHttpClient(new \GuzzleHttp\Client([
'timeout' => $timeout,
]))
->make();
}

public function request(string $model, array $contents): array
Expand Down
10 changes: 7 additions & 3 deletions src/AI/Clients/OpenAIClient.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,12 @@ public function __construct(string $apiKey)
*/
public function request(string $method, string $endpoint, array $data = []): array
{
$timeout = 1800; // 30 minutes

$response = Http::withHeaders([
'Authorization' => 'Bearer '.$this->apiKey,
'Content-Type' => 'application/json',
])->$method("{$this->baseUrl}/{$endpoint}", $data);
])->timeout($timeout)->$method("{$this->baseUrl}/{$endpoint}", $data);

if (! $response->successful()) {
throw new \Exception("OpenAI API error: {$response->body()}");
Expand Down Expand Up @@ -173,8 +175,10 @@ public function requestStream(string $method, string $endpoint, array $data, cal
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, strtoupper($method));
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_TIMEOUT, 120);

$timeout = 1800; // 30 minutes
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30); // Keep connection timeout at 30 seconds
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
Comment on lines +179 to +181
Copy link

@greptile-apps greptile-apps bot Oct 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: 타임아웃 값을 request() 메서드와 동일하게 설정했어요. 두 메서드 간 일관성을 위해 클래스 상수나 설정 값으로 추출하는 것을 고려해보세요.


if (strtoupper($method) !== 'GET') {
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data));
Expand Down
Loading