Stage 0: SAP(ZAA_ICF)에서 패키지 단위로 프로그램 소스 수집 (ingest/from_sap.py)

- sap/: abap-mcp 의 catalog/sap_client/mcp_server 복사. import 와 .env 탐색만 이 저장소에 맞춤
- ingest/from_sap.py: 패키지명 → GET_PROGRAM_LIST → GET_PROGRAM_SOURCE(+Include) → data/raw/*.txt
  (normalize 가 읽는 수집 JSON 형식 그대로). 받은 파일은 건너뛰고 --force 로 재수집
- normalize: 응답의 TCODE_LIST 를 tcodes.jsonl 로 (from_dir 와 같은 모양, 로더가 적재)
- .env.example 에 SAP_URL/SAP_USER/SAP_PASS, tests/test_from_sap.py

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
byeongwook.choi
2026-09-21 14:13:14 +09:00
co-authored by Claude Fable 5.1
parent 11ae3629b2
commit 921b8ce229
13 changed files with 1595 additions and 1 deletions
+10
View File
@@ -95,6 +95,7 @@ def run(raw_dir: Path, out_dir: Path, limit: int | None = None, dry_run: bool =
stats = {"files": 0, "programs": 0, "includes": 0, "package_rows": 0, "total_lines": 0, "errors": 0}
package_rows: list[dict] = []
tcode_rows: list[dict] = [] # ingest/from_sap.py 가 실어 보내는 TCODE_LIST → tcodes.jsonl
error_lines: list[str] = []
files = sorted(p for p in raw_dir.rglob("*") if p.is_file() and p.suffix.lower() in {".txt", ".json"})
@@ -132,6 +133,11 @@ def run(raw_dir: Path, out_dir: Path, limit: int | None = None, dry_run: bool =
stats["programs"] += 1
stats["includes"] += len(meta["includes"])
stats["total_lines"] += meta["total_lines"]
for t in data.get("TCODE_LIST") or []:
tcode = clean_text_field(t.get("TCODE")).upper()
if tcode:
tcode_rows.append({"tcode": tcode, "program": meta["program"],
"text_ko": clean_text_field(t.get("TTEXT")) or meta["description"]})
else:
stats["errors"] += 1
error_lines.append(f"{path}\tUNKNOWN_FORMAT")
@@ -141,6 +147,10 @@ def run(raw_dir: Path, out_dir: Path, limit: int | None = None, dry_run: bool =
with packages_path.open("w", encoding="utf-8") as f:
for row in package_rows:
f.write(json.dumps(row, ensure_ascii=False) + "\n")
if tcode_rows:
# from_dir.py 와 같은 파일·모양. 로더(index/loader.py load_tcodes)가 tcode 테이블에 넣는다
(out_dir / "tcodes.jsonl").write_text(
"".join(json.dumps(r, ensure_ascii=False) + "\n" for r in tcode_rows), encoding="utf-8")
if error_lines:
errors_log.write_text("\n".join(error_lines) + "\n", encoding="utf-8")