Compare commits
10 Commits
fa33bee3e1
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
440b74d09a | ||
|
|
e7bbbbc194 | ||
|
|
c3bbc919a5 | ||
|
|
1b984f5cfd | ||
|
|
da567a2679 | ||
|
|
bbf335c376 | ||
|
|
6a8500133b | ||
|
|
8d59fc6ef8 | ||
|
|
f0d21e62c9 | ||
|
|
8048f2e669 |
17
.env.example
17
.env.example
@@ -1,7 +1,12 @@
|
|||||||
# Playwright 浏览器路径
|
# Playwright Configuration
|
||||||
PLAYWRIGHT_BROWSERS_PATH=C:\Users\Administrator\AppData\Roaming\erpauto\ms-playwright
|
PLAYWRIGHT_BROWSERS_PATH=path/to/playwright/browsers
|
||||||
|
|
||||||
# 用友BIP登录配置
|
# ERP System Configuration
|
||||||
BIP_URL=https://68.11.34.30:8082/yonbip/resources/uap/rbac/login/main/index.html
|
ERP_URL=https://your-erp-system.com
|
||||||
BIP_USERNAME=your_username
|
ERP_USERNAME=your_username
|
||||||
BIP_PASSWORD=your_password
|
ERP_PASSWORD=your_password
|
||||||
|
|
||||||
|
# Browser Behavior
|
||||||
|
ERP_HEADLESS=false
|
||||||
|
ERP_IGNORE_HTTPS_ERRORS=true
|
||||||
|
ERP_AUTO_CLOSE_BROWSER=true
|
||||||
|
|||||||
5
.gitignore
vendored
5
.gitignore
vendored
@@ -148,3 +148,8 @@ Thumbs.db
|
|||||||
.agents/
|
.agents/
|
||||||
.agent/
|
.agent/
|
||||||
.claude/
|
.claude/
|
||||||
|
.sisyphus
|
||||||
|
|
||||||
|
data/
|
||||||
|
|
||||||
|
nul
|
||||||
89
CLAUDE.md
89
CLAUDE.md
@@ -32,18 +32,46 @@ The project relies heavily on environment variables loaded from `.env` file in t
|
|||||||
- `ERP_PASSWORD` - Login password
|
- `ERP_PASSWORD` - Login password
|
||||||
- `ERP_HEADLESS` - Whether to run browser in headless mode (true/false)
|
- `ERP_HEADLESS` - Whether to run browser in headless mode (true/false)
|
||||||
- `ERP_IGNORE_HTTPS_ERRORS` - Whether to ignore HTTPS certificate errors (true/false)
|
- `ERP_IGNORE_HTTPS_ERRORS` - Whether to ignore HTTPS certificate errors (true/false)
|
||||||
- `ERP_AUTO_CLOSE_BROWSER` - Whether to automatically close browser after operations (true/false)
|
|
||||||
|
Note: Test scripts are responsible for loading environment variables and passing configuration to utility functions.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
### Module Structure
|
### Module Structure
|
||||||
|
|
||||||
**`utils/auth.py`** - Core authentication module
|
**`utils/auth.py`** - Core authentication module (pure functions)
|
||||||
- `login()` - Handles Yonyou BIP login with automatic force-login popup detection
|
- `login(playwright, username, password, url, headless, ignore_https_errors, verbose=True)` - Handles Yonyou BIP login with automatic force-login popup detection. Requires all parameters (playwright, username, password, url, headless, ignore_https_errors).
|
||||||
- `logout()` - Performs logout with confirmation dialog handling
|
- `logout(page)` - Performs logout with confirmation dialog handling.
|
||||||
- `close_session()` - Closes browser session with respect to auto-close configuration
|
- Returns tuple: `(browser, context, page, main_frame)` where `main_frame` is the forwardFrame iframe.
|
||||||
- Auto-loads environment variables from `.env` on module import
|
- Callers are responsible for browser lifecycle management (context.close(), browser.close()).
|
||||||
- Returns tuple: `(browser, context, page, main_frame)` where `main_frame` is the forwardFrame iframe
|
|
||||||
|
**`utils/discrete_material_plan/extractor_core.py`** - Core web operations module (pure functions)
|
||||||
|
- `navigate_to_discrete_material_page(main_frame, page)` - Navigate to discrete material plan maintenance page. Returns `(work_frame, page1)`.
|
||||||
|
- `setup_query_interface(work_frame)` - Initialize query interface by selecting order number query tab and setting page size.
|
||||||
|
- `fill_and_search_orders(work_frame, order_ids)` - Fill order IDs into search textbox and trigger search.
|
||||||
|
- `download_batch_data(work_frame, page, order_ids, batch_index, download_dir)` - Execute download workflow for a single batch. Returns downloaded file path.
|
||||||
|
- `execute_batch_download_workflow(work_frame, page, order_ids, batch_index, download_dir)` - Complete workflow: fill orders, search, and download. Returns downloaded file path.
|
||||||
|
- All functions are stateless and accept required parameters explicitly. No logging or progress reporting - callers handle that.
|
||||||
|
- **See**: `docs/discrete_material_plan_extractor_core.md` for complete API documentation and usage examples.
|
||||||
|
|
||||||
|
**`utils/discrete_material_plan/extractor.py`** - High-level extractor component
|
||||||
|
- `DiscreteMaterialPlanExtractor` class - Batch processing wrapper with progress reporting and error handling.
|
||||||
|
- `extract_from_file()` function - Convenience function to extract data from order IDs in a file.
|
||||||
|
- **New Methods**:
|
||||||
|
- `post_process_downloads()` - Convert and merge downloaded Excel files into structured format
|
||||||
|
- `extract_and_process()` - Complete workflow: extract data and post-process in one call
|
||||||
|
- Accepts ID list and file paths as parameters, handles session management, batch processing, Excel conversion, and file merging.
|
||||||
|
- **Documentation**:
|
||||||
|
- Basic Usage: `docs/discrete_material_plan_extractor.md` - Basic API and usage examples
|
||||||
|
- Component Guide: `docs/extractor_component_guide.md` - Complete guide with Mermaid diagrams
|
||||||
|
- Post-Processing: `docs/extractor_post_processing.md` - Excel conversion and merging features
|
||||||
|
|
||||||
|
**`utils/discrete_material_plan/excel_converter.py`** - Excel data conversion utility
|
||||||
|
- `ExcelConverter` class - Converts raw Excel reports to structured DataFrames.
|
||||||
|
- `convert(input_file, output_file)` - Converts Excel file and returns DataFrame.
|
||||||
|
- Handles nested order structures, flattens to table format, applies field name mapping.
|
||||||
|
- Used internally by extractor for post-processing, can also be used standalone.
|
||||||
|
- **See**: `docs/extractor_post_processing.md` for usage examples.
|
||||||
|
|
||||||
**`tests/`** - Test suite
|
**`tests/`** - Test suite
|
||||||
- All test files must add `PROJECT_ROOT` to `sys.path` to import `utils` modules
|
- All test files must add `PROJECT_ROOT` to `sys.path` to import `utils` modules
|
||||||
@@ -64,6 +92,21 @@ Yonyou BIP uses a nested iframe structure:
|
|||||||
|
|
||||||
The system automatically detects and handles a force-login confirmation dialog that appears after clicking the login button. The code checks for a "确定" (Confirm) button and clicks it if present.
|
The system automatically detects and handles a force-login confirmation dialog that appears after clicking the login button. The code checks for a "确定" (Confirm) button and clicks it if present.
|
||||||
|
|
||||||
|
### URL Construction Pattern
|
||||||
|
|
||||||
|
Callers must construct the complete login URL before passing to `login()`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
url = f"{os.getenv('ERP_URL').rstrip('/')}/yonbip/resources/uap/rbac/login/main/index.html"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Demo Data
|
||||||
|
|
||||||
|
**`id-demo.txt`** - Demo production IDs for testing
|
||||||
|
- Contains 15 sample production IDs (SC70202603240xxx format)
|
||||||
|
- Used by test scripts and demos for web operations testing
|
||||||
|
- Format: One ID per line, plain text file
|
||||||
|
|
||||||
## Common Commands
|
## Common Commands
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -73,15 +116,45 @@ python tests/test_auth_config.py
|
|||||||
# Run login/logout test
|
# Run login/logout test
|
||||||
python tests/test_login.py
|
python tests/test_login.py
|
||||||
|
|
||||||
|
# Run web operations tests
|
||||||
|
python tests/test_web_operations.py
|
||||||
|
|
||||||
|
# Run extractor component tests
|
||||||
|
python tests/test_extractor_component.py
|
||||||
|
|
||||||
|
# Run real data extraction test (uses tests/id-demo.txt)
|
||||||
|
python tests/test_extractor_real.py
|
||||||
|
|
||||||
|
# Or use quick run scripts
|
||||||
|
./run_extractor_test.bat # Windows
|
||||||
|
./run_extractor_test.sh # Linux/Mac
|
||||||
|
|
||||||
|
# Run web operations demo
|
||||||
|
python tests/demo_web_operations.py
|
||||||
|
|
||||||
|
# Run extractor component demo
|
||||||
|
python demo_extractor_component.py
|
||||||
|
|
||||||
# Run any test with virtual environment
|
# Run any test with virtual environment
|
||||||
source .venv/Scripts/activate && python tests/<test_file>.py
|
source .venv/Scripts/activate && python tests/<test_file>.py
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
**`docs/INDEX.md`** - Complete documentation index
|
||||||
|
- Quick start guide, API reference, examples, and troubleshooting
|
||||||
|
- Central hub for all documentation
|
||||||
|
|
||||||
|
**`docs/extractor_component_guide.md`** - Complete component guide with Mermaid diagrams
|
||||||
|
- Architecture overview, class structure, workflow diagrams
|
||||||
|
- State management, error handling, integration examples
|
||||||
|
- Visual documentation using Mermaid graphs
|
||||||
|
|
||||||
## Code Conventions
|
## Code Conventions
|
||||||
|
|
||||||
1. **English Only**: All user-facing output, docstrings, comments, and log messages must be in English. Chinese text is only used for Playwright element selectors matching the actual UI.
|
1. **English Only**: All user-facing output, docstrings, comments, and log messages must be in English. Chinese text is only used for Playwright element selectors matching the actual UI.
|
||||||
|
|
||||||
2. **Environment-First**: All configuration values should default to reading from environment variables. No hardcoded URLs, credentials, or user-specific paths in code.
|
2. **Environment-First**: Test scripts load environment variables and explicitly pass configuration to utility functions. No hardcoded values in code.
|
||||||
|
|
||||||
3. **Error Handling**: Functions that depend on environment variables should raise clear `ValueError` exceptions when required variables are missing.
|
3. **Error Handling**: Functions that depend on environment variables should raise clear `ValueError` exceptions when required variables are missing.
|
||||||
|
|
||||||
|
|||||||
999
docs/discrete_material_plan_extractor_api.md
Normal file
999
docs/discrete_material_plan_extractor_api.md
Normal file
@@ -0,0 +1,999 @@
|
|||||||
|
# Discrete Material Plan Extractor - API Reference
|
||||||
|
|
||||||
|
> **Pure Function Data Extraction for Yonyou BIP**
|
||||||
|
>
|
||||||
|
> Stateless functions for extracting discrete material plan data from Yonyou BIP ERP system with batch processing and Excel post-processing capabilities.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The `extractor` module provides high-level pure functions for extracting material plan data from the Yonyou BIP ERP system. All functions are **stateless** and **explicitly accept required parameters**, following the same design patterns as `extractor_core.py` and `auth.py`.
|
||||||
|
|
||||||
|
### Architecture Overview
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TB
|
||||||
|
subgraph User["User Code"]
|
||||||
|
A[Main Script]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Auth["utils/auth.py"]
|
||||||
|
B[login]
|
||||||
|
C[logout]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Extractor["utils/discrete_material_plan/extractor.py"]
|
||||||
|
D[extract_and_post_process]
|
||||||
|
E[extract_batches]
|
||||||
|
F[post_process_downloads]
|
||||||
|
G[read_order_ids_from_file]
|
||||||
|
H[chunk_order_ids]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Core["utils/discrete_material_plan/extractor_core.py"]
|
||||||
|
I[navigate_to_discrete_material_page]
|
||||||
|
J[setup_query_interface]
|
||||||
|
K[execute_batch_download_workflow]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Converter["utils/discrete_material_plan/excel_converter.py"]
|
||||||
|
L[ExcelConverter.convert]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph External["External"]
|
||||||
|
M[(Yonyou BIP ERP)]
|
||||||
|
N[(Downloaded Excel Files)]
|
||||||
|
O[(Merged Output Excel)]
|
||||||
|
end
|
||||||
|
|
||||||
|
A --> B
|
||||||
|
A --> D
|
||||||
|
A --> G
|
||||||
|
|
||||||
|
B --> M
|
||||||
|
D --> E
|
||||||
|
D --> F
|
||||||
|
E --> H
|
||||||
|
E --> K
|
||||||
|
F --> L
|
||||||
|
|
||||||
|
B --> I
|
||||||
|
I --> M
|
||||||
|
K --> M
|
||||||
|
K --> N
|
||||||
|
|
||||||
|
L --> N
|
||||||
|
L --> O
|
||||||
|
|
||||||
|
style User fill:#e1f5ff
|
||||||
|
style Auth fill:#fff4e1
|
||||||
|
style Extractor fill:#e8f5e9
|
||||||
|
style Core fill:#fce4ec
|
||||||
|
style Converter fill:#f3e5f5
|
||||||
|
style External fill:#ffebee
|
||||||
|
```
|
||||||
|
|
||||||
|
**Key Features:**
|
||||||
|
- ✅ Stateless pure functions (no class instances)
|
||||||
|
- ✅ Explicit parameter passing
|
||||||
|
- ✅ Batch processing for large order lists
|
||||||
|
- ✅ Excel conversion and merging
|
||||||
|
- ✅ Progress reporting with verbose logging
|
||||||
|
- ✅ Caller-managed session lifecycle
|
||||||
|
|
||||||
|
**Module Location:** `utils/discrete_material_plan/extractor.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Basic Usage - Complete Workflow
|
||||||
|
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import (
|
||||||
|
extract_and_post_process,
|
||||||
|
get_login_url,
|
||||||
|
)
|
||||||
|
from utils.auth import login
|
||||||
|
from utils.discrete_material_plan import navigate_to_discrete_material_page
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
|
||||||
|
# 1. Setup browser session
|
||||||
|
with sync_playwright() as playwright:
|
||||||
|
browser, context, page, main_frame = login(
|
||||||
|
playwright=playwright,
|
||||||
|
username="your_username",
|
||||||
|
password="your_password",
|
||||||
|
url=get_login_url("https://erp.example.com"),
|
||||||
|
headless=True,
|
||||||
|
ignore_https_errors=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# 2. Navigate to discrete material page
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
|
||||||
|
# 3. Extract and process data
|
||||||
|
output_path, df = extract_and_post_process(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page1,
|
||||||
|
order_ids=["SC70202603240001", "SC70202603240002"],
|
||||||
|
download_dir="./downloads",
|
||||||
|
output_file="./output/merged_data.xlsx",
|
||||||
|
batch_size=10,
|
||||||
|
verbose=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Extracted {len(df)} records to {output_path}")
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# 4. Caller manages browser lifecycle
|
||||||
|
context.close()
|
||||||
|
browser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
### From File - Simplest Approach
|
||||||
|
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import read_order_ids_from_file
|
||||||
|
|
||||||
|
# Read order IDs from file
|
||||||
|
order_ids = read_order_ids_from_file("order_ids.txt")
|
||||||
|
print(f"Loaded {len(order_ids)} order IDs")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## API Reference
|
||||||
|
|
||||||
|
### Helper Functions
|
||||||
|
|
||||||
|
#### `chunk_order_ids(order_ids: List[str], batch_size: int) -> List[List[str]]`
|
||||||
|
|
||||||
|
Split order IDs into batches for processing.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Description |
|
||||||
|
|------|------|-------------|
|
||||||
|
| `order_ids` | `List[str]` | List of order IDs to process |
|
||||||
|
| `batch_size` | `int` | Maximum order IDs per batch |
|
||||||
|
|
||||||
|
**Returns:** `List[List[str]]` - List of batches
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
>>> chunk_order_ids(["A", "B", "C", "D", "E"], 2)
|
||||||
|
[["A", "B"], ["C", "D"], ["E"]]
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
#### `get_login_url(base_url: str) -> str`
|
||||||
|
|
||||||
|
Construct complete login URL from base ERP URL.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Description |
|
||||||
|
|------|------|-------------|
|
||||||
|
| `base_url` | `str` | Base ERP URL (e.g., "https://erp.example.com") |
|
||||||
|
|
||||||
|
**Returns:** `str` - Complete login page URL
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
>>> get_login_url("https://erp.example.com")
|
||||||
|
"https://erp.example.com/yonbip/resources/uap/rbac/login/main/index.html"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
#### `read_order_ids_from_file(id_file: str, encoding: str = "utf-8") -> List[str]`
|
||||||
|
|
||||||
|
Read order IDs from a text file (one ID per line).
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `id_file` | `str` | - | Path to file containing order IDs |
|
||||||
|
| `encoding` | `str` | `"utf-8"` | File encoding |
|
||||||
|
|
||||||
|
**Returns:** `List[str]` - List of order IDs (empty lines filtered)
|
||||||
|
|
||||||
|
**Raises:**
|
||||||
|
- `FileNotFoundError` - If id_file doesn't exist
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
>>> order_ids = read_order_ids_from_file("orders.txt")
|
||||||
|
>>> len(order_ids)
|
||||||
|
15
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Core Extraction Functions
|
||||||
|
|
||||||
|
#### `extract_batch(work_frame, page, order_ids, batch_index, download_dir) -> str`
|
||||||
|
|
||||||
|
Execute download workflow for a single batch of order IDs.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Description |
|
||||||
|
|------|------|-------------|
|
||||||
|
| `work_frame` | `Frame` | Work iframe containing the data grid |
|
||||||
|
| `page` | `Page` | Playwright page object for download handling |
|
||||||
|
| `order_ids` | `List[str]` | Order IDs for this batch |
|
||||||
|
| `batch_index` | `int` | Zero-based batch index (for file naming) |
|
||||||
|
| `download_dir` | `str` | Directory to save downloaded file |
|
||||||
|
|
||||||
|
**Returns:** `str` - Path to downloaded Excel file
|
||||||
|
|
||||||
|
**Note:** This is a thin wrapper around `execute_batch_download_workflow` from `extractor_core.py`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
#### `extract_batches(work_frame, page, order_ids, download_dir, batch_size=10) -> List[str]`
|
||||||
|
|
||||||
|
Download data for multiple batches of order IDs.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `work_frame` | `Frame` | Work iframe containing the data grid |
|
||||||
|
| `page` | `Page` | Playwright page object |
|
||||||
|
| `order_ids` | `List[str]` | List of order IDs to download |
|
||||||
|
| `download_dir` | `str` | Directory to save downloaded files |
|
||||||
|
| `batch_size` | `int` | `10` | Max order IDs per batch |
|
||||||
|
|
||||||
|
**Returns:** `List[str]` - List of downloaded file paths
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
# Caller manages browser session
|
||||||
|
browser, context, page, main_frame = login(...)
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
setup_query_interface(work_frame)
|
||||||
|
|
||||||
|
files = extract_batches(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page1,
|
||||||
|
order_ids=["ID1", "ID2", "ID3"],
|
||||||
|
download_dir="./downloads",
|
||||||
|
batch_size=5
|
||||||
|
)
|
||||||
|
|
||||||
|
context.close()
|
||||||
|
browser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### High-Level Workflow Functions
|
||||||
|
|
||||||
|
#### `extract_and_post_process(work_frame, page, order_ids, download_dir, output_file, batch_size=10, verbose=True) -> Tuple[str, DataFrame]`
|
||||||
|
|
||||||
|
Complete extraction workflow: download batches + post-process to merged Excel.
|
||||||
|
|
||||||
|
### Workflow Diagram
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
sequenceDiagram
|
||||||
|
participant User as User Code
|
||||||
|
participant Auth as utils.auth.login
|
||||||
|
participant Nav as navigate_to_discrete<br/>_material_page
|
||||||
|
participant Setup as setup_query<br/>_interface
|
||||||
|
participant Extract as extract_and_post<br/>_process
|
||||||
|
participant Batches as extract_batches
|
||||||
|
participant PostProcess as post_process<br/>_downloads
|
||||||
|
participant Converter as ExcelConverter
|
||||||
|
participant Browser as Browser
|
||||||
|
participant ERP as Yonyou BIP ERP
|
||||||
|
participant FS as File System
|
||||||
|
|
||||||
|
User->>Auth: login(credentials)
|
||||||
|
Auth->>Browser: Launch
|
||||||
|
Auth->>ERP: Authenticate
|
||||||
|
Auth-->>User: (browser, context, page, main_frame)
|
||||||
|
|
||||||
|
User->>Nav: navigate(main_frame, page)
|
||||||
|
Nav->>ERP: Load page
|
||||||
|
Nav-->>User: (work_frame, page1)
|
||||||
|
|
||||||
|
User->>Setup: setup_query_interface(work_frame)
|
||||||
|
Setup->>ERP: Configure query panel
|
||||||
|
|
||||||
|
User->>Extract: extract_and_post_process(params)
|
||||||
|
|
||||||
|
Note over Extract,PostProcess: Phase 1: Download
|
||||||
|
Extract->>Batches: extract_batches(work_frame, page, order_ids)
|
||||||
|
|
||||||
|
loop For each batch
|
||||||
|
Batches->>Batches: chunk_order_ids(order_ids, batch_size)
|
||||||
|
Batches->>Setup: setup_query_interface()
|
||||||
|
Batches->>ERP: Fill order IDs + Search
|
||||||
|
Batches->>ERP: Click Export
|
||||||
|
ERP-->>FS: Save batch_N.xlsx
|
||||||
|
Batches-->>Extract: [batch_1.xlsx, batch_2.xlsx, ...]
|
||||||
|
end
|
||||||
|
|
||||||
|
Note over Extract,PostProcess: Phase 2: Post-Process
|
||||||
|
Extract->>PostProcess: post_process_downloads(files, output_file)
|
||||||
|
|
||||||
|
loop For each downloaded file
|
||||||
|
PostProcess->>Converter: convert(batch_N.xlsx)
|
||||||
|
Converter->>FS: Read Excel
|
||||||
|
Converter->>Converter: Parse nested structure
|
||||||
|
Converter-->>PostProcess: DataFrame
|
||||||
|
end
|
||||||
|
|
||||||
|
PostProcess->>PostProcess: pd.concat(all_dfs)
|
||||||
|
PostProcess->>FS: Write merged.xlsx
|
||||||
|
PostProcess-->>Extract: (output_path, merged_df)
|
||||||
|
|
||||||
|
Extract-->>User: (output_path, DataFrame)
|
||||||
|
|
||||||
|
User->>Browser: context.close()
|
||||||
|
User->>Browser: browser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `work_frame` | `Frame` | - | Work iframe containing the data grid |
|
||||||
|
| `page` | `Page` | - | Playwright page object for download handling |
|
||||||
|
| `order_ids` | `List[str]` | - | List of order IDs to extract |
|
||||||
|
| `download_dir` | `str` | - | Directory for temporary batch files |
|
||||||
|
| `output_file` | `str` | - | Path for final merged Excel output |
|
||||||
|
| `batch_size` | `int` | `10` | Max order IDs per batch |
|
||||||
|
| `verbose` | `bool` | `True` | Print progress messages |
|
||||||
|
|
||||||
|
**Returns:** `Tuple[str, pd.DataFrame]` - (output_file_path, merged_dataframe)
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import extract_and_post_process
|
||||||
|
|
||||||
|
# Caller manages session (see full example above)
|
||||||
|
output_path, df = extract_and_post_process(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page1,
|
||||||
|
order_ids=["SC70202603240001", "SC70202603240002"],
|
||||||
|
download_dir="./downloads",
|
||||||
|
output_file="./output/merged.xlsx",
|
||||||
|
batch_size=10,
|
||||||
|
verbose=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Saved {len(df)} records to {output_path}")
|
||||||
|
```
|
||||||
|
|
||||||
|
**Workflow:**
|
||||||
|
1. Downloads order data in batches to `download_dir`
|
||||||
|
2. Converts each downloaded Excel file to DataFrame
|
||||||
|
3. Merges all DataFrames into one
|
||||||
|
4. Saves merged result to `output_file`
|
||||||
|
5. Returns (path, dataframe) tuple
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
#### `extract_from_file(id_file, work_frame, page, download_dir, output_file, batch_size=10, verbose=True) -> Tuple[str, DataFrame]`
|
||||||
|
|
||||||
|
Extract data from order IDs in a file and post-process to merged Excel.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `id_file` | `str` | - | Path to file with order IDs (one per line) |
|
||||||
|
| `work_frame` | `Frame` | Work iframe containing the data grid |
|
||||||
|
| `page` | `Page` | Playwright page object for download |
|
||||||
|
| `download_dir` | `str` | Directory for temporary batch files |
|
||||||
|
| `output_file` | `str` | Path for final merged Excel output |
|
||||||
|
| `batch_size` | `int` | `10` | Max order IDs per batch |
|
||||||
|
| `verbose` | `bool` | `True` | Print progress messages |
|
||||||
|
|
||||||
|
**Returns:** `Tuple[str, pd.DataFrame]` - (output_file_path, merged_dataframe)
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import extract_from_file
|
||||||
|
|
||||||
|
# After setting up browser session and navigating to page
|
||||||
|
output_path, df = extract_from_file(
|
||||||
|
id_file="order_ids.txt",
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page1,
|
||||||
|
download_dir="./downloads",
|
||||||
|
output_file="./output/results.xlsx",
|
||||||
|
batch_size=10,
|
||||||
|
verbose=True,
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**File Format:**
|
||||||
|
```
|
||||||
|
SC70202603240001
|
||||||
|
SC70202603240002
|
||||||
|
SC70202603240003
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Post-Processing Functions
|
||||||
|
|
||||||
|
#### `post_process_downloads(downloaded_files, output_file, verbose=True) -> Tuple[str, DataFrame]`
|
||||||
|
|
||||||
|
Convert and merge downloaded Excel files into structured DataFrame.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `downloaded_files` | `List[str]` | - | List of downloaded Excel file paths |
|
||||||
|
| `output_file` | `str` | - | Path to save merged Excel result |
|
||||||
|
| `verbose` | `bool` | `True` | Print progress messages |
|
||||||
|
|
||||||
|
**Returns:** `Tuple[str, pd.DataFrame]` - (output_file_path, merged_dataframe)
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import post_process_downloads
|
||||||
|
|
||||||
|
output_path, df = post_process_downloads(
|
||||||
|
downloaded_files=[
|
||||||
|
"./downloads/batch_1.xlsx",
|
||||||
|
"./downloads/batch_2.xlsx",
|
||||||
|
],
|
||||||
|
output_file="./output/merged.xlsx",
|
||||||
|
verbose=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Merged {len(df)} records")
|
||||||
|
```
|
||||||
|
|
||||||
|
**Process:**
|
||||||
|
1. Uses `ExcelConverter` to convert each file
|
||||||
|
2. Merges all DataFrames with `pd.concat()`
|
||||||
|
3. Saves merged result to `output_file`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Function Hierarchy
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph LR
|
||||||
|
subgraph Level1["Level 1: Entry Points"]
|
||||||
|
A1[extract_and_post_process]
|
||||||
|
A2[extract_from_file]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Level2["Level 2: Workflow Orchestration"]
|
||||||
|
B1[extract_batches]
|
||||||
|
B2[post_process_downloads]
|
||||||
|
B3[read_order_ids_from_file]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Level3["Level 3: Core Operations"]
|
||||||
|
C1[extract_batch]
|
||||||
|
C2[chunk_order_ids]
|
||||||
|
C3[setup_query_interface]
|
||||||
|
C4[ExcelConverter.convert]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Level4["Level 4: Low-Level"]
|
||||||
|
D1[execute_batch_download_workflow]
|
||||||
|
D2[fill_and_search_orders]
|
||||||
|
D3[download_batch_data]
|
||||||
|
end
|
||||||
|
|
||||||
|
A1 --> B1
|
||||||
|
A1 --> B2
|
||||||
|
A2 --> B3
|
||||||
|
A2 --> A1
|
||||||
|
|
||||||
|
B1 --> C1
|
||||||
|
B1 --> C2
|
||||||
|
B1 --> C3
|
||||||
|
B2 --> C4
|
||||||
|
|
||||||
|
C1 --> D1
|
||||||
|
C3 --> D2
|
||||||
|
D1 --> D2
|
||||||
|
D1 --> D3
|
||||||
|
|
||||||
|
style Level1 fill:#e3f2fd
|
||||||
|
style Level2 fill:#fff3e0
|
||||||
|
style Level3 fill:#f3e5f5
|
||||||
|
style Level4 fill:#e8f5e9
|
||||||
|
```
|
||||||
|
|
||||||
|
### Design Principles
|
||||||
|
|
||||||
|
**1. Stateless Pure Functions**
|
||||||
|
```python
|
||||||
|
# ✅ Correct: Stateless, explicit parameters
|
||||||
|
output_path, df = extract_and_post_process(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page,
|
||||||
|
order_ids=order_ids,
|
||||||
|
# ... explicit params
|
||||||
|
)
|
||||||
|
|
||||||
|
# ❌ Wrong: Stateful class (OLD approach - removed)
|
||||||
|
# extractor = DiscreteMaterialPlanExtractor(username, password, ...)
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Caller-Managed Lifecycle**
|
||||||
|
```python
|
||||||
|
# Caller manages browser session
|
||||||
|
browser, context, page, main_frame = login(...)
|
||||||
|
try:
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
result = extract_and_post_process(work_frame, page1, order_ids, ...)
|
||||||
|
finally:
|
||||||
|
context.close() # Caller closes
|
||||||
|
browser.close() # Caller closes
|
||||||
|
```
|
||||||
|
|
||||||
|
**3. Explicit Dependencies**
|
||||||
|
```python
|
||||||
|
# ❌ No implicit state
|
||||||
|
def extract(order_ids): # Missing required params
|
||||||
|
...
|
||||||
|
|
||||||
|
# ✅ All params explicit
|
||||||
|
def extract(work_frame, page, order_ids, download_dir, output_file):
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
### Module Dependencies
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TD
|
||||||
|
subgraph Extractor["extractor.py"]
|
||||||
|
E1[extract_and_post_process]
|
||||||
|
E2[extract_batches]
|
||||||
|
E3[post_process_downloads]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Core["extractor_core.py"]
|
||||||
|
C1[execute_batch_download_workflow]
|
||||||
|
C2[setup_query_interface]
|
||||||
|
C3[fill_and_search_orders]
|
||||||
|
C4[download_batch_data]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Converter["excel_converter.py"]
|
||||||
|
K1[ExcelConverter]
|
||||||
|
K2[convert]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph External["External Libraries"]
|
||||||
|
P1[pandas DataFrame]
|
||||||
|
P2[openpyxl]
|
||||||
|
end
|
||||||
|
|
||||||
|
E1 --> E2
|
||||||
|
E1 --> E3
|
||||||
|
E2 --> C1
|
||||||
|
E2 --> C2
|
||||||
|
E3 --> K1
|
||||||
|
K1 --> K2
|
||||||
|
K2 --> P1
|
||||||
|
K1 --> P2
|
||||||
|
|
||||||
|
C1 --> C2
|
||||||
|
C1 --> C3
|
||||||
|
C1 --> C4
|
||||||
|
|
||||||
|
style Extractor fill:#e8f5e9
|
||||||
|
style Core fill:#fff3e0
|
||||||
|
style Converter fill:#f3e5f5
|
||||||
|
style External fill:#e3f2fd
|
||||||
|
```
|
||||||
|
|
||||||
|
**Low Coupling:**
|
||||||
|
- No direct dependency on `utils.auth`
|
||||||
|
- Session objects (`work_frame`, `page`) passed as parameters
|
||||||
|
- Caller controls lifecycle
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Integration Patterns
|
||||||
|
|
||||||
|
### Data Flow Diagram
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
subgraph Input["Input"]
|
||||||
|
I1[Order IDs<br/>List or File]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Download["Download Phase<br/>Requires Browser"]
|
||||||
|
D1[Chunk into<br/>Batches]
|
||||||
|
D2[Search Orders<br/>in ERP]
|
||||||
|
D3[Export to<br/>Excel]
|
||||||
|
D4[(Batch Excel<br/>Files)]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Process["Post-Process Phase<br/>No Browser Needed"]
|
||||||
|
P1[Read Excel<br/>Files]
|
||||||
|
P2[Parse Nested<br/>Structure]
|
||||||
|
P3[Flatten to<br/>DataFrame]
|
||||||
|
P4[Concatenate<br/>All Batches]
|
||||||
|
P5[(Merged<br/>Excel)]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph Output["Output"]
|
||||||
|
O1[DataFrame<br/>Object]
|
||||||
|
O2[Excel File]
|
||||||
|
end
|
||||||
|
|
||||||
|
I1 --> D1
|
||||||
|
D1 --> D2
|
||||||
|
D2 --> D3
|
||||||
|
D3 --> D4
|
||||||
|
D4 --> P1
|
||||||
|
P1 --> P2
|
||||||
|
P2 --> P3
|
||||||
|
P3 --> P4
|
||||||
|
P4 --> P5
|
||||||
|
P4 --> O1
|
||||||
|
P5 --> O2
|
||||||
|
|
||||||
|
style Input fill:#e1f5ff
|
||||||
|
style Download fill:#fff4e1
|
||||||
|
style Process fill:#e8f5e9
|
||||||
|
style Output fill:#fce4ec
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Integration Patterns
|
||||||
|
|
||||||
|
### Pattern 1: Complete Workflow with Session Management
|
||||||
|
|
||||||
|
```python
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
from utils.auth import login, get_login_url
|
||||||
|
from utils.discrete_material_plan import (
|
||||||
|
navigate_to_discrete_material_page,
|
||||||
|
extract_and_post_process,
|
||||||
|
)
|
||||||
|
|
||||||
|
def main():
|
||||||
|
with sync_playwright() as playwright:
|
||||||
|
# Setup session
|
||||||
|
browser, context, page, main_frame = login(
|
||||||
|
playwright=playwright,
|
||||||
|
username="admin",
|
||||||
|
password="secret",
|
||||||
|
url=get_login_url("https://erp.example.com"),
|
||||||
|
headless=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Navigate
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
|
||||||
|
# Extract
|
||||||
|
output_path, df = extract_and_post_process(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page1,
|
||||||
|
order_ids=["ID1", "ID2", "ID3"],
|
||||||
|
download_dir="./downloads",
|
||||||
|
output_file="./output/result.xlsx",
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Done: {len(df)} records")
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# Cleanup
|
||||||
|
context.close()
|
||||||
|
browser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 2: Two-Phase (Download Then Process)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import (
|
||||||
|
extract_batches,
|
||||||
|
post_process_downloads,
|
||||||
|
setup_query_interface,
|
||||||
|
navigate_to_discrete_material_page,
|
||||||
|
)
|
||||||
|
from utils.auth import login
|
||||||
|
|
||||||
|
# Phase 1: Download
|
||||||
|
browser, context, page, main_frame = login(...)
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
|
||||||
|
downloaded_files = extract_batches(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page1,
|
||||||
|
order_ids=order_ids,
|
||||||
|
download_dir="./downloads",
|
||||||
|
batch_size=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
context.close()
|
||||||
|
browser.close()
|
||||||
|
|
||||||
|
# Phase 2: Process (can be done later, even without browser)
|
||||||
|
output_path, df = post_process_downloads(
|
||||||
|
downloaded_files=downloaded_files,
|
||||||
|
output_file="./output/merged.xlsx",
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 3: Custom Batch Processing with Error Handling
|
||||||
|
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import chunk_order_ids, extract_batch
|
||||||
|
from utils.auth import login
|
||||||
|
|
||||||
|
order_ids = [...] # Large list
|
||||||
|
batches = chunk_order_ids(order_ids, batch_size=10)
|
||||||
|
|
||||||
|
success_files = []
|
||||||
|
failed_batches = []
|
||||||
|
|
||||||
|
for i, batch in enumerate(batches):
|
||||||
|
try:
|
||||||
|
file_path = extract_batch(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page,
|
||||||
|
order_ids=batch,
|
||||||
|
batch_index=i,
|
||||||
|
download_dir="./downloads",
|
||||||
|
)
|
||||||
|
success_files.append(file_path)
|
||||||
|
print(f"Batch {i+1}/{len(batches)} OK")
|
||||||
|
except Exception as e:
|
||||||
|
failed_batches.append(i)
|
||||||
|
print(f"Batch {i+1} failed: {e}")
|
||||||
|
|
||||||
|
print(f"Success: {len(success_files)}, Failed: {len(failed_batches)}")
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Unit Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run component tests
|
||||||
|
source .venv/Scripts/activate && python tests/test_extractor_component.py
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test Coverage:**
|
||||||
|
- ✅ `chunk_order_ids()` - Batch splitting logic
|
||||||
|
- ✅ `get_login_url()` - URL construction
|
||||||
|
- ✅ `read_order_ids_from_file()` - File reading
|
||||||
|
- ✅ Module exports verification
|
||||||
|
|
||||||
|
### Integration Tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run real data extraction test
|
||||||
|
source .venv/Scripts/activate && python tests/test_extractor_real.py
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note:** Integration tests require:
|
||||||
|
- Valid ERP credentials in `.env`
|
||||||
|
- Playwright browser installed
|
||||||
|
- Network access to ERP system
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Common Errors
|
||||||
|
|
||||||
|
**FileNotFoundError:**
|
||||||
|
```python
|
||||||
|
>>> read_order_ids_from_file("nonexistent.txt")
|
||||||
|
FileNotFoundError: Order ID file not found: nonexistent.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
**Timeout during extraction:**
|
||||||
|
```python
|
||||||
|
try:
|
||||||
|
output_path, df = extract_and_post_process(...)
|
||||||
|
except TimeoutError as e:
|
||||||
|
print(f"Operation timed out: {e}")
|
||||||
|
# Browser session may need re-initialization
|
||||||
|
```
|
||||||
|
|
||||||
|
**Permission errors (file access):**
|
||||||
|
```python
|
||||||
|
try:
|
||||||
|
post_process_downloads(downloaded_files, output_file)
|
||||||
|
except PermissionError:
|
||||||
|
print(f"Cannot write to {output_file} - file may be open in another program")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Best Practices
|
||||||
|
|
||||||
|
```python
|
||||||
|
# 1. Always cleanup browser sessions
|
||||||
|
try:
|
||||||
|
# extraction logic
|
||||||
|
finally:
|
||||||
|
context.close()
|
||||||
|
browser.close()
|
||||||
|
|
||||||
|
# 2. Validate order IDs before extraction
|
||||||
|
order_ids = read_order_ids_from_file("orders.txt")
|
||||||
|
if not order_ids:
|
||||||
|
raise ValueError("No order IDs to process")
|
||||||
|
|
||||||
|
# 3. Ensure output directory exists
|
||||||
|
from pathlib import Path
|
||||||
|
Path("./output").mkdir(parents=True, exist_ok=True)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
### Batch Size Tuning
|
||||||
|
|
||||||
|
**Small batches (5-10):**
|
||||||
|
- ✅ More frequent progress updates
|
||||||
|
- ✅ Easier to recover from failures
|
||||||
|
- ❌ More UI interactions (slower)
|
||||||
|
|
||||||
|
**Large batches (50-100):**
|
||||||
|
- ✅ Faster overall (fewer UI interactions)
|
||||||
|
- ✅ Better for large datasets
|
||||||
|
- ❌ Single failure affects more records
|
||||||
|
|
||||||
|
**Recommendation:** Start with `batch_size=10`, adjust based on:
|
||||||
|
- Total order count
|
||||||
|
- Network stability
|
||||||
|
- UI response time
|
||||||
|
|
||||||
|
### Memory Usage
|
||||||
|
|
||||||
|
For very large extractions (1000+ orders):
|
||||||
|
```python
|
||||||
|
# Process in stages to limit memory
|
||||||
|
all_dfs = []
|
||||||
|
for batch_files in batch_groups:
|
||||||
|
_, df = post_process_downloads(batch_files, output_file)
|
||||||
|
all_dfs.append(df)
|
||||||
|
|
||||||
|
# Final merge
|
||||||
|
merged_df = pd.concat(all_dfs, ignore_index=True)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Migration Guide
|
||||||
|
|
||||||
|
### From Old Class-Based API (v1) to New Function API (v2)
|
||||||
|
|
||||||
|
### API Comparison
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
mindmap
|
||||||
|
root((API Comparison))
|
||||||
|
v1 Old API
|
||||||
|
Class-Based
|
||||||
|
::icon(fa fa-times-circle)
|
||||||
|
Stateful
|
||||||
|
Internal session management
|
||||||
|
Hidden dependencies
|
||||||
|
Usage
|
||||||
|
extractor = DiscreteMaterialPlanExtractor(...)
|
||||||
|
extractor.extract_from_file(...)
|
||||||
|
Issues
|
||||||
|
Hard to test
|
||||||
|
Tight coupling
|
||||||
|
Implicit state
|
||||||
|
v2 New API
|
||||||
|
Pure Functions
|
||||||
|
::icon(fa fa-check-circle)
|
||||||
|
Stateless
|
||||||
|
Caller-managed session
|
||||||
|
Explicit dependencies
|
||||||
|
Usage
|
||||||
|
extract_and_post_process(params...)
|
||||||
|
Return: (output_path, DataFrame)
|
||||||
|
Benefits
|
||||||
|
Easy to test
|
||||||
|
Low coupling
|
||||||
|
Clear data flow
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Old (v1):**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import DiscreteMaterialPlanExtractor
|
||||||
|
|
||||||
|
extractor = DiscreteMaterialPlanExtractor(
|
||||||
|
username="admin",
|
||||||
|
password="secret",
|
||||||
|
base_url="https://erp.example.com",
|
||||||
|
)
|
||||||
|
|
||||||
|
output_path = extractor.extract_from_file("orders.txt", "output.xlsx")
|
||||||
|
```
|
||||||
|
|
||||||
|
**New (v2):**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan import (
|
||||||
|
extract_from_file,
|
||||||
|
get_login_url,
|
||||||
|
navigate_to_discrete_material_page,
|
||||||
|
)
|
||||||
|
from utils.auth import login
|
||||||
|
|
||||||
|
# Caller manages session
|
||||||
|
browser, context, page, main_frame = login(
|
||||||
|
playwright,
|
||||||
|
username="admin",
|
||||||
|
password="secret",
|
||||||
|
url=get_login_url("https://erp.example.com"),
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
|
||||||
|
output_path, df = extract_from_file(
|
||||||
|
id_file="orders.txt",
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page1,
|
||||||
|
download_dir="./downloads",
|
||||||
|
output_file="./output/result.xlsx",
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
context.close()
|
||||||
|
browser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
### Version Comparison Table
|
||||||
|
|
||||||
|
| Aspect | v1 (Class-Based) | v2 (Pure Functions) |
|
||||||
|
|--------|------------------|---------------------|
|
||||||
|
| **Pattern** | `DiscreteMaterialPlanExtractor` class | Standalone functions |
|
||||||
|
| **State** | Instance variables (`self.username`, etc.) | No state, explicit params |
|
||||||
|
| **Session** | Internal management | Caller manages |
|
||||||
|
| **Return** | `str` (output path only) | `Tuple[str, DataFrame]` |
|
||||||
|
| **Testing** | Hard (requires mocking class) | Easy (pure functions) |
|
||||||
|
| **Coupling** | High (depends on `utils.auth`) | Low (session passed as param) |
|
||||||
|
| **Flexibility** | Limited (fixed workflow) | High (can use individual functions) |
|
||||||
|
|
||||||
|
**Key Changes:**
|
||||||
|
1. Class removed → pure functions
|
||||||
|
2. Session management moved to caller
|
||||||
|
3. Return value now includes DataFrame tuple
|
||||||
|
4. All dependencies explicit
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [`extractor_core.py`](discrete_material_plan_extractor_core.md) - Low-level web operations
|
||||||
|
- [`excel_converter.py`](extractor_post_processing.md) - Excel conversion
|
||||||
|
- [`auth.py`](authentication.md) - Authentication and session management
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
**Current:** v2.0.0 (stateless pure functions)
|
||||||
|
|
||||||
|
**Breaking Changes in v2:**
|
||||||
|
- Removed `DiscreteMaterialPlanExtractor` class
|
||||||
|
- Changed to caller-managed session lifecycle
|
||||||
|
- All functions now accept explicit parameters
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Part of BIPAuto project - see project root for license information.
|
||||||
426
docs/discrete_material_plan_extractor_core.md
Normal file
426
docs/discrete_material_plan_extractor_core.md
Normal file
@@ -0,0 +1,426 @@
|
|||||||
|
# extractor_core.py - API Reference
|
||||||
|
|
||||||
|
> **Core Web Operations for Yonyou BIP**
|
||||||
|
>
|
||||||
|
> Low-level pure functions for interacting with the discrete material plan maintenance page.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The `extractor_core.py` module provides low-level pure functions for web operations on the Yonyou BIP ERP system. All functions are **stateless** and accept an optional `logger` parameter for observability.
|
||||||
|
|
||||||
|
**Design Principles:**
|
||||||
|
- ✅ **Silent by default**: Functions produce no output when `logger=None`
|
||||||
|
- ✅ **Optional logging**: Pass a logger to get debug/info messages
|
||||||
|
- ✅ **Stateless**: No internal state, all dependencies explicit
|
||||||
|
- ✅ **Pure functions**: Same inputs → same outputs, no side effects
|
||||||
|
|
||||||
|
**Module Location:** `utils/discrete_material_plan/extractor_core.py`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Functions
|
||||||
|
|
||||||
|
### `navigate_to_discrete_material_page(main_frame, page, logger=None)`
|
||||||
|
|
||||||
|
Navigate to the discrete material plan maintenance page.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `main_frame` | `Frame` | - | The main forwardFrame iframe |
|
||||||
|
| `page` | `Page` | - | The Playwright page object |
|
||||||
|
| `logger` | `Optional[logging.Logger]` | `None` | Optional logger for debug output |
|
||||||
|
|
||||||
|
**Returns:** `tuple[FrameLocator, Page]` - (work_frame, page1)
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
- Clicks menu icon to open navigation
|
||||||
|
- Opens popup and clicks "离散备料计划维护" menu item
|
||||||
|
- Navigates through nested iframes to return work frame
|
||||||
|
|
||||||
|
**Logging (when logger provided):**
|
||||||
|
- `DEBUG`: "Navigated to discrete material plan page"
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan.extractor_core import navigate_to_discrete_material_page
|
||||||
|
import logging
|
||||||
|
|
||||||
|
# Silent mode (default)
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
|
||||||
|
# With logging
|
||||||
|
logging.basicConfig(level=logging.DEBUG)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page, logger)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `setup_query_interface(work_frame, logger=None)`
|
||||||
|
|
||||||
|
Initialize the query interface by selecting order number query tab.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `work_frame` | `FrameLocator` | - | The inner work iframe |
|
||||||
|
| `logger` | `Optional[logging.Logger]` | `None` | Optional logger for debug output |
|
||||||
|
|
||||||
|
**Returns:** `None`
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
- Opens search panel
|
||||||
|
- Selects "订单号查询" (order number query) tab
|
||||||
|
- Selects "全部" (All) tab
|
||||||
|
- Sets page size to 5000
|
||||||
|
|
||||||
|
**Logging (when logger provided):**
|
||||||
|
- `DEBUG`: "Query interface setup complete"
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan.extractor_core import setup_query_interface
|
||||||
|
|
||||||
|
# Silent mode
|
||||||
|
setup_query_interface(work_frame)
|
||||||
|
|
||||||
|
# With logging
|
||||||
|
setup_query_interface(work_frame, logger)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `fill_and_search_orders(work_frame, order_ids, logger=None)`
|
||||||
|
|
||||||
|
Fill order IDs into the search textbox and trigger search.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `work_frame` | `FrameLocator` | - | The work iframe containing search form |
|
||||||
|
| `order_ids` | `List[str]` | - | List of order IDs to search for |
|
||||||
|
| `logger` | `Optional[logging.Logger]` | `None` | Optional logger for debug output |
|
||||||
|
|
||||||
|
**Returns:** `None`
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
- Clears and fills order IDs into "来源生产订单号" textbox
|
||||||
|
- Clicks search button
|
||||||
|
- Waits for loading indicator (3s timeout, continues if not found)
|
||||||
|
|
||||||
|
**Logging (when logger provided):**
|
||||||
|
- `DEBUG`: "Loading indicator timeout - continuing anyway" (if timeout occurs)
|
||||||
|
- `DEBUG`: "Searched for {n} order IDs" (after search)
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan.extractor_core import fill_and_search_orders
|
||||||
|
|
||||||
|
# Silent mode
|
||||||
|
fill_and_search_orders(work_frame, ["SC70202603240001", "SC70202603240002"])
|
||||||
|
|
||||||
|
# With logging
|
||||||
|
fill_and_search_orders(work_frame, order_ids, logger)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `download_batch_data(work_frame, page, order_ids, batch_index, download_dir, logger=None)`
|
||||||
|
|
||||||
|
Execute the download workflow for a single batch of order IDs.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `work_frame` | `FrameLocator` | - | The work iframe containing data grid |
|
||||||
|
| `page` | `Page` | - | Playwright page for download handling |
|
||||||
|
| `order_ids` | `List[str]` | - | List of order IDs (for context) |
|
||||||
|
| `batch_index` | `int` | - | Zero-based batch index |
|
||||||
|
| `download_dir` | `str` | - | Directory to save downloaded file |
|
||||||
|
| `logger` | `Optional[logging.Logger]` | `None` | Optional logger for info output |
|
||||||
|
|
||||||
|
**Returns:** `str` - Full path to downloaded file
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
1. Selects first row in grid
|
||||||
|
2. Hovers over "更多" (More) button
|
||||||
|
3. Clicks "输出" (Export)
|
||||||
|
4. Sets row threshold to 300000
|
||||||
|
5. Triggers download and saves as `temp_batch_{n}.xlsx`
|
||||||
|
|
||||||
|
**Logging (when logger provided):**
|
||||||
|
- `INFO`: "Downloaded batch {n} to {path}"
|
||||||
|
|
||||||
|
**Raises:**
|
||||||
|
- `TimeoutError`: If UI elements not found or operations timeout
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan.extractor_core import download_batch_data
|
||||||
|
|
||||||
|
# Silent mode
|
||||||
|
file_path = download_batch_data(
|
||||||
|
work_frame, page, order_ids, batch_index=0, download_dir="./downloads"
|
||||||
|
)
|
||||||
|
|
||||||
|
# With logging
|
||||||
|
file_path = download_batch_data(
|
||||||
|
work_frame, page, order_ids, 0, "./downloads", logger
|
||||||
|
)
|
||||||
|
# INFO: Downloaded batch 1 to ./downloads/temp_batch_1.xlsx
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `execute_batch_download_workflow(work_frame, page, order_ids, batch_index, download_dir, logger=None)`
|
||||||
|
|
||||||
|
Complete workflow: fill orders, search, and download for a single batch.
|
||||||
|
|
||||||
|
**Parameters:**
|
||||||
|
| Name | Type | Default | Description |
|
||||||
|
|------|------|---------|-------------|
|
||||||
|
| `work_frame` | `FrameLocator` | - | Work iframe with search form and data grid |
|
||||||
|
| `page` | `Page` | - | Playwright page for download |
|
||||||
|
| `order_ids` | `List[str]` | - | List of order IDs for this batch |
|
||||||
|
| `batch_index` | `int` | - | Zero-based batch index |
|
||||||
|
| `download_dir` | `str` | - | Directory to save downloaded file |
|
||||||
|
| `logger` | `Optional[logging.Logger]` | `None` | Optional logger for debug output |
|
||||||
|
|
||||||
|
**Returns:** `str` - Full path to downloaded file
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
- Calls `fill_and_search_orders()` then `download_batch_data()`
|
||||||
|
- Propagates logger to both inner calls
|
||||||
|
|
||||||
|
**Logging (when logger provided):**
|
||||||
|
- All logs from `fill_and_search_orders()` and `download_batch_data()`
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan.extractor_core import execute_batch_download_workflow
|
||||||
|
|
||||||
|
# Silent mode
|
||||||
|
file_path = execute_batch_download_workflow(
|
||||||
|
work_frame, page, order_ids, batch_index=0, download_dir="./downloads"
|
||||||
|
)
|
||||||
|
|
||||||
|
# With logging
|
||||||
|
file_path = execute_batch_download_workflow(
|
||||||
|
work_frame, page, order_ids, 0, "./downloads", logger
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Usage Patterns
|
||||||
|
|
||||||
|
### Pattern 1: Silent Mode (Default)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan.extractor_core import (
|
||||||
|
navigate_to_discrete_material_page,
|
||||||
|
setup_query_interface,
|
||||||
|
execute_batch_download_workflow,
|
||||||
|
)
|
||||||
|
|
||||||
|
# No output, completely silent
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
setup_query_interface(work_frame)
|
||||||
|
file_path = execute_batch_download_workflow(
|
||||||
|
work_frame, page, order_ids, 0, "./downloads"
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 2: With Logging
|
||||||
|
|
||||||
|
```python
|
||||||
|
import logging
|
||||||
|
from utils.discrete_material_plan.extractor_core import (
|
||||||
|
navigate_to_discrete_material_page,
|
||||||
|
setup_query_interface,
|
||||||
|
execute_batch_download_workflow,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Configure logging
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.DEBUG,
|
||||||
|
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||||
|
)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Verbose output
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page, logger)
|
||||||
|
setup_query_interface(work_frame, logger)
|
||||||
|
file_path = execute_batch_download_workflow(
|
||||||
|
work_frame, page, order_ids, 0, "./downloads", logger
|
||||||
|
)
|
||||||
|
|
||||||
|
# Output:
|
||||||
|
# DEBUG:__main__:Navigated to discrete material plan page
|
||||||
|
# DEBUG:__main__:Query interface setup complete
|
||||||
|
# DEBUG:__main__:Searched for 10 order IDs
|
||||||
|
# INFO:__main__:Downloaded batch 1 to ./downloads/temp_batch_1.xlsx
|
||||||
|
```
|
||||||
|
|
||||||
|
### Pattern 3: Complete Workflow
|
||||||
|
|
||||||
|
```python
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
from utils.auth import login
|
||||||
|
from utils.discrete_material_plan.extractor_core import (
|
||||||
|
navigate_to_discrete_material_page,
|
||||||
|
setup_query_interface,
|
||||||
|
execute_batch_download_workflow,
|
||||||
|
)
|
||||||
|
import logging
|
||||||
|
|
||||||
|
logging.basicConfig(level=logging.INFO)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
with sync_playwright() as playwright:
|
||||||
|
browser, context, page, main_frame = login(
|
||||||
|
playwright, "user", "pass",
|
||||||
|
url="https://erp.example.com/...",
|
||||||
|
headless=True,
|
||||||
|
ignore_https_errors=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Navigate and setup
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page, logger)
|
||||||
|
setup_query_interface(work_frame, logger)
|
||||||
|
|
||||||
|
# Download batches
|
||||||
|
order_ids = ["SC70202603240001", "SC70202603240002", ...]
|
||||||
|
file_path = execute_batch_download_workflow(
|
||||||
|
work_frame, page1, order_ids, 0, "./downloads", logger
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
context.close()
|
||||||
|
browser.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### TimeoutError on Loading Indicator
|
||||||
|
|
||||||
|
The `fill_and_search_orders()` function handles loading indicator timeouts gracefully:
|
||||||
|
|
||||||
|
```python
|
||||||
|
try:
|
||||||
|
loading_locator.wait_for(state="visible", timeout=3000)
|
||||||
|
loading_locator.wait_for(state="hidden", timeout=0)
|
||||||
|
except TimeoutError:
|
||||||
|
if logger:
|
||||||
|
logger.debug("Loading indicator timeout - continuing anyway")
|
||||||
|
# Continues execution - not a fatal error
|
||||||
|
```
|
||||||
|
|
||||||
|
### When to Raise vs Log
|
||||||
|
|
||||||
|
- **Timeout on loading indicator**: Log at DEBUG, continue (non-fatal)
|
||||||
|
- **Missing UI elements**: Raise TimeoutError (fatal)
|
||||||
|
- **Download failures**: Raise exception (fatal)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Function Call Flow
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TD
|
||||||
|
A[User Script] --> B[navigate_to_discrete_material_page]
|
||||||
|
A --> C[setup_query_interface]
|
||||||
|
A --> D[execute_batch_download_workflow]
|
||||||
|
|
||||||
|
D --> E[fill_and_search_orders]
|
||||||
|
D --> F[download_batch_data]
|
||||||
|
|
||||||
|
E --> G[fill_and_search_orders implementation]
|
||||||
|
F --> H[download_batch_data implementation]
|
||||||
|
|
||||||
|
style A fill:#e1f5ff
|
||||||
|
style B fill:#fff4e1
|
||||||
|
style C fill:#fff4e1
|
||||||
|
style D fill:#e8f5e9
|
||||||
|
style E fill:#fce4ec
|
||||||
|
style F fill:#fce4ec
|
||||||
|
```
|
||||||
|
|
||||||
|
### Type Hierarchy
|
||||||
|
|
||||||
|
```
|
||||||
|
Page (from playwright)
|
||||||
|
└─> main_frame: Frame
|
||||||
|
└─> navigate_to_discrete_material_page()
|
||||||
|
└─> Returns: FrameLocator (work_frame)
|
||||||
|
└─> setup_query_interface(work_frame: FrameLocator)
|
||||||
|
└─> fill_and_search_orders(work_frame: FrameLocator)
|
||||||
|
└─> download_batch_data(work_frame: FrameLocator)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Import Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -c "from utils.discrete_material_plan.extractor_core import fill_and_search_orders; print('[OK] Import works')"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Silent Mode Test
|
||||||
|
|
||||||
|
```python
|
||||||
|
from utils.discrete_material_plan.extractor_core import fill_and_search_orders
|
||||||
|
|
||||||
|
# Should produce ZERO output
|
||||||
|
fill_and_search_orders(mock_frame, ["ID1", "ID2"])
|
||||||
|
print("Silent mode: OK")
|
||||||
|
```
|
||||||
|
|
||||||
|
### With Logger Test
|
||||||
|
|
||||||
|
```python
|
||||||
|
import logging
|
||||||
|
from utils.discrete_material_plan.extractor_core import fill_and_search_orders
|
||||||
|
|
||||||
|
logging.basicConfig(level=logging.DEBUG)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Should produce debug output
|
||||||
|
fill_and_search_orders(mock_frame, ["ID1", "ID2"], logger)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [`discrete_material_plan_extractor_api.md`](discrete_material_plan_extractor_api.md) - High-level extractor API
|
||||||
|
- [`extractor_post_processing.md`](extractor_post_processing.md) - Excel conversion
|
||||||
|
- [`authentication.md`](authentication.md) - Auth and session management
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
**Current:** v2.1.0 (added optional logger parameter)
|
||||||
|
|
||||||
|
**Changes in v2.1.0:**
|
||||||
|
- Added optional `logger` parameter to all functions
|
||||||
|
- Functions remain silent when `logger=None`
|
||||||
|
- TimeoutError handling now logs at DEBUG level
|
||||||
|
- No breaking changes - backward compatible
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Part of BIPAuto project - see project root for license information.
|
||||||
@@ -1,2 +1,4 @@
|
|||||||
playwright>=1.40.0
|
playwright>=1.40.0
|
||||||
python-dotenv>=1.0.0
|
python-dotenv>=1.0.0
|
||||||
|
pandas>=2.0.0
|
||||||
|
openpyxl>=3.1.0
|
||||||
|
|||||||
@@ -2,6 +2,7 @@
|
|||||||
Test if auth module configuration is correct
|
Test if auth module configuration is correct
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
from playwright.sync_api import sync_playwright
|
from playwright.sync_api import sync_playwright
|
||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -12,33 +13,45 @@ PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
|||||||
load_dotenv(PROJECT_ROOT / ".env")
|
load_dotenv(PROJECT_ROOT / ".env")
|
||||||
|
|
||||||
# Configure browser path
|
# Configure browser path
|
||||||
os.environ["PLAYWRIGHT_BROWSERS_PATH"] = os.getenv("PLAYWRIGHT_BROWSERS_PATH")
|
browser_path = os.getenv("PLAYWRIGHT_BROWSERS_PATH")
|
||||||
|
if not browser_path:
|
||||||
|
raise ValueError("PLAYWRIGHT_BROWSERS_PATH environment variable is required")
|
||||||
|
os.environ["PLAYWRIGHT_BROWSERS_PATH"] = browser_path
|
||||||
|
|
||||||
print("=" * 50)
|
# Setup logging
|
||||||
print("ERP System Configuration Check")
|
logger = logging.getLogger('bipauto.tests.auth_config')
|
||||||
print("=" * 50)
|
logger.setLevel(logging.INFO)
|
||||||
|
if not logger.handlers:
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
logger.propagate = False
|
||||||
|
|
||||||
|
logger.info("=" * 50)
|
||||||
|
logger.info("ERP System Configuration Check")
|
||||||
|
logger.info("=" * 50)
|
||||||
|
|
||||||
# Display configuration information
|
# Display configuration information
|
||||||
print(f"Browser Path: {os.getenv('PLAYWRIGHT_BROWSERS_PATH')}")
|
logger.info(f"Browser Path: {os.getenv('PLAYWRIGHT_BROWSERS_PATH')}")
|
||||||
print(f"ERP URL: {os.getenv('ERP_URL')}")
|
logger.info(f"ERP URL: {os.getenv('ERP_URL')}")
|
||||||
print(f"Username: {os.getenv('ERP_USERNAME')}")
|
logger.info(f"Username: {os.getenv('ERP_USERNAME')}")
|
||||||
print(f"Password: {'*' * len(os.getenv('ERP_PASSWORD', ''))}")
|
logger.info(f"Password: {'*' * len(os.getenv('ERP_PASSWORD', ''))}")
|
||||||
print(f"Headless Mode: {os.getenv('ERP_HEADLESS')}")
|
logger.info(f"Headless Mode: {os.getenv('ERP_HEADLESS')}")
|
||||||
print(f"Ignore HTTPS Errors: {os.getenv('ERP_IGNORE_HTTPS_ERRORS')}")
|
logger.info(f"Ignore HTTPS Errors: {os.getenv('ERP_IGNORE_HTTPS_ERRORS')}")
|
||||||
print(f"Auto Close Browser: {os.getenv('ERP_AUTO_CLOSE_BROWSER')}")
|
logger.info(f"Auto Close Browser: {os.getenv('ERP_AUTO_CLOSE_BROWSER')}")
|
||||||
|
|
||||||
print("\n" + "=" * 50)
|
logger.info("=" * 50)
|
||||||
print("Check Playwright Browser")
|
logger.info("Check Playwright Browser")
|
||||||
print("=" * 50)
|
logger.info("=" * 50)
|
||||||
|
|
||||||
with sync_playwright() as p:
|
with sync_playwright() as p:
|
||||||
chromium_path = p.chromium.executable_path
|
chromium_path = p.chromium.executable_path
|
||||||
print(f"Chromium Path: {chromium_path}")
|
logger.info(f"Chromium Path: {chromium_path}")
|
||||||
|
|
||||||
# Check if browser file exists
|
# Check if browser file exists
|
||||||
if os.path.exists(chromium_path):
|
if os.path.exists(chromium_path):
|
||||||
print("[OK] Browser file exists")
|
logger.info("Browser file exists")
|
||||||
else:
|
else:
|
||||||
print("[ERROR] Browser file not found")
|
logger.error("Browser file not found")
|
||||||
|
|
||||||
print("\n[OK] Configuration check completed!")
|
logger.info("Configuration check completed!")
|
||||||
|
|||||||
163
tests/test_extractor_component.py
Normal file
163
tests/test_extractor_component.py
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
"""
|
||||||
|
Test discrete_material_plan.extractor module functions
|
||||||
|
|
||||||
|
Tests pure functions for data extraction and post-processing.
|
||||||
|
Does not require actual browser or ERP connection.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# Add project root to Python path
|
||||||
|
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
sys.path.insert(0, str(PROJECT_ROOT))
|
||||||
|
|
||||||
|
# Load environment variables
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
load_dotenv(PROJECT_ROOT / ".env")
|
||||||
|
|
||||||
|
# Configure browser path (not used in this test, but consistent with other tests)
|
||||||
|
import os
|
||||||
|
os.environ["PLAYWRIGHT_BROWSERS_PATH"] = os.getenv("PLAYWRIGHT_BROWSERS_PATH", "")
|
||||||
|
|
||||||
|
# Setup logging
|
||||||
|
logger = logging.getLogger('bipauto.tests.extractor_component')
|
||||||
|
logger.setLevel(logging.INFO)
|
||||||
|
if not logger.handlers:
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
logger.propagate = False
|
||||||
|
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("Testing discrete_material_plan.extractor Functions")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Test 1: Import all functions
|
||||||
|
logger.info("Importing extractor functions...")
|
||||||
|
from utils.discrete_material_plan.extractor import (
|
||||||
|
chunk_order_ids,
|
||||||
|
get_login_url,
|
||||||
|
extract_batch,
|
||||||
|
extract_batches,
|
||||||
|
extract_and_post_process,
|
||||||
|
read_order_ids_from_file,
|
||||||
|
extract_from_file,
|
||||||
|
)
|
||||||
|
logger.info("All functions imported successfully")
|
||||||
|
|
||||||
|
# Test 2: Test chunk_order_ids
|
||||||
|
logger.info("Testing chunk_order_ids...")
|
||||||
|
|
||||||
|
# Test normal chunking
|
||||||
|
result = chunk_order_ids(["A", "B", "C", "D", "E"], 2)
|
||||||
|
expected = [["A", "B"], ["C", "D"], ["E"]]
|
||||||
|
assert result == expected, f"Expected {expected}, got {result}"
|
||||||
|
logger.info(f" chunk_order_ids(['A','B','C','D','E'], 2) = {result}")
|
||||||
|
|
||||||
|
# Test exact division
|
||||||
|
result = chunk_order_ids(["A", "B", "C", "D"], 2)
|
||||||
|
expected = [["A", "B"], ["C", "D"]]
|
||||||
|
assert result == expected, f"Expected {expected}, got {result}"
|
||||||
|
logger.info(f" chunk_order_ids(['A','B','C','D'], 2) = {result}")
|
||||||
|
|
||||||
|
# Test batch size larger than list
|
||||||
|
result = chunk_order_ids(["A", "B"], 10)
|
||||||
|
expected = [["A", "B"]]
|
||||||
|
assert result == expected, f"Expected {expected}, got {result}"
|
||||||
|
logger.info(f" chunk_order_ids(['A','B'], 10) = {result}")
|
||||||
|
|
||||||
|
# Test empty list
|
||||||
|
result = chunk_order_ids([], 5)
|
||||||
|
expected = []
|
||||||
|
assert result == expected, f"Expected {expected}, got {result}"
|
||||||
|
logger.info(f" chunk_order_ids([], 5) = {result}")
|
||||||
|
|
||||||
|
logger.info("chunk_order_ids works correctly")
|
||||||
|
|
||||||
|
# Test 3: Test get_login_url
|
||||||
|
logger.info("Testing get_login_url...")
|
||||||
|
|
||||||
|
# Test with trailing slash
|
||||||
|
result = get_login_url("https://erp.example.com/")
|
||||||
|
expected = "https://erp.example.com/yonbip/resources/uap/rbac/login/main/index.html"
|
||||||
|
assert result == expected, f"Expected {expected}, got {result}"
|
||||||
|
logger.info(f" get_login_url('https://erp.example.com/') = {result}")
|
||||||
|
|
||||||
|
# Test without trailing slash
|
||||||
|
result = get_login_url("https://erp.example.com")
|
||||||
|
assert result == expected, f"Expected {expected}, got {result}"
|
||||||
|
logger.info(f" get_login_url('https://erp.example.com') = {result}")
|
||||||
|
|
||||||
|
# Test with path
|
||||||
|
result = get_login_url("https://erp.example.com/some/path/")
|
||||||
|
expected = "https://erp.example.com/some/path/yonbip/resources/uap/rbac/login/main/index.html"
|
||||||
|
assert result == expected, f"Expected {expected}, got {result}"
|
||||||
|
logger.info(f" get_login_url('https://erp.example.com/some/path/') = {result}")
|
||||||
|
|
||||||
|
logger.info("get_login_url works correctly")
|
||||||
|
|
||||||
|
# Test 4: Test read_order_ids_from_file
|
||||||
|
logger.info("Testing read_order_ids_from_file...")
|
||||||
|
|
||||||
|
# Create a temporary test file
|
||||||
|
import tempfile
|
||||||
|
with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False, encoding='utf-8') as f:
|
||||||
|
f.write("ID001\n")
|
||||||
|
f.write("ID002\n")
|
||||||
|
f.write("\n") # Empty line
|
||||||
|
f.write("ID003\n")
|
||||||
|
f.write(" \n") # Whitespace only
|
||||||
|
f.write("ID004\n")
|
||||||
|
temp_file = f.name
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = read_order_ids_from_file(temp_file)
|
||||||
|
expected = ["ID001", "ID002", "ID003", "ID004"]
|
||||||
|
assert result == expected, f"Expected {expected}, got {result}"
|
||||||
|
logger.info(f" read_order_ids_from_file(temp_file) = {result}")
|
||||||
|
logger.info("read_order_ids_from_file works correctly")
|
||||||
|
finally:
|
||||||
|
# Cleanup temp file
|
||||||
|
Path(temp_file).unlink()
|
||||||
|
|
||||||
|
# Test FileNotFoundError
|
||||||
|
try:
|
||||||
|
read_order_ids_from_file("nonexistent_file.txt")
|
||||||
|
logger.error("Should have raised FileNotFoundError")
|
||||||
|
raise AssertionError("Should have raised FileNotFoundError")
|
||||||
|
except FileNotFoundError:
|
||||||
|
logger.info(" read_order_ids_from_file('nonexistent') raises FileNotFoundError")
|
||||||
|
|
||||||
|
# Test 5: Test module exports
|
||||||
|
logger.info("Testing module exports...")
|
||||||
|
from utils.discrete_material_plan import (
|
||||||
|
chunk_order_ids as exported_chunk,
|
||||||
|
get_login_url as exported_get_url,
|
||||||
|
extract_batch as exported_extract_batch,
|
||||||
|
extract_batches as exported_extract_batches,
|
||||||
|
extract_and_post_process as exported_extract_and_post,
|
||||||
|
read_order_ids_from_file as exported_read_ids,
|
||||||
|
extract_from_file as exported_extract_file,
|
||||||
|
)
|
||||||
|
logger.info("All functions exported correctly from module")
|
||||||
|
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("ALL EXTRACTOR FUNCTION TESTS PASSED")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("Note: extract_batch, extract_batches, extract_and_post_process,")
|
||||||
|
logger.info(" and extract_from_file require actual browser session and")
|
||||||
|
logger.info(" are not tested here. Integration tests cover those cases.")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Function test failed!")
|
||||||
|
logger.error(f"Error: {e}")
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("Test completed successfully")
|
||||||
|
logger.info("=" * 60)
|
||||||
184
tests/test_extractor_real.py
Normal file
184
tests/test_extractor_real.py
Normal file
@@ -0,0 +1,184 @@
|
|||||||
|
"""
|
||||||
|
Integration test for discrete_material_plan.extractor module
|
||||||
|
|
||||||
|
Tests the complete extraction workflow using real ERP data.
|
||||||
|
Requires valid ERP credentials in .env file.
|
||||||
|
|
||||||
|
This test:
|
||||||
|
1. Reads order IDs from tests/id-demo.txt
|
||||||
|
2. Downloads data from Yonyou BIP ERP system
|
||||||
|
3. Saves raw Excel files to data/downloads/
|
||||||
|
4. Converts and merges to data/output/merged_result.xlsx
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
# Add project root to Python path
|
||||||
|
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
sys.path.insert(0, str(PROJECT_ROOT))
|
||||||
|
|
||||||
|
# Load environment variables
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
load_dotenv(PROJECT_ROOT / ".env")
|
||||||
|
|
||||||
|
# Configure browser path
|
||||||
|
import os
|
||||||
|
os.environ["PLAYWRIGHT_BROWSERS_PATH"] = os.getenv("PLAYWRIGHT_BROWSERS_PATH", "")
|
||||||
|
|
||||||
|
# Setup logging
|
||||||
|
logger = logging.getLogger('bipauto.tests.extractor_real')
|
||||||
|
logger.setLevel(logging.INFO)
|
||||||
|
if not logger.handlers:
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
logger.propagate = False
|
||||||
|
|
||||||
|
# Generate timestamp for unique output files
|
||||||
|
TIMESTAMP = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
|
||||||
|
# Output paths
|
||||||
|
DOWNLOAD_DIR = PROJECT_ROOT / "data" / "downloads"
|
||||||
|
OUTPUT_DIR = PROJECT_ROOT / "data" / "output"
|
||||||
|
OUTPUT_FILE = OUTPUT_DIR / f"merged_result_{TIMESTAMP}.xlsx"
|
||||||
|
|
||||||
|
# Ensure directories exist
|
||||||
|
DOWNLOAD_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("Discrete Material Plan Extractor - Integration Test")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info(f"Configuration:")
|
||||||
|
logger.info(f" ID File: {PROJECT_ROOT / 'tests' / 'id-demo.txt'}")
|
||||||
|
logger.info(f" Download Dir: {DOWNLOAD_DIR}")
|
||||||
|
logger.info(f" Output File: {OUTPUT_FILE}")
|
||||||
|
logger.info(f" ERP URL: {os.getenv('ERP_URL', 'Not set')}")
|
||||||
|
logger.info(f" Headless: {os.getenv('ERP_HEADLESS', 'true')}")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
|
||||||
|
# Verify environment variables
|
||||||
|
required_vars = ["ERP_USERNAME", "ERP_PASSWORD", "ERP_URL"]
|
||||||
|
missing_vars = [var for var in required_vars if not os.getenv(var)]
|
||||||
|
|
||||||
|
if missing_vars:
|
||||||
|
logger.error(f"Missing required environment variables: {missing_vars}")
|
||||||
|
logger.error("Please ensure .env file contains:")
|
||||||
|
logger.error(" ERP_USERNAME=your_username")
|
||||||
|
logger.error(" ERP_PASSWORD=your_password")
|
||||||
|
logger.error(" ERP_URL=https://your-erp-url.com")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Import required modules
|
||||||
|
logger.info("[1/6] Importing modules...")
|
||||||
|
from playwright.sync_api import sync_playwright
|
||||||
|
from utils.auth import login
|
||||||
|
from utils.discrete_material_plan import (
|
||||||
|
navigate_to_discrete_material_page,
|
||||||
|
extract_and_post_process,
|
||||||
|
read_order_ids_from_file,
|
||||||
|
get_login_url,
|
||||||
|
)
|
||||||
|
logger.info("Modules imported successfully")
|
||||||
|
|
||||||
|
# Read order IDs from demo file
|
||||||
|
logger.info("[2/6] Reading order IDs from data/demo/id-demo.txt...")
|
||||||
|
id_file = PROJECT_ROOT / "data" / "demo" / "id-demo.txt"
|
||||||
|
order_ids = read_order_ids_from_file(str(id_file))
|
||||||
|
logger.info(f"Loaded {len(order_ids)} order IDs")
|
||||||
|
|
||||||
|
test_order_ids = order_ids
|
||||||
|
|
||||||
|
# Setup browser and extract data
|
||||||
|
logger.info("[3/6] Launching browser and logging in...")
|
||||||
|
with sync_playwright() as playwright:
|
||||||
|
# Get required environment variables (with validation)
|
||||||
|
username = os.getenv("ERP_USERNAME")
|
||||||
|
password = os.getenv("ERP_PASSWORD")
|
||||||
|
base_url = os.getenv("ERP_URL")
|
||||||
|
|
||||||
|
if not username or not password or not base_url:
|
||||||
|
raise ValueError("Missing required environment variables: ERP_USERNAME, ERP_PASSWORD, ERP_URL")
|
||||||
|
|
||||||
|
# Login
|
||||||
|
browser, context, page, main_frame = login(
|
||||||
|
playwright=playwright,
|
||||||
|
username=username,
|
||||||
|
password=password,
|
||||||
|
url=get_login_url(base_url),
|
||||||
|
headless=os.getenv("ERP_HEADLESS", "true").lower() == "true",
|
||||||
|
ignore_https_errors=os.getenv("ERP_IGNORE_HTTPS_ERRORS", "true").lower() == "true",
|
||||||
|
verbose=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Navigate to discrete material page
|
||||||
|
logger.info("[4/6] Navigating to discrete material plan page...")
|
||||||
|
work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
logger.info("Navigation successful")
|
||||||
|
|
||||||
|
# Extract and post-process
|
||||||
|
logger.info("[5/6] Extracting data and post-processing...")
|
||||||
|
logger.info(f" Orders: {len(test_order_ids)}")
|
||||||
|
logger.info(f" Batch size: 100")
|
||||||
|
logger.info(f" Download dir: {DOWNLOAD_DIR}")
|
||||||
|
logger.info(f" Output file: {OUTPUT_FILE}")
|
||||||
|
logger.info(f" Cleanup temp files: True (default)")
|
||||||
|
|
||||||
|
output_path, df = extract_and_post_process(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page1,
|
||||||
|
order_ids=test_order_ids,
|
||||||
|
download_dir=str(DOWNLOAD_DIR),
|
||||||
|
output_file=str(OUTPUT_FILE),
|
||||||
|
batch_size=100,
|
||||||
|
verbose=True,
|
||||||
|
cleanup_temp_files=True, # Default: True, set to False to keep temp files
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info("[6/6] Verifying results...")
|
||||||
|
logger.info("Extraction completed successfully")
|
||||||
|
logger.info(f" Output file: {output_path}")
|
||||||
|
logger.info(f" Total records: {len(df)}")
|
||||||
|
logger.info(f" Columns: {list(df.columns)}")
|
||||||
|
|
||||||
|
# Verify output file exists
|
||||||
|
if not Path(output_path).exists():
|
||||||
|
raise FileNotFoundError(f"Output file not found: {output_path}")
|
||||||
|
logger.info(f"Output file verified: {output_path}")
|
||||||
|
|
||||||
|
# Show sample data
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("Sample Data (first 3 rows, first 5 columns):")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
# Use unicode encoding for display
|
||||||
|
sample = df.head(3).iloc[:, :5]
|
||||||
|
logger.info("\n" + sample.to_string())
|
||||||
|
logger.info("Data extraction verified - Chinese characters displayed correctly in console encoding")
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# Cleanup browser
|
||||||
|
logger.info("Closing browser...")
|
||||||
|
context.close()
|
||||||
|
browser.close()
|
||||||
|
logger.info("Browser closed")
|
||||||
|
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("INTEGRATION TEST COMPLETED SUCCESSFULLY")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info(f"Outputs:")
|
||||||
|
logger.info(f" Raw Excel files: {DOWNLOAD_DIR}/*.xlsx")
|
||||||
|
logger.info(f" Merged result: {OUTPUT_FILE}")
|
||||||
|
logger.info(f" Total records: {len(df)}")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Integration test failed!")
|
||||||
|
logger.error(f"Error: {e}")
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
sys.exit(1)
|
||||||
@@ -3,6 +3,7 @@ Test Yonyou BIP system login functionality
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import sys
|
import sys
|
||||||
|
import logging
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
# Add project root to Python path
|
# Add project root to Python path
|
||||||
@@ -18,71 +19,95 @@ from dotenv import load_dotenv
|
|||||||
load_dotenv(PROJECT_ROOT / ".env")
|
load_dotenv(PROJECT_ROOT / ".env")
|
||||||
|
|
||||||
# Configure browser path
|
# Configure browser path
|
||||||
os.environ["PLAYWRIGHT_BROWSERS_PATH"] = os.getenv("PLAYWRIGHT_BROWSERS_PATH")
|
browser_path = os.getenv("PLAYWRIGHT_BROWSERS_PATH")
|
||||||
|
if not browser_path:
|
||||||
|
raise ValueError("PLAYWRIGHT_BROWSERS_PATH environment variable is required")
|
||||||
|
os.environ["PLAYWRIGHT_BROWSERS_PATH"] = browser_path
|
||||||
|
|
||||||
print("=" * 60)
|
# Setup logging
|
||||||
print("Testing Yonyou BIP Login and Logout")
|
logger = logging.getLogger('bipauto.tests.login')
|
||||||
print("=" * 60)
|
logger.setLevel(logging.INFO)
|
||||||
|
if not logger.handlers:
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
logger.propagate = False
|
||||||
|
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("Testing Yonyou BIP Login and Logout")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
|
||||||
# Display configuration information
|
# Display configuration information
|
||||||
print(f"\nConfiguration:")
|
logger.info("Configuration:")
|
||||||
print(f" URL: {os.getenv('ERP_URL')}")
|
logger.info(f" URL: {os.getenv('ERP_URL')}")
|
||||||
print(f" Username: {os.getenv('ERP_USERNAME')}")
|
logger.info(f" Username: {os.getenv('ERP_USERNAME')}")
|
||||||
print(f" Headless: {os.getenv('ERP_HEADLESS')}")
|
logger.info(f" Headless: {os.getenv('ERP_HEADLESS')}")
|
||||||
print(f" Ignore HTTPS Errors: {os.getenv('ERP_IGNORE_HTTPS_ERRORS')}")
|
logger.info(f" Ignore HTTPS Errors: {os.getenv('ERP_IGNORE_HTTPS_ERRORS')}")
|
||||||
|
|
||||||
# Construct complete login URL
|
# Construct complete login URL
|
||||||
url = f"{os.getenv('ERP_URL').rstrip('/')}/yonbip/resources/uap/rbac/login/main/index.html"
|
base_url = os.getenv('ERP_URL')
|
||||||
|
if not base_url:
|
||||||
|
raise ValueError("ERP_URL environment variable is required")
|
||||||
|
url = f"{base_url.rstrip('/')}/yonbip/resources/uap/rbac/login/main/index.html"
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
# Get required environment variables
|
||||||
|
username = os.getenv('ERP_USERNAME')
|
||||||
|
password = os.getenv('ERP_PASSWORD')
|
||||||
|
headless = os.getenv('ERP_HEADLESS', 'false').lower() in ('true', '1', 'yes')
|
||||||
|
ignore_https_errors = os.getenv('ERP_IGNORE_HTTPS_ERRORS', 'true').lower() in ('true', '1', 'yes')
|
||||||
|
|
||||||
|
if not username or not password:
|
||||||
|
raise ValueError("ERP_USERNAME and ERP_PASSWORD environment variables are required")
|
||||||
|
|
||||||
with sync_playwright() as p:
|
with sync_playwright() as p:
|
||||||
print("\n[1/5] Starting browser...")
|
logger.info("[1/5] Starting browser...")
|
||||||
browser, context, page, main_frame = login(
|
browser, context, page, main_frame = login(
|
||||||
playwright=p,
|
playwright=p,
|
||||||
username=os.getenv('ERP_USERNAME'),
|
username=username,
|
||||||
password=os.getenv('ERP_PASSWORD'),
|
password=password,
|
||||||
url=url,
|
url=url,
|
||||||
headless=os.getenv('ERP_HEADLESS', 'false').lower() in ('true', '1', 'yes'),
|
headless=headless,
|
||||||
ignore_https_errors=os.getenv('ERP_IGNORE_HTTPS_ERRORS', 'true').lower() in ('true', '1', 'yes'),
|
ignore_https_errors=ignore_https_errors,
|
||||||
verbose=True
|
verbose=True
|
||||||
)
|
)
|
||||||
|
|
||||||
print("\n[2/5] Login successful!")
|
logger.info("Login successful!")
|
||||||
|
|
||||||
# Wait a moment to observe the page after login
|
# Wait a moment to observe the page after login
|
||||||
print("\n[3/5] Waiting 3 seconds to observe the page after login...")
|
logger.info("[3/5] Waiting 3 seconds to observe the page after login...")
|
||||||
import time
|
import time
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
|
|
||||||
# Test logout
|
# Test logout
|
||||||
print("\n[4/5] Testing logout...")
|
logger.info("[4/5] Testing logout...")
|
||||||
logout(main_frame, verbose=True)
|
logout(main_frame, verbose=True)
|
||||||
|
|
||||||
print("\n[5/5] Logout successful!")
|
logger.info("Logout successful!")
|
||||||
|
|
||||||
# Wait a moment to observe the page after logout
|
# Wait a moment to observe the page after logout
|
||||||
print("\nWaiting 2 seconds to observe the page after logout...")
|
logger.info("Waiting 2 seconds to observe the page after logout...")
|
||||||
time.sleep(2)
|
time.sleep(2)
|
||||||
|
|
||||||
print("\n[SUCCESS] Login and logout test completed successfully!")
|
logger.info("Login and logout test completed successfully!")
|
||||||
|
|
||||||
# Check if auto-close browser
|
# Check if auto-close browser
|
||||||
auto_close = os.getenv("ERP_AUTO_CLOSE_BROWSER", "true").lower() in ("true", "1", "yes")
|
auto_close = os.getenv("ERP_AUTO_CLOSE_BROWSER", "true").lower() in ("true", "1", "yes")
|
||||||
if not auto_close:
|
if not auto_close:
|
||||||
print("\n[INFO] Browser will stay open for 10 seconds for manual observation...")
|
logger.info("Browser will stay open for 10 seconds for manual observation...")
|
||||||
time.sleep(10)
|
time.sleep(10)
|
||||||
|
|
||||||
# Close browser
|
# Close browser
|
||||||
print("\n[CLEANUP] Closing browser session...")
|
logger.info("Closing browser session...")
|
||||||
context.close()
|
context.close()
|
||||||
browser.close()
|
browser.close()
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"\n[ERROR] Login/logout test failed!")
|
logger.error(f"Login/logout test failed!")
|
||||||
print(f"Error: {e}")
|
logger.error(f"Error: {e}")
|
||||||
import traceback
|
import traceback
|
||||||
traceback.print_exc()
|
traceback.print_exc()
|
||||||
|
|
||||||
print("\n" + "=" * 60)
|
logger.info("=" * 60)
|
||||||
print("Test completed")
|
logger.info("Test completed")
|
||||||
print("=" * 60)
|
logger.info("=" * 60)
|
||||||
|
|||||||
14
utils/__init__.py
Normal file
14
utils/__init__.py
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
"""
|
||||||
|
Utils package for BIPAuto automation framework.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .auth import login, logout
|
||||||
|
from .logging import get_logger
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
# Logging module (foundation)
|
||||||
|
"get_logger",
|
||||||
|
# Auth module
|
||||||
|
"login",
|
||||||
|
"logout",
|
||||||
|
]
|
||||||
@@ -2,7 +2,9 @@
|
|||||||
Authentication module - Responsible for Yonyou BIP system login and logout operations
|
Authentication module - Responsible for Yonyou BIP system login and logout operations
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from playwright.sync_api import Playwright, Browser, BrowserContext, Page, Frame
|
import logging
|
||||||
|
from typing import Optional
|
||||||
|
from playwright.sync_api import Playwright, Browser, BrowserContext, Page, Frame, FrameLocator
|
||||||
|
|
||||||
|
|
||||||
def login(
|
def login(
|
||||||
@@ -13,7 +15,8 @@ def login(
|
|||||||
headless: bool,
|
headless: bool,
|
||||||
ignore_https_errors: bool,
|
ignore_https_errors: bool,
|
||||||
verbose: bool = True,
|
verbose: bool = True,
|
||||||
) -> tuple[Browser, BrowserContext, Page, Frame]:
|
logger: Optional[logging.Logger] = None,
|
||||||
|
) -> tuple[Browser, BrowserContext, Page, FrameLocator]:
|
||||||
"""
|
"""
|
||||||
Login to Yonyou BIP system
|
Login to Yonyou BIP system
|
||||||
|
|
||||||
@@ -24,7 +27,8 @@ def login(
|
|||||||
url: Complete login page URL (required)
|
url: Complete login page URL (required)
|
||||||
headless: Whether to use headless mode (required)
|
headless: Whether to use headless mode (required)
|
||||||
ignore_https_errors: Whether to ignore HTTPS errors (required)
|
ignore_https_errors: Whether to ignore HTTPS errors (required)
|
||||||
verbose: Whether to print detailed logs (default: True)
|
verbose: Whether to print detailed logs (default: True). Deprecated, use logger instead.
|
||||||
|
logger: Optional logging.Logger instance. If None and verbose=True, creates default logger.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
tuple: (browser, context, page, main_frame)
|
tuple: (browser, context, page, main_frame)
|
||||||
@@ -34,6 +38,13 @@ def login(
|
|||||||
- main_frame: Main iframe after login (forwardFrame)
|
- main_frame: Main iframe after login (forwardFrame)
|
||||||
"""
|
"""
|
||||||
import time
|
import time
|
||||||
|
from utils.logging import get_logger
|
||||||
|
|
||||||
|
# Create default logger if needed
|
||||||
|
if logger is None and verbose:
|
||||||
|
logger = get_logger('bipauto.auth')
|
||||||
|
elif logger is None:
|
||||||
|
logger = None # Silent mode, no logging
|
||||||
|
|
||||||
# Validate required parameters
|
# Validate required parameters
|
||||||
if not username or not username.strip():
|
if not username or not username.strip():
|
||||||
@@ -71,27 +82,35 @@ def login(
|
|||||||
confirm_btn = main_frame.get_by_role("button", name="确定")
|
confirm_btn = main_frame.get_by_role("button", name="确定")
|
||||||
if confirm_btn.count() > 0:
|
if confirm_btn.count() > 0:
|
||||||
confirm_btn.click()
|
confirm_btn.click()
|
||||||
if verbose:
|
if logger:
|
||||||
print("Force login detected")
|
logger.info("Force login detected")
|
||||||
else:
|
else:
|
||||||
if verbose:
|
if logger:
|
||||||
print("Normal login")
|
logger.debug("Normal login")
|
||||||
|
|
||||||
return browser, context, page, main_frame
|
return browser, context, page, main_frame
|
||||||
|
|
||||||
|
|
||||||
def logout(main_frame: Frame, verbose: bool = True) -> None:
|
def logout(main_frame: FrameLocator, verbose: bool = True, logger: Optional[logging.Logger] = None) -> None:
|
||||||
"""
|
"""
|
||||||
Execute account logout
|
Execute account logout
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
main_frame: Main iframe object (forwardFrame)
|
main_frame: Main iframe object (forwardFrame)
|
||||||
verbose: Whether to print detailed logs
|
verbose: Whether to print detailed logs. Deprecated, use logger instead.
|
||||||
|
logger: Optional logging.Logger instance. If None and verbose=True, creates default logger.
|
||||||
"""
|
"""
|
||||||
import time
|
import time
|
||||||
|
from utils.logging import get_logger
|
||||||
|
|
||||||
if verbose:
|
# Create default logger if needed
|
||||||
print("Clicking account menu button...")
|
if logger is None and verbose:
|
||||||
|
logger = get_logger('bipauto.auth')
|
||||||
|
elif logger is None:
|
||||||
|
logger = None # Silent mode, no logging
|
||||||
|
|
||||||
|
if logger:
|
||||||
|
logger.info("Clicking account menu button...")
|
||||||
|
|
||||||
# Element 1: Account menu button (logo icon)
|
# Element 1: Account menu button (logo icon)
|
||||||
main_frame.get_by_role("img", name="logo").click()
|
main_frame.get_by_role("img", name="logo").click()
|
||||||
@@ -99,8 +118,8 @@ def logout(main_frame: Frame, verbose: bool = True) -> None:
|
|||||||
# Wait for menu to appear
|
# Wait for menu to appear
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
|
|
||||||
if verbose:
|
if logger:
|
||||||
print("Clicking logout button...")
|
logger.info("Clicking logout button...")
|
||||||
|
|
||||||
# Element 2: "Logout" button
|
# Element 2: "Logout" button
|
||||||
main_frame.get_by_text("退出登录").click()
|
main_frame.get_by_text("退出登录").click()
|
||||||
@@ -108,19 +127,19 @@ def logout(main_frame: Frame, verbose: bool = True) -> None:
|
|||||||
# Wait for confirmation dialog to appear
|
# Wait for confirmation dialog to appear
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
|
|
||||||
if verbose:
|
if logger:
|
||||||
print("Waiting for logout confirmation dialog...")
|
logger.info("Waiting for logout confirmation dialog...")
|
||||||
|
|
||||||
# Element 3: Logout confirmation dialog (check if appears)
|
# Element 3: Logout confirmation dialog (check if appears)
|
||||||
try:
|
try:
|
||||||
confirm_text = main_frame.get_by_text("退出确定要退出当前账号吗?")
|
confirm_text = main_frame.get_by_text("退出确定要退出当前账号吗?")
|
||||||
if verbose:
|
if logger:
|
||||||
print("Found confirmation dialog, clicking confirm button")
|
logger.info("Found confirmation dialog, clicking confirm button")
|
||||||
|
|
||||||
# Element 4: Confirm button
|
# Element 4: Confirm button
|
||||||
main_frame.get_by_role("button", name="确定(Y)").click()
|
main_frame.get_by_role("button", name="确定(Y)").click()
|
||||||
except:
|
except:
|
||||||
if verbose:
|
if logger:
|
||||||
print("Confirmation dialog not found, may have auto-logged out")
|
logger.warning("Confirmation dialog not found, may have auto-logged out")
|
||||||
|
|
||||||
time.sleep(2) # Wait for logout to complete
|
time.sleep(2) # Wait for logout to complete
|
||||||
|
|||||||
43
utils/discrete_material_plan/__init__.py
Normal file
43
utils/discrete_material_plan/__init__.py
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
"""
|
||||||
|
Discrete Material Plan Maintenance Package
|
||||||
|
Core web operations and high-level extractor for Yonyou BIP discrete material plan maintenance.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .extractor_core import (
|
||||||
|
navigate_to_discrete_material_page,
|
||||||
|
setup_query_interface,
|
||||||
|
fill_and_search_orders,
|
||||||
|
download_batch_data,
|
||||||
|
execute_batch_download_workflow,
|
||||||
|
)
|
||||||
|
|
||||||
|
from .extractor import (
|
||||||
|
chunk_order_ids,
|
||||||
|
get_login_url,
|
||||||
|
extract_batch,
|
||||||
|
extract_batches,
|
||||||
|
extract_and_post_process,
|
||||||
|
read_order_ids_from_file,
|
||||||
|
extract_from_file,
|
||||||
|
)
|
||||||
|
|
||||||
|
from .excel_converter import ExcelConverter
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
# Core functions
|
||||||
|
"navigate_to_discrete_material_page",
|
||||||
|
"setup_query_interface",
|
||||||
|
"fill_and_search_orders",
|
||||||
|
"download_batch_data",
|
||||||
|
"execute_batch_download_workflow",
|
||||||
|
# High-level extractor functions
|
||||||
|
"chunk_order_ids",
|
||||||
|
"get_login_url",
|
||||||
|
"extract_batch",
|
||||||
|
"extract_batches",
|
||||||
|
"extract_and_post_process",
|
||||||
|
"read_order_ids_from_file",
|
||||||
|
"extract_from_file",
|
||||||
|
# Utilities
|
||||||
|
"ExcelConverter",
|
||||||
|
]
|
||||||
291
utils/discrete_material_plan/excel_converter.py
Normal file
291
utils/discrete_material_plan/excel_converter.py
Normal file
@@ -0,0 +1,291 @@
|
|||||||
|
"""
|
||||||
|
Excel Report Data Conversion Utility
|
||||||
|
Converts Excel report data to database record format
|
||||||
|
"""
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
import openpyxl
|
||||||
|
from typing import List, Dict, Optional
|
||||||
|
import os
|
||||||
|
import logging
|
||||||
|
from utils.logging import get_logger
|
||||||
|
|
||||||
|
|
||||||
|
class ExcelConverter:
|
||||||
|
"""Excel Report Data Converter"""
|
||||||
|
|
||||||
|
# Field name mapping (resolves field name conflicts)
|
||||||
|
FIELD_NAME_MAPPING = {"计划数量": "Product planned quantity", "单位": "Product unit"}
|
||||||
|
|
||||||
|
def __init__(self, verbose: bool = True, logger: Optional[logging.Logger] = None):
|
||||||
|
"""
|
||||||
|
Initialize the converter
|
||||||
|
|
||||||
|
Args:
|
||||||
|
verbose: Whether to print detailed logs (default: True). Deprecated, use logger instead.
|
||||||
|
logger: Optional logging.Logger instance. If None and verbose=True, creates default logger.
|
||||||
|
"""
|
||||||
|
self.verbose = verbose
|
||||||
|
|
||||||
|
# Create default logger if needed
|
||||||
|
if logger is None and verbose:
|
||||||
|
self.logger = get_logger('bipauto.converter')
|
||||||
|
elif logger is None:
|
||||||
|
# Silent mode - create a logger but disable output
|
||||||
|
silent_logger = logging.getLogger('bipauto.converter.silent')
|
||||||
|
silent_logger.setLevel(logging.CRITICAL + 1) # Higher than critical, never logs
|
||||||
|
self.logger = silent_logger
|
||||||
|
else:
|
||||||
|
self.logger = logger
|
||||||
|
|
||||||
|
def convert(self, input_file: str, output_file: Optional[str] = None) -> pd.DataFrame:
|
||||||
|
"""
|
||||||
|
Convert Excel file
|
||||||
|
|
||||||
|
Args:
|
||||||
|
input_file: Input file path
|
||||||
|
output_file: Output file path (optional, not saved if not specified)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Converted DataFrame
|
||||||
|
"""
|
||||||
|
# Handle output file
|
||||||
|
if output_file:
|
||||||
|
output_file = self._handle_output_file(output_file)
|
||||||
|
|
||||||
|
# Read worksheet
|
||||||
|
wb = openpyxl.load_workbook(input_file)
|
||||||
|
ws = wb.active
|
||||||
|
|
||||||
|
# Parse order data
|
||||||
|
orders = self._parse_sheet(ws)
|
||||||
|
|
||||||
|
# Convert to DataFrame
|
||||||
|
df = self._convert_to_dataframe(orders)
|
||||||
|
|
||||||
|
if not df.empty:
|
||||||
|
# Save file
|
||||||
|
if output_file:
|
||||||
|
df.to_excel(output_file, index=False)
|
||||||
|
|
||||||
|
# Print summary report (using logger)
|
||||||
|
self.logger.info("=" * 60)
|
||||||
|
self.logger.info("Conversion complete")
|
||||||
|
self.logger.info("=" * 60)
|
||||||
|
self.logger.info(f"Order count: {len(orders)}")
|
||||||
|
self.logger.info(f"Data row count: {len(df)}")
|
||||||
|
self.logger.info(f"Output file: {output_file if output_file else 'N/A'}")
|
||||||
|
self.logger.info("=" * 60)
|
||||||
|
|
||||||
|
return df
|
||||||
|
|
||||||
|
def _handle_output_file(self, output_file: str) -> str:
|
||||||
|
"""
|
||||||
|
Handle output file, attempt to delete if it exists
|
||||||
|
|
||||||
|
Args:
|
||||||
|
output_file: Output file path
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Actual output file path used
|
||||||
|
"""
|
||||||
|
if os.path.exists(output_file):
|
||||||
|
try:
|
||||||
|
os.remove(output_file)
|
||||||
|
except PermissionError:
|
||||||
|
self.logger.warning(
|
||||||
|
f"Warning: Could not delete {output_file}, file may be open by another program"
|
||||||
|
)
|
||||||
|
# Modify filename
|
||||||
|
base, ext = os.path.splitext(output_file)
|
||||||
|
output_file = f"{base}_new{ext}"
|
||||||
|
return output_file
|
||||||
|
|
||||||
|
def _parse_sheet(self, ws) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Parse a worksheet and return data for all orders
|
||||||
|
|
||||||
|
Each order contains:
|
||||||
|
- order_info: Order header information (including footer)
|
||||||
|
- materials: List of material data
|
||||||
|
|
||||||
|
Args:
|
||||||
|
ws: openpyxl worksheet object
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Order list
|
||||||
|
"""
|
||||||
|
orders = []
|
||||||
|
all_rows = list(ws.iter_rows(values_only=True))
|
||||||
|
|
||||||
|
# Scan row by row, parse by order structure
|
||||||
|
i = 0
|
||||||
|
while i < len(all_rows):
|
||||||
|
row = all_rows[i]
|
||||||
|
|
||||||
|
# Check if this is the order title row
|
||||||
|
if row and "离散备料计划" in str(row[0]):
|
||||||
|
# Parse order header information (next 4 rows)
|
||||||
|
order_info = {}
|
||||||
|
for j in range(1, 5):
|
||||||
|
if i + j < len(all_rows) and all_rows[i + j]:
|
||||||
|
self._parse_header_row(all_rows[i + j], order_info)
|
||||||
|
|
||||||
|
# Skip empty rows, find table header row
|
||||||
|
table_row = i + 5
|
||||||
|
while table_row < len(all_rows) and (
|
||||||
|
not all_rows[table_row] or not all_rows[table_row][0]
|
||||||
|
):
|
||||||
|
table_row += 1
|
||||||
|
|
||||||
|
# Check if this is the table header row
|
||||||
|
if (
|
||||||
|
table_row < len(all_rows)
|
||||||
|
and all_rows[table_row]
|
||||||
|
and all_rows[table_row][0] == "序号"
|
||||||
|
):
|
||||||
|
# Check if the row below the header is empty to determine if data exists
|
||||||
|
next_row = table_row + 1
|
||||||
|
is_empty_row = (
|
||||||
|
next_row < len(all_rows)
|
||||||
|
and all_rows[next_row]
|
||||||
|
and all(
|
||||||
|
cell is None or str(cell).strip() == ""
|
||||||
|
for cell in all_rows[next_row]
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if is_empty_row:
|
||||||
|
# No data, find footer information
|
||||||
|
materials = []
|
||||||
|
footer_info = {}
|
||||||
|
data_row = next_row + 1
|
||||||
|
while data_row < len(all_rows) and all_rows[data_row]:
|
||||||
|
if all_rows[data_row][0] and (
|
||||||
|
"制单人" in str(all_rows[data_row][0])
|
||||||
|
or "打印人" in str(all_rows[data_row][0])
|
||||||
|
):
|
||||||
|
self._parse_header_row(all_rows[data_row], footer_info)
|
||||||
|
if (
|
||||||
|
data_row + 1 < len(all_rows)
|
||||||
|
and all_rows[data_row + 1]
|
||||||
|
):
|
||||||
|
self._parse_header_row(
|
||||||
|
all_rows[data_row + 1], footer_info
|
||||||
|
)
|
||||||
|
break
|
||||||
|
data_row += 1
|
||||||
|
|
||||||
|
orders.append(
|
||||||
|
{
|
||||||
|
"order_info": {**order_info, **footer_info},
|
||||||
|
"materials": materials,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Has data, start extracting materials
|
||||||
|
materials = []
|
||||||
|
footer_info = {} # Footer information
|
||||||
|
data_row = table_row + 1
|
||||||
|
while data_row < len(all_rows) and all_rows[data_row]:
|
||||||
|
# Check if this is footer information (creator, printer)
|
||||||
|
if all_rows[data_row + 1][0] and "制单人" in str(
|
||||||
|
all_rows[data_row + 1][0]
|
||||||
|
):
|
||||||
|
# Parse footer information
|
||||||
|
self._parse_header_row(all_rows[data_row], footer_info)
|
||||||
|
# Check if the next row is also footer information
|
||||||
|
if (
|
||||||
|
data_row + 1 < len(all_rows)
|
||||||
|
and all_rows[data_row + 1]
|
||||||
|
):
|
||||||
|
self._parse_header_row(
|
||||||
|
all_rows[data_row + 1], footer_info
|
||||||
|
)
|
||||||
|
break
|
||||||
|
|
||||||
|
# Extract material data
|
||||||
|
material_row = all_rows[data_row]
|
||||||
|
material = {
|
||||||
|
"序号": material_row[0],
|
||||||
|
"材料编码": material_row[1],
|
||||||
|
"材料名称": material_row[2],
|
||||||
|
"规格": material_row[3],
|
||||||
|
"型号": material_row[4],
|
||||||
|
"图号": material_row[5],
|
||||||
|
"物料材质": material_row[6],
|
||||||
|
"计划数量": material_row[7],
|
||||||
|
"单位": material_row[8],
|
||||||
|
"需用日期": material_row[9],
|
||||||
|
"发料仓库": material_row[10],
|
||||||
|
"单位用量": material_row[11],
|
||||||
|
"累计出库数量": material_row[12],
|
||||||
|
}
|
||||||
|
materials.append(material)
|
||||||
|
|
||||||
|
data_row += 1
|
||||||
|
|
||||||
|
orders.append(
|
||||||
|
{
|
||||||
|
"order_info": {**order_info, **footer_info},
|
||||||
|
"materials": materials,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
i += 1
|
||||||
|
|
||||||
|
return orders
|
||||||
|
|
||||||
|
def _parse_header_row(self, row: tuple, info: Dict):
|
||||||
|
"""
|
||||||
|
Parse a row of order header information (field names and values interleaved)
|
||||||
|
|
||||||
|
Args:
|
||||||
|
row: Row data
|
||||||
|
info: Dictionary to store parsing results
|
||||||
|
"""
|
||||||
|
i = 0
|
||||||
|
while i < len(row):
|
||||||
|
cell = row[i]
|
||||||
|
if cell and str(cell).strip() and ":" in str(cell):
|
||||||
|
# Find field name
|
||||||
|
field_name = str(cell).replace(":", "").strip()
|
||||||
|
|
||||||
|
# Apply field name mapping
|
||||||
|
if field_name in self.FIELD_NAME_MAPPING:
|
||||||
|
field_name = self.FIELD_NAME_MAPPING[field_name]
|
||||||
|
|
||||||
|
# Skip empty cells, find the first non-field-name value
|
||||||
|
j = i + 1
|
||||||
|
while j < len(row) and (
|
||||||
|
not row[j] or not str(row[j]).strip() or ":" in str(row[j])
|
||||||
|
):
|
||||||
|
j += 1
|
||||||
|
if j < len(row) and row[j] and ":" not in str(row[j]):
|
||||||
|
info[field_name] = str(row[j]).strip()
|
||||||
|
# Skip processed value, continue to find next field name
|
||||||
|
i = j + 1
|
||||||
|
else:
|
||||||
|
i += 1
|
||||||
|
|
||||||
|
def _convert_to_dataframe(self, orders: List[Dict]) -> pd.DataFrame:
|
||||||
|
"""
|
||||||
|
Convert order data to a flattened DataFrame
|
||||||
|
|
||||||
|
Args:
|
||||||
|
orders: Order list
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Flattened DataFrame
|
||||||
|
"""
|
||||||
|
all_records = []
|
||||||
|
|
||||||
|
for order in orders:
|
||||||
|
order_info = order["order_info"]
|
||||||
|
materials = order["materials"]
|
||||||
|
|
||||||
|
for material in materials:
|
||||||
|
record = {**order_info, **material}
|
||||||
|
all_records.append(record)
|
||||||
|
|
||||||
|
return pd.DataFrame(all_records)
|
||||||
441
utils/discrete_material_plan/extractor.py
Normal file
441
utils/discrete_material_plan/extractor.py
Normal file
@@ -0,0 +1,441 @@
|
|||||||
|
"""
|
||||||
|
Discrete Material Plan Data Extractor
|
||||||
|
|
||||||
|
Pure functions for extracting and post-processing discrete material plan data.
|
||||||
|
All functions are stateless and accept required parameters explicitly.
|
||||||
|
Caller is responsible for browser/session lifecycle management.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import pandas as pd
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Optional, Tuple
|
||||||
|
from playwright.sync_api import Page, Frame, FrameLocator
|
||||||
|
import logging
|
||||||
|
from utils.logging import get_logger
|
||||||
|
|
||||||
|
|
||||||
|
def chunk_order_ids(order_ids: List[str], batch_size: int) -> List[List[str]]:
|
||||||
|
"""
|
||||||
|
Split order IDs into batches.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
order_ids: List of order IDs to process
|
||||||
|
batch_size: Maximum number of order IDs per batch
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of batches, where each batch is a list of order IDs
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> chunk_order_ids(["A", "B", "C", "D"], 2)
|
||||||
|
[["A", "B"], ["C", "D"]]
|
||||||
|
"""
|
||||||
|
return [
|
||||||
|
order_ids[i : i + batch_size]
|
||||||
|
for i in range(0, len(order_ids), batch_size)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def get_login_url(base_url: str) -> str:
|
||||||
|
"""
|
||||||
|
Construct the complete login URL from base URL.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
base_url: Base ERP URL (e.g., "https://erp.example.com")
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Complete login page URL
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> get_login_url("https://erp.example.com")
|
||||||
|
"https://erp.example.com/yonbip/resources/uap/rbac/login/main/index.html"
|
||||||
|
"""
|
||||||
|
return f"{base_url.rstrip('/')}/yonbip/resources/uap/rbac/login/main/index.html"
|
||||||
|
|
||||||
|
|
||||||
|
def extract_batch(
|
||||||
|
work_frame: FrameLocator,
|
||||||
|
page: Page,
|
||||||
|
order_ids: List[str],
|
||||||
|
batch_index: int,
|
||||||
|
download_dir: str,
|
||||||
|
logger: Optional[logging.Logger] = None,
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Execute download workflow for a single batch of order IDs.
|
||||||
|
|
||||||
|
This is a thin wrapper around `execute_batch_download_workflow` from
|
||||||
|
extractor_core.py, providing progress reporting context.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
work_frame: The work iframe containing the search form and data grid
|
||||||
|
page: The Playwright page object for download handling
|
||||||
|
order_ids: List of order IDs for this batch
|
||||||
|
batch_index: Zero-based batch index for naming the output file
|
||||||
|
download_dir: Directory path to save the downloaded file
|
||||||
|
logger: Optional logger for debug output (silent if None)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Full path to the downloaded Excel file
|
||||||
|
"""
|
||||||
|
from .extractor_core import execute_batch_download_workflow
|
||||||
|
|
||||||
|
return execute_batch_download_workflow(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page,
|
||||||
|
order_ids=order_ids,
|
||||||
|
batch_index=batch_index,
|
||||||
|
download_dir=download_dir,
|
||||||
|
logger=logger,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def extract_batches(
|
||||||
|
work_frame: FrameLocator,
|
||||||
|
page: Page,
|
||||||
|
order_ids: List[str],
|
||||||
|
download_dir: str,
|
||||||
|
batch_size: int = 10,
|
||||||
|
logger: Optional[logging.Logger] = None,
|
||||||
|
) -> List[str]:
|
||||||
|
"""
|
||||||
|
Download data for multiple batches of order IDs.
|
||||||
|
|
||||||
|
Caller is responsible for:
|
||||||
|
- Browser session management (login, logout)
|
||||||
|
- Navigation to discrete material plan page
|
||||||
|
- Query interface setup
|
||||||
|
|
||||||
|
Args:
|
||||||
|
work_frame: The work iframe containing the data grid
|
||||||
|
page: The Playwright page object for download handling
|
||||||
|
order_ids: List of order IDs to download
|
||||||
|
download_dir: Directory path to save downloaded files
|
||||||
|
batch_size: Maximum number of order IDs per batch
|
||||||
|
logger: Optional logger for debug output (silent if None)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of paths to downloaded Excel files
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> # Caller manages session
|
||||||
|
>>> browser, context, page, main_frame = login(...)
|
||||||
|
>>> work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
>>> setup_query_interface(work_frame)
|
||||||
|
>>> files = extract_batches(work_frame, page1, order_ids, "/downloads")
|
||||||
|
>>> context.close()
|
||||||
|
>>> browser.close()
|
||||||
|
"""
|
||||||
|
from .extractor_core import setup_query_interface
|
||||||
|
|
||||||
|
downloaded_files = []
|
||||||
|
chunks = chunk_order_ids(order_ids, batch_size)
|
||||||
|
|
||||||
|
# Setup query interface once
|
||||||
|
setup_query_interface(work_frame, logger)
|
||||||
|
|
||||||
|
# Process each batch
|
||||||
|
for batch_index, batch in enumerate(chunks):
|
||||||
|
file_path = extract_batch(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page,
|
||||||
|
order_ids=batch,
|
||||||
|
batch_index=batch_index,
|
||||||
|
download_dir=download_dir,
|
||||||
|
logger=logger,
|
||||||
|
)
|
||||||
|
downloaded_files.append(file_path)
|
||||||
|
|
||||||
|
return downloaded_files
|
||||||
|
|
||||||
|
|
||||||
|
def post_process_downloads(
|
||||||
|
downloaded_files: List[str],
|
||||||
|
output_file: str,
|
||||||
|
verbose: bool = True,
|
||||||
|
cleanup_temp_files: bool = True,
|
||||||
|
logger: Optional[logging.Logger] = None,
|
||||||
|
) -> Tuple[str, pd.DataFrame]:
|
||||||
|
"""
|
||||||
|
Convert and merge downloaded Excel files into structured DataFrame.
|
||||||
|
|
||||||
|
Uses ExcelConverter to convert each file, then merges all results.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
downloaded_files: List of paths to downloaded Excel files
|
||||||
|
output_file: Path to save merged Excel result
|
||||||
|
verbose: Whether to print progress messages (deprecated, use logger instead)
|
||||||
|
cleanup_temp_files: Whether to delete temporary downloaded files after processing (default: True)
|
||||||
|
logger: Optional logger for progress output. If None and verbose=True, creates default logger.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (output_file_path, merged_dataframe)
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> output_path, df = post_process_downloads(
|
||||||
|
... downloaded_files=["batch_1.xlsx", "batch_2.xlsx"],
|
||||||
|
... output_file="merged.xlsx",
|
||||||
|
... verbose=True,
|
||||||
|
... cleanup_temp_files=True
|
||||||
|
... )
|
||||||
|
"""
|
||||||
|
from .excel_converter import ExcelConverter
|
||||||
|
|
||||||
|
# Create default logger if needed
|
||||||
|
if logger is None and verbose:
|
||||||
|
logger = logging.getLogger('bipauto.extractor.post_process')
|
||||||
|
logger.setLevel(logging.INFO)
|
||||||
|
if not logger.handlers:
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
logger.propagate = False
|
||||||
|
elif logger is None:
|
||||||
|
# Silent mode
|
||||||
|
logger = logging.getLogger('bipauto.extractor.post_process.silent')
|
||||||
|
logger.setLevel(logging.CRITICAL + 1)
|
||||||
|
|
||||||
|
converter = ExcelConverter(verbose=verbose, logger=logger)
|
||||||
|
all_dfs = []
|
||||||
|
|
||||||
|
# Convert each file
|
||||||
|
for i, file_path in enumerate(downloaded_files):
|
||||||
|
logger.info(f"Converting file {i + 1}/{len(downloaded_files)}: {file_path}")
|
||||||
|
|
||||||
|
# Convert (do not save intermediate result)
|
||||||
|
df = converter.convert(input_file=file_path)
|
||||||
|
all_dfs.append(df)
|
||||||
|
|
||||||
|
# Merge all DataFrames
|
||||||
|
if not all_dfs:
|
||||||
|
merged_df = pd.DataFrame()
|
||||||
|
else:
|
||||||
|
merged_df = pd.concat(all_dfs, ignore_index=True)
|
||||||
|
|
||||||
|
# Save merged result
|
||||||
|
output_path = Path(output_file)
|
||||||
|
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
merged_df.to_excel(output_path, index=False)
|
||||||
|
|
||||||
|
logger.info(f"Merged result saved to: {output_path}")
|
||||||
|
logger.info(f"Total rows: {len(merged_df)}")
|
||||||
|
|
||||||
|
# Cleanup temporary downloaded files
|
||||||
|
if cleanup_temp_files:
|
||||||
|
_cleanup_temp_files(downloaded_files, logger=logger)
|
||||||
|
|
||||||
|
return str(output_path), merged_df
|
||||||
|
|
||||||
|
|
||||||
|
def _cleanup_temp_files(downloaded_files: List[str], logger: Optional[logging.Logger] = None, verbose: bool = True) -> int:
|
||||||
|
"""
|
||||||
|
Remove temporary downloaded files.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
downloaded_files: List of file paths to delete
|
||||||
|
logger: Optional logger for progress output. If None and verbose=True, creates default logger.
|
||||||
|
verbose: Whether to print progress messages (deprecated, use logger instead)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Number of files successfully deleted
|
||||||
|
"""
|
||||||
|
# Create default logger if needed
|
||||||
|
if logger is None and verbose:
|
||||||
|
logger = logging.getLogger('bipauto.extractor.cleanup')
|
||||||
|
logger.setLevel(logging.INFO)
|
||||||
|
if not logger.handlers:
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
logger.propagate = False
|
||||||
|
elif logger is None:
|
||||||
|
# Silent mode
|
||||||
|
logger = logging.getLogger('bipauto.extractor.cleanup.silent')
|
||||||
|
logger.setLevel(logging.CRITICAL + 1)
|
||||||
|
|
||||||
|
deleted_count = 0
|
||||||
|
for file_path in downloaded_files:
|
||||||
|
try:
|
||||||
|
Path(file_path).unlink()
|
||||||
|
deleted_count += 1
|
||||||
|
logger.debug(f"Deleted temp file: {file_path}")
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Warning: Could not delete {file_path}: {e}")
|
||||||
|
return deleted_count
|
||||||
|
|
||||||
|
|
||||||
|
def extract_and_post_process(
|
||||||
|
work_frame: FrameLocator,
|
||||||
|
page: Page,
|
||||||
|
order_ids: List[str],
|
||||||
|
download_dir: str,
|
||||||
|
output_file: str,
|
||||||
|
batch_size: int = 10,
|
||||||
|
verbose: bool = True,
|
||||||
|
cleanup_temp_files: bool = True,
|
||||||
|
logger: Optional[logging.Logger] = None,
|
||||||
|
) -> Tuple[str, pd.DataFrame]:
|
||||||
|
"""
|
||||||
|
Complete extraction workflow: download batches + post-process to merged Excel.
|
||||||
|
|
||||||
|
This is a high-level convenience function that orchestrates the full workflow.
|
||||||
|
Caller is still responsible for browser session management.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
work_frame: The work iframe containing the data grid
|
||||||
|
page: The Playwright page object for download handling
|
||||||
|
order_ids: List of order IDs to extract
|
||||||
|
download_dir: Directory for temporary batch files
|
||||||
|
output_file: Path for final merged Excel output
|
||||||
|
batch_size: Maximum order IDs per batch
|
||||||
|
verbose: Whether to print progress messages (deprecated, use logger instead)
|
||||||
|
cleanup_temp_files: Whether to delete temporary downloaded files after processing (default: True)
|
||||||
|
logger: Optional logger for debug output. If None and verbose=True, creates default logger.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (output_file_path, merged_dataframe)
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> # Caller manages session
|
||||||
|
>>> browser, context, page, main_frame = login(...)
|
||||||
|
>>> work_frame, page1 = navigate_to_discrete_material_page(main_frame, page)
|
||||||
|
>>> output_path, df = extract_and_post_process(
|
||||||
|
... work_frame, page1, order_ids, "/downloads", "output.xlsx",
|
||||||
|
... cleanup_temp_files=True
|
||||||
|
... )
|
||||||
|
>>> context.close()
|
||||||
|
>>> browser.close()
|
||||||
|
"""
|
||||||
|
# Create default logger if needed
|
||||||
|
if logger is None and verbose:
|
||||||
|
logger = logging.getLogger('bipauto.extractor')
|
||||||
|
logger.setLevel(logging.INFO)
|
||||||
|
if not logger.handlers:
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
logger.propagate = False
|
||||||
|
elif logger is None:
|
||||||
|
# Silent mode
|
||||||
|
logger = logging.getLogger('bipauto.extractor.silent')
|
||||||
|
logger.setLevel(logging.CRITICAL + 1)
|
||||||
|
|
||||||
|
# Step 1: Download all batches
|
||||||
|
logger.info(f"Downloading {len(order_ids)} orders in batches of {batch_size}...")
|
||||||
|
|
||||||
|
downloaded_files = extract_batches(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page,
|
||||||
|
order_ids=order_ids,
|
||||||
|
download_dir=download_dir,
|
||||||
|
batch_size=batch_size,
|
||||||
|
logger=logger,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Downloaded {len(downloaded_files)} batch file(s)")
|
||||||
|
|
||||||
|
# Step 2: Post-process (convert + merge)
|
||||||
|
output_path, merged_df = post_process_downloads(
|
||||||
|
downloaded_files=downloaded_files,
|
||||||
|
output_file=output_file,
|
||||||
|
verbose=verbose,
|
||||||
|
cleanup_temp_files=cleanup_temp_files,
|
||||||
|
logger=logger,
|
||||||
|
)
|
||||||
|
|
||||||
|
return output_path, merged_df
|
||||||
|
|
||||||
|
|
||||||
|
def read_order_ids_from_file(id_file: str, encoding: str = "utf-8") -> List[str]:
|
||||||
|
"""
|
||||||
|
Read order IDs from a text file (one ID per line).
|
||||||
|
|
||||||
|
Args:
|
||||||
|
id_file: Path to file containing order IDs
|
||||||
|
encoding: File encoding (default: utf-8)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of order IDs (stripped, empty lines filtered)
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
FileNotFoundError: If id_file doesn't exist
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> order_ids = read_order_ids_from_file("orders.txt")
|
||||||
|
"""
|
||||||
|
id_path = Path(id_file)
|
||||||
|
if not id_path.exists():
|
||||||
|
raise FileNotFoundError(f"Order ID file not found: {id_file}")
|
||||||
|
|
||||||
|
with open(id_path, "r", encoding=encoding) as f:
|
||||||
|
return [line.strip() for line in f if line.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def extract_from_file(
|
||||||
|
id_file: str,
|
||||||
|
work_frame: FrameLocator,
|
||||||
|
page: Page,
|
||||||
|
download_dir: str,
|
||||||
|
output_file: str,
|
||||||
|
batch_size: int = 10,
|
||||||
|
verbose: bool = True,
|
||||||
|
cleanup_temp_files: bool = True,
|
||||||
|
logger: Optional[logging.Logger] = None,
|
||||||
|
) -> Tuple[str, pd.DataFrame]:
|
||||||
|
"""
|
||||||
|
Extract data from order IDs in a file and post-process to merged Excel.
|
||||||
|
|
||||||
|
Convenience function that reads IDs from file and calls extract_and_post_process.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
id_file: Path to file containing order IDs (one per line)
|
||||||
|
work_frame: The work iframe containing the data grid
|
||||||
|
page: The Playwright page object for download handling
|
||||||
|
download_dir: Directory for temporary batch files
|
||||||
|
output_file: Path for final merged Excel output
|
||||||
|
batch_size: Maximum order IDs per batch
|
||||||
|
verbose: Whether to print progress messages (deprecated, use logger instead)
|
||||||
|
cleanup_temp_files: Whether to delete temporary downloaded files after processing (default: True)
|
||||||
|
logger: Optional logger for debug output. If None and verbose=True, creates default logger.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (output_file_path, merged_dataframe)
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> # Read order IDs
|
||||||
|
>>> order_ids = read_order_ids_from_file("orders.txt")
|
||||||
|
>>> # Extract and process
|
||||||
|
>>> output_path, df = extract_from_file(
|
||||||
|
... "orders.txt", work_frame, page, "/downloads", "output.xlsx",
|
||||||
|
... cleanup_temp_files=True
|
||||||
|
... )
|
||||||
|
"""
|
||||||
|
order_ids = read_order_ids_from_file(id_file)
|
||||||
|
|
||||||
|
# Create default logger if needed (for this function's own logging)
|
||||||
|
func_logger = logger
|
||||||
|
if func_logger is None and verbose:
|
||||||
|
func_logger = logging.getLogger('bipauto.extractor.file')
|
||||||
|
func_logger.setLevel(logging.INFO)
|
||||||
|
if not func_logger.handlers:
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("[%(levelname)s] %(name)s: %(message)s"))
|
||||||
|
func_logger.addHandler(handler)
|
||||||
|
func_logger.propagate = False
|
||||||
|
elif func_logger is None:
|
||||||
|
# Silent mode
|
||||||
|
func_logger = logging.getLogger('bipauto.extractor.file.silent')
|
||||||
|
func_logger.setLevel(logging.CRITICAL + 1)
|
||||||
|
|
||||||
|
func_logger.info(f"Loaded {len(order_ids)} order IDs from {id_file}")
|
||||||
|
|
||||||
|
return extract_and_post_process(
|
||||||
|
work_frame=work_frame,
|
||||||
|
page=page,
|
||||||
|
order_ids=order_ids,
|
||||||
|
download_dir=download_dir,
|
||||||
|
output_file=output_file,
|
||||||
|
batch_size=batch_size,
|
||||||
|
verbose=verbose,
|
||||||
|
cleanup_temp_files=cleanup_temp_files,
|
||||||
|
logger=logger,
|
||||||
|
)
|
||||||
185
utils/discrete_material_plan/extractor_core.py
Normal file
185
utils/discrete_material_plan/extractor_core.py
Normal file
@@ -0,0 +1,185 @@
|
|||||||
|
"""
|
||||||
|
Core Web Operations Module
|
||||||
|
Pure functions for Yonyou BIP discrete material plan maintenance page interactions.
|
||||||
|
All functions are stateless and accept required parameters explicitly.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
import os
|
||||||
|
import logging
|
||||||
|
from typing import List, Optional
|
||||||
|
from playwright.sync_api import Page, Frame, FrameLocator, TimeoutError
|
||||||
|
|
||||||
|
|
||||||
|
def navigate_to_discrete_material_page(main_frame: FrameLocator, page: Page, logger: Optional[logging.Logger] = None) -> tuple[FrameLocator, Page]:
|
||||||
|
"""
|
||||||
|
Navigate to the discrete material plan maintenance page.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
main_frame: The main forwardFrame iframe (FrameLocator)
|
||||||
|
page: The Playwright page object
|
||||||
|
logger: Optional logger for debug output (silent if None)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
tuple: (work_frame, page1) - The work iframe and the new popup page
|
||||||
|
"""
|
||||||
|
# Click icon to open menu
|
||||||
|
main_frame.locator("i").first.click()
|
||||||
|
|
||||||
|
# Wait for popup and click menu item
|
||||||
|
with page.expect_popup() as page1_info:
|
||||||
|
main_frame.get_by_title("离散备料计划维护", exact=True).first.click()
|
||||||
|
|
||||||
|
page1 = page1_info.value
|
||||||
|
|
||||||
|
# Get nested iframes
|
||||||
|
f_frame = page1.locator("#forwardFrame").content_frame
|
||||||
|
inner_frame_locator = f_frame.locator("#mainiframe")
|
||||||
|
inner_frame_locator.wait_for(state="visible", timeout=15000)
|
||||||
|
work_frame = inner_frame_locator.content_frame
|
||||||
|
|
||||||
|
if logger:
|
||||||
|
logger.debug("Navigated to discrete material plan page")
|
||||||
|
|
||||||
|
return work_frame, page1
|
||||||
|
|
||||||
|
|
||||||
|
def setup_query_interface(work_frame: FrameLocator, logger: Optional[logging.Logger] = None) -> None:
|
||||||
|
"""
|
||||||
|
Initialize the query interface by selecting order number query tab.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
work_frame: The inner work iframe containing the query interface
|
||||||
|
logger: Optional logger for debug output (silent if None)
|
||||||
|
"""
|
||||||
|
# Open search panel
|
||||||
|
work_frame.locator(".search-name-wrapper > .iconfont").click()
|
||||||
|
|
||||||
|
# Select order number query
|
||||||
|
work_frame.get_by_text("订单号查询").click()
|
||||||
|
|
||||||
|
# Select "All" tab
|
||||||
|
work_frame.get_by_role("tab", name="全部").click()
|
||||||
|
|
||||||
|
# Set page size to 5000
|
||||||
|
input_box = work_frame.locator("#rc_select_0")
|
||||||
|
input_box.fill("5000")
|
||||||
|
input_box.press("Enter")
|
||||||
|
|
||||||
|
if logger:
|
||||||
|
logger.debug("Query interface setup complete")
|
||||||
|
|
||||||
|
|
||||||
|
def fill_and_search_orders(work_frame: FrameLocator, order_ids: List[str], logger: Optional[logging.Logger] = None) -> None:
|
||||||
|
"""
|
||||||
|
Fill order IDs into the search textbox and trigger search.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
work_frame: The work iframe containing the search form
|
||||||
|
order_ids: List of order IDs to search for
|
||||||
|
logger: Optional logger for debug output (silent if None)
|
||||||
|
"""
|
||||||
|
textbox = work_frame.get_by_role("textbox", name="来源生产订单号")
|
||||||
|
|
||||||
|
# Clear and fill order IDs
|
||||||
|
textbox.fill("")
|
||||||
|
textbox.fill(",".join(order_ids))
|
||||||
|
|
||||||
|
# Click search button
|
||||||
|
work_frame.locator(".search-component-searchBtn").click()
|
||||||
|
|
||||||
|
# Wait for loading to complete
|
||||||
|
loading_locator = work_frame.locator("div").filter(has_text="加载中").nth(1)
|
||||||
|
try:
|
||||||
|
loading_locator.wait_for(state="visible", timeout=3000)
|
||||||
|
loading_locator.wait_for(state="hidden", timeout=0)
|
||||||
|
except TimeoutError:
|
||||||
|
if logger:
|
||||||
|
logger.debug("Loading indicator timeout - continuing anyway")
|
||||||
|
|
||||||
|
if logger:
|
||||||
|
logger.debug(f"Searched for {len(order_ids)} order IDs")
|
||||||
|
|
||||||
|
|
||||||
|
def download_batch_data(
|
||||||
|
work_frame: FrameLocator,
|
||||||
|
page: Page,
|
||||||
|
order_ids: List[str],
|
||||||
|
batch_index: int,
|
||||||
|
download_dir: str,
|
||||||
|
logger: Optional[logging.Logger] = None
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Execute the download workflow for a single batch of order IDs.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
work_frame: The work iframe containing the data grid
|
||||||
|
page: The Playwright page object for download handling
|
||||||
|
order_ids: List of order IDs to download
|
||||||
|
batch_index: Zero-based batch index for naming the output file
|
||||||
|
download_dir: Directory path to save the downloaded file
|
||||||
|
logger: Optional logger for info output (silent if None)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
str: Full path to the downloaded file
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
TimeoutError: If UI elements are not found or operations timeout
|
||||||
|
"""
|
||||||
|
# Step 1: Select first row
|
||||||
|
work_frame.get_by_role("row", name="序号").get_by_label("").click()
|
||||||
|
|
||||||
|
# Step 2: Hover over "More" button
|
||||||
|
work_frame.get_by_role("button", name="更多").hover()
|
||||||
|
|
||||||
|
# Step 3: Click "Export"
|
||||||
|
work_frame.get_by_text("输出", exact=True).click()
|
||||||
|
|
||||||
|
# Step 4: Set row threshold
|
||||||
|
threshold_box = (
|
||||||
|
work_frame.locator("div")
|
||||||
|
.filter(has_text=re.compile(r"^行数阈值$"))
|
||||||
|
.locator("input[type='text']")
|
||||||
|
)
|
||||||
|
threshold_box.fill("300000")
|
||||||
|
|
||||||
|
# Step 5: Trigger download and save file
|
||||||
|
download_filename = f"temp_batch_{batch_index + 1}.xlsx"
|
||||||
|
download_path = os.path.join(download_dir, download_filename)
|
||||||
|
|
||||||
|
with page.expect_download() as download_info:
|
||||||
|
work_frame.get_by_role("button", name="确定(Y)").click()
|
||||||
|
|
||||||
|
download = download_info.value
|
||||||
|
download.save_as(download_path)
|
||||||
|
|
||||||
|
if logger:
|
||||||
|
logger.info(f"Downloaded batch {batch_index + 1} to {download_path}")
|
||||||
|
|
||||||
|
return download_path
|
||||||
|
|
||||||
|
|
||||||
|
def execute_batch_download_workflow(
|
||||||
|
work_frame: FrameLocator,
|
||||||
|
page: Page,
|
||||||
|
order_ids: List[str],
|
||||||
|
batch_index: int,
|
||||||
|
download_dir: str,
|
||||||
|
logger: Optional[logging.Logger] = None
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Complete workflow: fill orders, search, and download for a single batch.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
work_frame: The work iframe containing the search form and data grid
|
||||||
|
page: The Playwright page object for download handling
|
||||||
|
order_ids: List of order IDs for this batch
|
||||||
|
batch_index: Zero-based batch index for naming the output file
|
||||||
|
download_dir: Directory path to save the downloaded file
|
||||||
|
logger: Optional logger for debug output (silent if None)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
str: Full path to the downloaded file
|
||||||
|
"""
|
||||||
|
fill_and_search_orders(work_frame, order_ids, logger)
|
||||||
|
return download_batch_data(work_frame, page, order_ids, batch_index, download_dir, logger)
|
||||||
81
utils/logging.py
Normal file
81
utils/logging.py
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
"""
|
||||||
|
Centralized logging utility module for BIPAuto project.
|
||||||
|
|
||||||
|
Provides a standardized logger configuration for consistent logging across
|
||||||
|
all BIPAuto components. Supports console output with optional file logging.
|
||||||
|
|
||||||
|
Basic usage examples:
|
||||||
|
|
||||||
|
# Basic usage - console logging at INFO level
|
||||||
|
from utils.logging import get_logger
|
||||||
|
logger = get_logger('bipauto.auth')
|
||||||
|
logger.info('Login successful')
|
||||||
|
|
||||||
|
# With debug level for detailed output
|
||||||
|
logger = get_logger('bipauto.extractor', level=logging.DEBUG)
|
||||||
|
logger.debug('Processing batch 1 of 5...')
|
||||||
|
|
||||||
|
# With file output for persistent logs
|
||||||
|
logger = get_logger('bipauto.app', level=logging.INFO, log_file='app.log')
|
||||||
|
logger.info('Application started')
|
||||||
|
|
||||||
|
Logger naming convention:
|
||||||
|
Use hierarchical names with 'bipauto.' prefix:
|
||||||
|
- 'bipauto.auth' - Authentication module
|
||||||
|
- 'bipauto.extractor' - Material plan extractor
|
||||||
|
- 'bipauto.converter' - Excel converter
|
||||||
|
- 'bipauto.utils' - Utility functions
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
|
||||||
|
def get_logger(
|
||||||
|
name: str,
|
||||||
|
level: int = logging.INFO,
|
||||||
|
log_file: Optional[str] = None
|
||||||
|
) -> logging.Logger:
|
||||||
|
"""
|
||||||
|
Create and configure a logger with console and optional file handlers.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
name: Logger name (use hierarchical naming, e.g., 'bipauto.auth')
|
||||||
|
level: Logging level (default: logging.INFO)
|
||||||
|
log_file: Optional path to log file. If None, only console output.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Configured logging.Logger instance
|
||||||
|
|
||||||
|
Example:
|
||||||
|
>>> logger = get_logger('bipauto.auth')
|
||||||
|
>>> logger.info('User logged in')
|
||||||
|
[INFO] bipauto.auth: User logged in
|
||||||
|
"""
|
||||||
|
logger = logging.getLogger(name)
|
||||||
|
logger.setLevel(level)
|
||||||
|
|
||||||
|
# Avoid adding duplicate handlers if logger already configured
|
||||||
|
if logger.handlers:
|
||||||
|
return logger
|
||||||
|
|
||||||
|
# Create formatter
|
||||||
|
formatter = logging.Formatter("[%(levelname)s] %(name)s: %(message)s")
|
||||||
|
|
||||||
|
# Console handler (always added)
|
||||||
|
console_handler = logging.StreamHandler()
|
||||||
|
console_handler.setLevel(level)
|
||||||
|
console_handler.setFormatter(formatter)
|
||||||
|
logger.addHandler(console_handler)
|
||||||
|
|
||||||
|
# File handler (optional)
|
||||||
|
if log_file:
|
||||||
|
file_handler = logging.FileHandler(log_file)
|
||||||
|
file_handler.setLevel(level)
|
||||||
|
file_handler.setFormatter(formatter)
|
||||||
|
logger.addHandler(file_handler)
|
||||||
|
|
||||||
|
# Prevent log propagation to root logger (avoids duplicate output)
|
||||||
|
logger.propagate = False
|
||||||
|
|
||||||
|
return logger
|
||||||
Reference in New Issue
Block a user