node-transfer

# node-transfer

High-speed, memory-efficient file transfer between OpenClaw nodes using native Node.js streams.

## 📋 Table of Contents

- [Problem Solved](#problem-solved)
- [Architecture](#architecture)
- [Requirements](#requirements)
- [Installation](#installation)
- [Usage](#usage)
- [API Reference](#api-reference)
- [Troubleshooting](#troubleshooting)

---

## 🎯 Problem Solved

### The Original Problem

When transferring large files between OpenClaw nodes using the standard `nodes.invoke` mechanism, we encountered several critical issues:

| Issue | Impact |
|-------|--------|
| **Base64 Encoding Overhead** | 33% larger payload, slower transfers |
| **Memory Exhaustion (OOM)** | Loading multi-GB files into memory crashes the process |
| **Transfer Latency** | JSON serialization/deserialization adds significant delay |
| **9-Minute Deployments** | Re-deploying scripts on every transfer |

### The Solution

`node-transfer` uses **native HTTP streaming** with Node.js streams, providing:

- ✅ **Zero memory overhead** - Files stream directly from disk to network
- ✅ **No Base64 encoding** - Raw binary transfer
- ✅ **Speed** - Line-speed limited only by network bandwidth
- ✅ **Install Once, Run Many** - Scripts persist on nodes after first deployment

### Performance Comparison

| Metric | Base64 Transfer | node-transfer | Improvement |
|--------|----------------|---------------|-------------|
| 1GB file transfer time | ~15-30 min | ~8 sec | **~150x faster** |
| Memory usage | 1GB+ | <10MB | **99% reduction** |
| First transfer overhead | N/A | ~30 sec (one-time install) | - |
| Subsequent transfers | ~15-30 min | **<1 sec** check + ~8 sec transfer | **~200x faster** |

---

## 🏗️ Architecture

### How It Works

```
┌──────────────┐     HTTP Stream      ┌──────────────┐
│  send.js     │ ◄──────────────────► │ receive.js   │
│  (Source)    │   (Token-protected)  │ (Destination)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  Read Stream │                      │ Write Stream │
│  (fs.create  │                      │ (fs.create   │
│   ReadStream)│                      │  WriteStream)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  File on     │                      │  File on     │
│  Disk        │                      │  Disk        │
└──────────────┘                      └──────────────┘
```

### Security Model

1. **One-time Token**: 256-bit cryptographically random token (64 hex chars)
2. **Single Connection**: Only one download allowed per token
3. **Auto-shutdown**: Server closes after transfer completes or disconnects
4. **Token Validation**: Every request must include the correct token

### Data Flow

1. **Sender** (`send.js`):
   - Generates random port and security token
   - Starts HTTP server on ephemeral port
   - Streams file directly from disk to HTTP response
   - Auto-shutdown after transfer or timeout (5 min default)

2. **Receiver** (`receive.js`):
   - Connects to sender URL with token
   - Streams HTTP response directly to disk
   - Reports progress, speed, and completion status
   - Validates received bytes match expected size

---

## 📦 Requirements

- **Node.js**: 14.0.0 or higher
- **Network**: TCP connectivity between nodes (any port 1024-65535)
- **Firewall**: Must allow outbound connections and inbound on ephemeral ports
- **Disk Space**: Sufficient space on destination for received files

---

## 🚀 Installation

### The "Install Once" Pattern

Instead of deploying scripts on every transfer, we deploy them **once per node** and use a fast version check for subsequent transfers.

### Method 1: Using deploy.js (Recommended)

```bash
# Generate deployment script for a target node
node deploy.js E3V3

# This outputs a PowerShell script that you can execute via nodes.invoke()
```

### Method 2: Manual Deployment

On each target node, create the directory and copy files:

```powershell
# Create directory
mkdir C:/openclaw/skills/node-transfer/scripts -Force

# Copy these files (ensure UTF-8 without BOM encoding):
# - send.js
# - receive.js
# - ensure-installed.js
# - version.js
```

### Method 3: Via OpenClaw Agent

```javascript
// 1. Check if already installed (< 100ms)
const check = await nodes.invoke({
    node: 'E3V3',
    command: ['node', 'C:/openclaw/skills/node-transfer/scripts/ensure-installed.js', 
              'C:/openclaw/skills/node-transfer/scripts']
});

const checkResult = JSON.parse(check.output);

if (!checkResult.installed) {
    // 2. Deploy if needed (one-time, ~30 seconds)
    // Use the deploy.js output or manually copy files
    console.log('Deploying node-transfer to E3V3...');
    // ... deployment code ...
}
```

---

## 💡 Usage

### Basic Transfer Workflow

```javascript
const INSTALL_DIR = 'C:/openclaw/skills/node-transfer/scripts';
const SOURCE_NODE = 'E3V3';
const DEST_NODE = 'E3V3-Docker';

// Step 1: Check installation on both nodes (fast!)
const [sourceCheck, destCheck] = await Promise.all([
    nodes.invoke({
        node: SOURCE_NODE,
        command: ['node', `${INSTALL_DIR}/ensure-installed.js`, INSTALL_DIR]
    }),
    nodes.invoke({
        node: DEST_NODE,
        command: ['node', `${INSTALL_DIR}/ensure-installed.js`, INSTALL_DIR]
    })
]);

// Deploy if needed (usually only once per node ever)
// ... deployment code if not installed ...

// Step 2: Start sender on source node
const sendResult = await nodes.invoke({
    node: SOURCE_NODE,
    command: ['node', `${INSTALL_DIR}/send.js`, 'C:/data/large-file.zip']
});

const { url, token, fileSize, fileName } = JSON.parse(sendResult.output);

// Step 3: Start receiver on destination node
const receiveResult = await nodes.invoke({
    node: DEST_NODE,
    command: ['node', `${INSTALL_DIR}/receive.js`, url, token, '/incoming/file.zip']
});

const result = JSON.parse(receiveResult.output);
console.log(`Transferred ${result.bytesReceived} bytes in ${result.duration}s at ${result.speedMBps} MB/s`);
```

### Using the Command Line

#### Sender

```bash
node send.js /path/to/file.zip
```

Output:
```json
{
  "url": "http://192.168.1.10:54321/transfer",
  "token": "a1b2c3d4e5f6789...",
  "fileSize": 1073741824,
  "fileName": "file.zip",
  "sourceIp": "192.168.1.10",
  "port": 54321,
  "version": "1.0.0"
}
```

Options:
```bash
node send.js /path/to/file.zip --port 8080 --timeout 10
node send.js --help
node send.js --version
```

#### Receiver

```bash
node receive.js "http://192.168.1.10:54321/transfer" "token-here..." /path/to/save.zip
```

Output:
```json
{
  "success": true,
  "bytesReceived": 1073741824,
  "totalBytes": 1073741824,
  "duration": 8.42,
  "speedMBps": 121.5,
  "outputPath": "/path/to/save.zip"
}
```

Options:
```bash
node receive.js <url> <token> <output> --timeout 60 --no-progress
node receive.js --help
node receive.js --version
```

---

## 📚 API Reference

### send.js

Starts an HTTP server to stream a file.

**Usage:** `node send.js <filePath> [options]`

**Arguments:**
- `filePath` (required): Path to the file to send

**Options:**
- `--port <n>`: Use specific port (default: random ephemeral)
- `--timeout <n>`: Timeout in minutes (default: 5)

**Output (JSON):**
| Field | Type | Description |
|-------|------|-------------|
| `url` | string | HTTP URL for receiver to connect to |
| `token` | string | Security token (64 hex chars) |
| `fileSize` | number | File size in bytes |
| `fileName` | string | Original filename |
| `sourceIp` | string | IP address of sender |
| `port` | number | TCP port used |
| `version` | string | Version of send.js |

**Exit Codes:**
- `0`: Success (transfer completed or info displayed)
- `1`: Error (check stderr for JSON error details)

**Error Output (JSON):**
```json
{
  "error": "ERROR_CODE",
  "message": "Human-readable description"
}
```

Error codes: `FILE_NOT_FOUND`, `NOT_A_FILE`, `SERVER_ERROR`, `TIMEOUT`, `READ_ERROR`, `RESPONSE_ERROR`

---

### receive.js

Connects to a sender and downloads a file.

**Usage:** `node receive.js <url> <token> <outputPath> [options]`

**Arguments:**
- `url` (required): URL from send.js output
- `token` (required): Security token from send.js output
- `outputPath` (required): Path to save the received file

**Options:**
- `--timeout <n>`: Connection timeout in seconds (default: 30)
- `--no-progress`: Suppress progress updates

**Output (JSON):**
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on success |
| `bytesReceived` | number | Actual bytes received |
| `totalBytes` | number | Expected bytes (from Content-Length) |
| `duration` | number | Transfer time in seconds |
| `speedMBps` | number | Average speed in MB/s |
| `outputPath` | string | Absolute path to saved file |

**Progress Updates (when not using `--no-progress`):**
```json
{
  "progress": true,
  "receivedBytes": 536870912,
  "totalBytes": 1073741824,
  "percent": 50,
  "speedMBps": 125.4
}
```

**Exit Codes:**
- `0`: Success
- `1`: Error (check stderr for JSON error details)

Error codes: `INVALID_ARGS`, `INVALID_URL`, `CONNECTION_ERROR`, `HTTP_ERROR`, `TIMEOUT`, `WRITE_ERROR`, `SIZE_MISMATCH`, `FILE_EXISTS`, `NO_DATA`

---

### ensure-installed.js

Fast check if node-transfer is installed on a node.

**Usage:** `node ensure-installed.js <targetDir>`

**Arguments:**
- `targetDir` (required): Directory to check

**Output (JSON):**

Installed:
```json
{
  "installed": true,
  "version": "1.0.0",
  "message": "node-transfer is installed and up-to-date"
}
```

Needs installation:
```json
{
  "installed": false,
  "missing": ["send.js"],
  "mismatched": [],
  "currentVersion": null,
  "requiredVersion": "1.0.0",
  "action": "DEPLOY",
  "message": "Installation needed: 1 missing, 0 outdated"
}
```

**Exit Codes:**
- `0`: Already installed and up-to-date
- `1`: Needs installation/update
- `2`: Error (invalid directory, etc.)

---

### deploy.js

Generates deployment scripts for the main agent.

**Usage:** `node deploy.js <nodeId> [targetDir]`

**Output:** JSON with:
- `script`: PowerShell script to deploy files
- `escapedScript`: Escaped version for command-line use
- `usage`: Example code for JavaScript and CLI usage

---

## 🔧 Troubleshooting

### "Connection timeout"

**Cause:** Network connectivity issue or firewall blocking connection.

**Solutions:**
- Verify both nodes can reach each other
- Check firewall rules allow outbound connections
- Try specifying a specific port with `--port`
- Increase timeout with `--timeout`

### "403 Forbidden: Invalid or missing token"

**Cause:** Token mismatch or URL manipulation.

**Solutions:**
- Use the exact token from send.js output
- Don't modify the URL
- Ensure the token hasn't expired (sender times out after 5 minutes)

### "409 Conflict: Transfer already in progress"

**Cause:** Multiple connections attempted with same token.

**Solutions:**
- Each sender URL/token can only be used once
- Start a new sender if you need to retry

### "FILE_NOT_FOUND" or "NOT_A_FILE"

**Cause:** Invalid file path on sender.

**Solutions:**
- Use absolute paths
- Verify file exists
- Check file permissions

### "SIZE_MISMATCH"

**Cause:** Connection interrupted or network error.

**Solutions:**
- Retry the transfer
- Check network stability
- The partial file is automatically cleaned up

### "Hash mismatch" during ensure-installed

**Cause:** Files were modified or corrupted.

**Solutions:**
- Re-deploy scripts using deploy.js
- Ensure files are copied without modification
- Check encoding (must be UTF-8 without BOM)

### Slow transfers on subsequent runs

**Cause:** Not using `ensure-installed.js` check pattern.

**Solutions:**
- Always check installation first (< 100ms)
- Only deploy if `installed: false`
- Follow the "Install Once, Run Many" pattern

---

## 📄 Files

| File | Purpose |
|------|---------|
| `send.js` | HTTP server that streams files to receivers |
| `receive.js` | HTTP client that downloads files from senders |
| `ensure-installed.js` | Fast version/integrity check for deployment |
| `version.js` | Version manifest for update detection |
| `deploy.js` | Generates deployment scripts for agents |

---

## 🤝 Contributing

See [CONTRIBUTING_PROPOSAL.md](./CONTRIBUTING_PROPOSAL.md) for information on how this could be integrated into OpenClaw core.

---

*Built for OpenClaw - No Base64, No OOM, No Waiting.*