# ISP360 Deployment Specification

**Version:** 1.0  
**Date:** 2026-06-13  
**Prepared by:** Polakis Technologies  

Companion: [PROJECT_STRUCTURE.md](PROJECT_STRUCTURE.md) | [DATABASE_DESIGN.md](DATABASE_DESIGN.md) | [RBAC_SPEC.md](RBAC_SPEC.md) | [PROJECT_RESET.md](PROJECT_RESET.md)

---

## 1. Overview

This document defines how ISP360 is **installed, configured, operated, backed up, monitored, and recovered** in production. It covers:

- Installation wizard (`/install/`)
- Server requirements and directory layout
- Nginx + PHP-FPM configuration
- Environment variables and secrets
- Cron job registry (consolidated from all module specs)
- Backup and restore procedures
- Monitoring and health checks
- Disaster recovery runbook
- Security hardening checklist

### Deployment Targets

| Target | Use Case |
|--------|----------|
| **Single server** | Small ISP (&lt; 5,000 customers) — all services on one VM |
| **Split stack** | Medium ISP — app + DB separate; FreeRADIUS on network node |
| **HA pair** | Large ISP / SaaS — load-balanced app tier, MySQL replica, Redis |

### Non-Goals (v1.0)

- Kubernetes / container orchestration (documented as future path only)
- Multi-region active-active
- Automated cross-cloud failover

---

## 2. Architecture

```mermaid
flowchart TB
    subgraph internet [Internet]
        Users["Staff / Customers / APIs"]
        Routers["MikroTik NAS"]
        Gateways["M-Pesa / MoMo Callbacks"]
    end

    subgraph lb [Edge]
        Nginx["Nginx — SSL termination"]
    end

    subgraph app [Application Tier]
        PHPFPM["PHP-FPM 8.3"]
        ISP360["ISP360 /var/www/isp360"]
        Cron["cron + systemd timers"]
    end

    subgraph data [Data Tier]
        MySQL["MySQL 8 — isp360"]
        Redis["Redis — optional cache"]
    end

    subgraph network [Network Services]
        FR["FreeRADIUS 3.x"]
        WG["WireGuard host"]
    end

    subgraph storage [Persistence]
        Uploads["/uploads"]
        Backups["/backups"]
        Logs["/logs"]
    end

    Users --> Nginx
    Gateways --> Nginx
    Nginx --> PHPFPM
    PHPFPM --> ISP360
    ISP360 --> MySQL
    ISP360 --> Redis
    Routers --> FR
    FR --> MySQL
    ISP360 --> FR
    Cron --> ISP360
    ISP360 --> Uploads
    Cron --> Backups
    ISP360 --> Logs
```

---

## 3. Server Requirements

### 3.1 Minimum (Single Server — Dev / Small ISP)

| Resource | Spec |
|----------|------|
| OS | Ubuntu 22.04 LTS or Rocky Linux 9 |
| CPU | 4 vCPU |
| RAM | 8 GB |
| Disk | 100 GB SSD |
| PHP | 8.3+ with extensions (see below) |
| MySQL | 8.0+ |
| Web | Nginx 1.22+ + PHP-FPM |

### 3.2 Recommended (Production — Medium ISP)

| Resource | Spec |
|----------|------|
| App server | 8 vCPU, 16 GB RAM |
| DB server | 8 vCPU, 32 GB RAM, 500 GB SSD |
| Redis | 2 GB RAM (optional) |
| FreeRADIUS | 2 vCPU, 4 GB (can co-locate on app for &lt; 2k sessions) |

### 3.3 PHP Extensions (Required)

```
pdo_mysql, mbstring, json, curl, openssl, sodium,
gd or imagick, zip, xml, fileinfo, intl
```

### 3.4 PHP Extensions (Recommended)

```
redis, opcache, apcu
```

### 3.5 External Services

| Service | Required | Notes |
|---------|----------|-------|
| MySQL 8 | Yes | Schema from `sql/database_schema.sql` |
| Nginx | Yes | Reverse proxy to PHP-FPM |
| FreeRADIUS 3.x | Yes (PPPoE/Hotspot) | Per [RADIUS_SPEC.md](RADIUS_SPEC.md) |
| Redis 7 | Optional | Session/cache/queue; falls back to DB |
| WireGuard | Optional | Per [WIREGUARD_SPEC.md](WIREGUARD_SPEC.md) |
| SMTP | Recommended | PHPMailer via `vendor/` |
| SMS gateway | Recommended | Africa's Talking, Twilio, etc. |

---

## 4. Directory Layout

Production install path: `/var/www/isp360`

| Path | Owner | Permissions | Purpose |
|------|-------|-------------|---------|
| `/var/www/isp360` | `root:www-data` | 755 | Application root |
| `/var/www/isp360/uploads` | `www-data` | 750 | Tenant/customer uploads |
| `/var/www/isp360/logs` | `www-data` | 750 | Application logs |
| `/var/www/isp360/backups` | `root` | 700 | Backup output (cron) |
| `/var/www/isp360/install` | `root` | 750 | Wizard (disable after install) |
| `/etc/isp360` | `root` | 600 | `env.php` secrets (outside web root) |

**Never web-accessible:** `/includes/`, `/sql/`, `/logs/`, `/backups/`, `/vendor/`, `/etc/isp360/`

---

## 5. Installation Wizard

**Entry:** `/install/install.php`  
**Lock:** Create `/var/www/isp360/install/.installed` on completion; wizard refuses to run if file exists.

### 5.1 Wizard Steps

```mermaid
flowchart LR
    S1[1. Requirements] --> S2[2. Database]
    S2 --> S3[3. Schema Import]
    S3 --> S4[4. Redis optional]
    S4 --> S5[5. Super Admin]
    S5 --> S6[6. First Tenant]
    S6 --> S7[7. Complete]
```

| Step | File | Actions |
|------|------|---------|
| 1. Requirements | `install/steps/requirements.php` | PHP version, extensions, writable dirs, `sql/database_schema.sql` exists |
| 2. Database | `install/steps/database.php` | Test PDO connection; write `/etc/isp360/env.php` |
| 3. Schema | (inline in database step) | Execute `sql/database_schema.sql`; verify table count ≈ 210 |
| 4. Redis | `install/steps/redis.php` (optional) | Test Redis PING; store host/port in `platform_settings` |
| 5. Super Admin | `install/steps/admin.php` | Create platform user (`tenant_id NULL`), `super_admin` role |
| 6. First Tenant | `install/steps/tenant.php` | Create tenant, admin domain, default `tenant_themes`, seed portal/captive themes |
| 7. Complete | `install/steps/complete.php` | Write `.installed`; display cron setup instructions; **delete or block `/install/`** |

### 5.2 Generated Configuration

**`/etc/isp360/env.php`** (included by `includes/config.php`):

```php
<?php
return [
    'db_host'     => '127.0.0.1',
    'db_port'     => '3306',
    'db_name'     => 'isp360',
    'db_user'     => 'isp360_app',
    'db_pass'     => '<generated>',
    'redis_host'  => '127.0.0.1',
    'redis_port'  => 6379,
    'app_key'     => '<32-byte random>',  // encryption key seed
    'app_env'     => 'production',
];
```

Environment variables override file values:

| Variable | Purpose |
|----------|---------|
| `ISP360_DB_HOST` | MySQL host |
| `ISP360_DB_NAME` | Database name |
| `ISP360_DB_USER` | DB user |
| `ISP360_DB_PASS` | DB password |
| `ISP360_REDIS_HOST` | Redis host |
| `ISP360_APP_KEY` | Encryption master key |
| `ISP360_APP_ENV` | `production` / `staging` / `local` |

### 5.3 Database User Privileges

```sql
CREATE USER 'isp360_app'@'localhost' IDENTIFIED BY '...';
GRANT SELECT, INSERT, UPDATE, DELETE ON isp360.* TO 'isp360_app'@'localhost';
-- FreeRADIUS sql user (separate):
CREATE USER 'radius'@'localhost' IDENTIFIED BY '...';
GRANT SELECT ON isp360.radcheck TO 'radius'@'localhost';
GRANT SELECT ON isp360.radreply TO 'radius'@'localhost';
-- (full grants per RADIUS_SPEC.md)
FLUSH PRIVILEGES;
```

### 5.4 Post-Install Checklist

- [ ] Wizard shows green on all requirements
- [ ] Super Admin can log in at `login.php`
- [ ] `dashboard.php` loads with KPI cards
- [ ] `/install/` returns 403 or is removed
- [ ] Cron entries added (Section 7)
- [ ] Nginx vhost active with SSL
- [ ] `api/v1/health.php` returns `{"status":"ok"}`
- [ ] First backup script tested manually

---

## 6. Nginx Configuration

### 6.1 Admin + API Vhost

**File:** `/etc/nginx/sites-available/isp360.conf`

```nginx
upstream isp360_php {
    server unix:/run/php/php8.3-fpm-isp360.sock;
}

# Rate limit zones
limit_req_zone $binary_remote_addr zone=login_limit:10m rate=10r/m;
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=60r/m;
limit_req_zone $binary_remote_addr zone=callback_limit:10m rate=120r/m;

server {
    listen 443 ssl http2;
    server_name admin.isp360.example.com;

    root /var/www/isp360;
    index index.php;

    ssl_certificate     /etc/letsencrypt/live/admin.isp360.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/admin.isp360.example.com/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    # Block sensitive paths
    location ~ ^/(includes|sql|logs|backups|vendor|install|cron)/ {
        deny all;
        return 404;
    }

    location ~ /\. {
        deny all;
    }

    # Static assets — long cache
    location /assets/ {
        expires 30d;
        add_header Cache-Control "public, immutable";
        try_files $uri =404;
    }

    location /uploads/ {
        internal;  # only X-Accel-Redirect from PHP
    }

    # Login rate limit
    location = /login.php {
        limit_req zone=login_limit burst=5 nodelay;
        include fastcgi_params;
        fastcgi_pass isp360_php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }

    # Payment callbacks — higher limit, no auth
    location ~ ^/api/v1/payments/callback/ {
        limit_req zone=callback_limit burst=20 nodelay;
        try_files $uri $uri/ /api/v1/payments/callback/$1.php?$query_string;
    }

    # API rate limit
    location /api/ {
        limit_req zone=api_limit burst=30 nodelay;
        try_files $uri $uri/ $uri.php?$query_string;
    }

    # PHP front controller — flat root pages
    location / {
        try_files $uri $uri/ $uri.php?$query_string;
    }

    location ~ \.php$ {
        include fastcgi_params;
        fastcgi_pass isp360_php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_read_timeout 120s;
        fastcgi_buffers 16 16k;
    }
}
```

### 6.2 Multi-Domain Tenants

Per-tenant domains from `tenant_domains`:

| `domain_type` | Example | Root behavior |
|---------------|---------|---------------|
| `admin` | `admin.acmeisp.co.ke` | Staff UI (`login.php`, `dashboard.php`) |
| `portal` | `my.acmeisp.co.ke` | `portal_login.php` default |
| `captive` | `wifi.acmeisp.co.ke` | `captive.php` default |
| `api` | `api.acmeisp.co.ke` | `/api/v1/` only |

**Wildcard vhost** (optional):

```nginx
server {
    listen 443 ssl http2;
    server_name *.acmeisp.co.ke;
    # Same root; PHP resolves tenant from Host header
    ...
}
```

### 6.3 PHP-FPM Pool

**File:** `/etc/php/8.3/fpm/pool.d/isp360.conf`

```ini
[isp360]
user = www-data
group = www-data
listen = /run/php/php8.3-fpm-isp360.sock
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 20
pm.max_requests = 500
php_admin_value[upload_max_filesize] = 32M
php_admin_value[post_max_size] = 32M
php_admin_value[memory_limit] = 256M
php_admin_value[max_execution_time] = 120
```

---

## 7. Cron Job Registry

All jobs run as `www-data` or dedicated `isp360-cron` user from `/var/www/isp360/cron/`.

**Crontab file:** `/etc/cron.d/isp360`

```cron
# ISP360 — see docs/DEPLOYMENT_SPEC.md
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin

# ── Every minute ─────────────────────────────────────────
*  *  * * * www-data /usr/bin/php /var/www/isp360/cron/router_command_processor.php >> /var/www/isp360/logs/cron.log 2>&1
*  *  * * * www-data /usr/bin/php /var/www/isp360/cron/notification_queue.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Every 2 minutes ──────────────────────────────────────
*/2 * * * * www-data /usr/bin/php /var/www/isp360/cron/session_sync.php >> /var/www/isp360/logs/cron.log 2>&1
*/2 * * * * www-data /usr/bin/php /var/www/isp360/cron/vpn_status_sync.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Every 5 minutes ──────────────────────────────────────
*/5 * * * * www-data /usr/bin/php /var/www/isp360/cron/router_health.php >> /var/www/isp360/logs/cron.log 2>&1
*/5 * * * * www-data /usr/bin/php /var/www/isp360/cron/radius_sync.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Every 10 minutes ─────────────────────────────────────
*/10 * * * * www-data /usr/bin/php /var/www/isp360/cron/payment_status_check.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Every 15 minutes ─────────────────────────────────────
*/15 * * * * www-data /usr/bin/php /var/www/isp360/cron/payment_reconcile.php >> /var/www/isp360/logs/cron.log 2>&1
*/15 * * * * www-data /usr/bin/php /var/www/isp360/cron/dashboard_metrics.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Hourly ───────────────────────────────────────────────
0 * * * * www-data /usr/bin/php /var/www/isp360/cron/radius_accounting_rollup.php >> /var/www/isp360/logs/cron.log 2>&1
0 * * * * www-data /usr/bin/php /var/www/isp360/cron/cleanup_sessions.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Every 4 hours ──────────────────────────────────────────
0 */4 * * * www-data /usr/bin/php /var/www/isp360/cron/report_cache.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Daily ──────────────────────────────────────────────────
0 0 * * * www-data /usr/bin/php /var/www/isp360/cron/voucher_expiry.php >> /var/www/isp360/logs/cron.log 2>&1
0 1 * * * www-data /usr/bin/php /var/www/isp360/cron/invoice_generate.php >> /var/www/isp360/logs/cron.log 2>&1
0 2 * * * www-data /usr/bin/php /var/www/isp360/cron/router_backup.php >> /var/www/isp360/logs/cron.log 2>&1
0 3 * * * www-data /usr/bin/php /var/www/isp360/cron/payment_idempotency_cleanup.php >> /var/www/isp360/logs/cron.log 2>&1
0 4 * * * www-data /usr/bin/php /var/www/isp360/cron/tenant_usage_snapshot.php >> /var/www/isp360/logs/cron.log 2>&1
0 6 * * * www-data /usr/bin/php /var/www/isp360/cron/billing_run.php >> /var/www/isp360/logs/cron.log 2>&1
0 8 * * * www-data /usr/bin/php /var/www/isp360/cron/dunning_process.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Weekly ─────────────────────────────────────────────────
0 3 * * 0 www-data /usr/bin/php /var/www/isp360/cron/subscription_renewal.php >> /var/www/isp360/logs/cron.log 2>&1
0 5 * * 0 www-data /usr/bin/php /var/www/isp360/cron/audit_archive.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Monthly ────────────────────────────────────────────────
0 0 1 * * www-data /usr/bin/php /var/www/isp360/cron/vpn_key_rotation.php >> /var/www/isp360/logs/cron.log 2>&1

# ── Backups (root) ─────────────────────────────────────────
30 2 * * * root /var/www/isp360/backups/backup_database.sh >> /var/www/isp360/logs/backup.log 2>&1
0 3 * * * root /var/www/isp360/backups/backup_uploads.sh >> /var/www/isp360/logs/backup.log 2>&1
30 3 * * 0 root /var/www/isp360/backups/backup_router_configs.sh >> /var/www/isp360/logs/backup.log 2>&1
```

### 7.1 Cron Job Index

| Script | Schedule | Spec Reference |
|--------|----------|----------------|
| `router_command_processor.php` | `* * * * *` | [MIKROTIK_INTEGRATION.md](MIKROTIK_INTEGRATION.md) |
| `notification_queue.php` | `* * * * *` | Notifications module |
| `session_sync.php` | `*/2 * * * *` | [RADIUS_SPEC.md](RADIUS_SPEC.md) |
| `vpn_status_sync.php` | `*/2 * * * *` | [WIREGUARD_SPEC.md](WIREGUARD_SPEC.md) |
| `router_health.php` | `*/5 * * * *` | [MIKROTIK_INTEGRATION.md](MIKROTIK_INTEGRATION.md) |
| `radius_sync.php` | `*/5 * * * *` | [RADIUS_SPEC.md](RADIUS_SPEC.md) |
| `payment_status_check.php` | `*/10 * * * *` | [PAYMENTS_SPEC.md](PAYMENTS_SPEC.md) |
| `payment_reconcile.php` | `*/15 * * * *` | [PAYMENTS_SPEC.md](PAYMENTS_SPEC.md) |
| `dashboard_metrics.php` | `*/15 * * * *` | Dashboard |
| `radius_accounting_rollup.php` | `0 * * * *` | [RADIUS_SPEC.md](RADIUS_SPEC.md) |
| `cleanup_sessions.php` | `0 * * * *` | Staff + portal + captive sessions |
| `report_cache.php` | `0 */4 * * *` | [REPORTING_SPEC.md](REPORTING_SPEC.md) |
| `invoice_generate.php` | `0 1 * * *` | Billing |
| `router_backup.php` | `0 2 * * *` | [MIKROTIK_INTEGRATION.md](MIKROTIK_INTEGRATION.md) |
| `billing_run.php` | `0 6 * * *` | Billing |
| `dunning_process.php` | `0 8 * * *` | Billing |
| `voucher_expiry.php` | `0 0 * * *` | Vouchers |
| `subscription_renewal.php` | `0 3 * * 0` | [CUSTOMER_PORTAL_SPEC.md](CUSTOMER_PORTAL_SPEC.md) |
| `vpn_key_rotation.php` | `0 0 1 * *` | [WIREGUARD_SPEC.md](WIREGUARD_SPEC.md) |
| `audit_archive.php` | `0 5 * * 0` | Audit |
| `tenant_usage_snapshot.php` | `0 4 * * *` | SaaS metering |

### 7.2 Systemd Alternative (Recommended for Production)

**File:** `/etc/systemd/system/isp360-cron@.service`

```ini
[Unit]
Description=ISP360 Cron %i
After=network.target mysql.service

[Service]
Type=oneshot
User=www-data
WorkingDirectory=/var/www/isp360
ExecStart=/usr/bin/php /var/www/isp360/cron/%i.php
StandardOutput=append:/var/www/isp360/logs/cron-%i.log
StandardError=append:/var/www/isp360/logs/cron-%i.log
```

Enable timers matching cron schedules for critical jobs (`router_command_processor`, `notification_queue`).

### 7.3 Cron Wrapper Template

Each `cron/*.php` file:

```php
<?php
// Prevent web access
if (php_sapi_name() !== 'cli') {
    exit('CLI only');
}

require_once __DIR__ . '/../includes/config.php';
require_once __DIR__ . '/../includes/db.php';

$lock_file = sys_get_temp_dir() . '/isp360_' . basename(__FILE__) . '.lock';
$fp = fopen($lock_file, 'c');
if (!flock($fp, LOCK_EX | LOCK_NB)) {
    exit(0);  // already running
}

try {
    // job logic
} catch (Throwable $e) {
    error_log('Cron ' . basename(__FILE__) . ': ' . $e->getMessage());
    exit(1);
} finally {
    flock($fp, LOCK_UN);
}
```

---

## 8. Backup Strategy

### 8.1 What to Back Up

| Asset | Method | Frequency | Retention |
|-------|--------|-----------|-----------|
| MySQL `isp360` | `mysqldump` + binlog | Daily 02:30 | 30 daily, 12 monthly |
| `/uploads/` | `tar.gz` | Daily 03:00 | 14 daily |
| Router configs | `backup_router_configs.sh` | Weekly | 8 weekly |
| `/etc/isp360/env.php` | Manual / config mgmt | On change | Forever |
| Nginx + FreeRADIUS configs | Git / Ansible | On change | Version controlled |

### 8.2 `backups/backup_database.sh`

```bash
#!/bin/bash
set -euo pipefail
BACKUP_DIR="/var/backups/isp360/db"
DATE=$(date +%Y%m%d_%H%M%S)
RETAIN_DAYS=30
mkdir -p "$BACKUP_DIR"

source /etc/isp360/backup.env  # DB credentials

mysqldump \
  --single-transaction \
  --routines --triggers \
  --host="$DB_HOST" --user="$DB_USER" --password="$DB_PASS" \
  "$DB_NAME" | gzip > "$BACKUP_DIR/isp360_${DATE}.sql.gz"

find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETAIN_DAYS -delete
```

### 8.3 `backups/restore_database.sh`

```bash
#!/bin/bash
# Usage: restore_database.sh /var/backups/isp360/db/isp360_20260613.sql.gz
set -euo pipefail
BACKUP_FILE="${1:?Backup file required}"
source /etc/isp360/backup.env

echo "STOPPING: Put app in maintenance mode first."
gunzip -c "$BACKUP_FILE" | mysql -h"$DB_HOST" -u"$DB_USER" -p"$DB_PASS" "$DB_NAME"
echo "Restore complete. Verify and remove maintenance mode."
```

### 8.4 Off-Site Backup

- Sync `/var/backups/isp360/` to S3-compatible storage (rclone, AWS CLI)
- Encrypt dumps at rest (GPG or SSE-S3)
- Test restore quarterly

### 8.5 RPO / RTO Targets

| Tier | RPO | RTO |
|------|-----|-----|
| Database | 24 hours (daily dump) + binlog for point-in-time | 4 hours |
| Uploads | 24 hours | 2 hours |
| Application code | Git deploy | 1 hour |
| Full stack DR | 24 hours | 8 hours |

---

## 9. Monitoring & Health

### 9.1 Health Endpoint

**URL:** `GET /api/v1/health.php`

```json
{
  "status": "ok",
  "version": "1.0.0",
  "checks": {
    "database": "ok",
    "redis": "ok",
    "disk_free_mb": 45000,
    "cron_last_run": {
      "router_command_processor": "2026-06-13T10:01:00+03:00",
      "notification_queue": "2026-06-13T10:01:00+03:00"
    }
  },
  "timestamp": "2026-06-13T10:05:00+03:00"
}
```

Returns HTTP 503 if database unreachable.

### 9.2 Metrics to Monitor

| Metric | Source | Alert Threshold |
|--------|--------|-----------------|
| HTTP 5xx rate | Nginx access log | &gt; 1% over 5 min |
| PHP-FPM queue | `pm.status` | &gt; 10 waiting |
| MySQL connections | `SHOW STATUS` | &gt; 80% max |
| Disk usage | `df` | &gt; 85% |
| Cron staleness | `logs/cron.log` mtime | Job not run in 2× interval |
| Router offline % | `vw_router_health` | &gt; 20% offline |
| Payment callback failures | `payment_callbacks` | &gt; 10 failed/hour |
| SSL expiry | certbot | &lt; 14 days |

### 9.3 Log Rotation

**File:** `/etc/logrotate.d/isp360`

```
/var/www/isp360/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 0640 www-data www-data
    sharedscripts
    postrotate
        systemctl reload php8.3-fpm > /dev/null 2>&1 || true
    endscript
}
```

### 9.4 Application Logging

| Log file | Content |
|----------|---------|
| `logs/app.log` | PHP errors, uncaught exceptions |
| `logs/cron.log` | Cron stdout aggregate |
| `logs/backup.log` | Backup script output |
| `logs/payment_callbacks.log` | Gateway callback debug (optional) |
| `logs/audit.log` | Mirror of critical `audit_logs` (optional) |

`includes/config.php` sets `error_log` to `logs/app.log` in production; `display_errors = Off`.

---

## 10. FreeRADIUS Deployment

Per [RADIUS_SPEC.md](RADIUS_SPEC.md):

| Component | Location |
|-----------|----------|
| FreeRADIUS | `/etc/freeradius/3.0/` |
| SQL module | Points to `isp360` MySQL |
| NAS entries | Synced from `nas` table via ISP360 |
| Firewall | UDP 1812/1813 from MikroTik NAS IPs only |

**Co-location:** FreeRADIUS on same host as MySQL for small deployments; separate NIC on network segment for large ISPs.

---

## 11. WireGuard Deployment

Per [WIREGUARD_SPEC.md](WIREGUARD_SPEC.md):

| Component | Notes |
|-----------|-------|
| WireGuard host | Linux VM with `wg` tools |
| ISP360 | Manages peers via SSH or RouterManager |
| Firewall | UDP 51820 (or per-server port) |
| Key storage | Encrypted in DB; `ISP360_APP_KEY` required |

Optional **systemd unit:** `wg-quick@wg0` enabled on VPN concentrator (not on app server).

---

## 12. Redis (Optional)

When enabled:

| Use | Key pattern | TTL |
|-----|-------------|-----|
| Report cache | `rpt:{tenant}:{code}:{hash}` | Per [REPORTING_SPEC.md](REPORTING_SPEC.md) |
| Session cache | `sess:{token_hash}` | 8 hours |
| Rate limiting | `rl:{ip}:{endpoint}` | 15 min |
| Idempotency | `idem:{key}` | 24 hours |

Fallback: all caches use `report_cache` table and DB sessions if Redis unavailable.

---

## 13. Security Hardening

### 13.1 Production Checklist

- [ ] Remove or 403 `/install/` after setup
- [ ] `ISP360_APP_ENV=production`; `display_errors=Off`
- [ ] Secrets in `/etc/isp360/env.php` — not in git
- [ ] MySQL: separate app + radius users; least privilege
- [ ] Nginx: block `/includes`, `/sql`, `/cron`, `/vendor`
- [ ] SSL on all vhosts (Let's Encrypt certbot)
- [ ] Firewall: 443 public; 1812/1813 from NAS only; 8729 router API from app server only
- [ ] Fail2ban on SSH and Nginx login limit
- [ ] PHP-FPM `security.limit_extensions = .php`
- [ ] Disable `phpinfo`, unused PHP modules
- [ ] Regular OS security updates (unattended-upgrades)

### 13.2 Maintenance Mode

**File:** `/var/www/isp360/maintenance.flag`

When present, `includes/config.php` shows maintenance page for non–Super Admin users.

```php
if (file_exists(ISP360_BASE_PATH . '/maintenance.flag') && !role_check('super_admin')) {
    http_response_code(503);
    include ISP360_BASE_PATH . '/layouts/maintenance.php';
    exit;
}
```

Used during DB restore and major upgrades.

---

## 14. Disaster Recovery Runbook

### 14.1 Scenario: Database Corruption / Accidental Drop

1. Enable maintenance mode (`touch maintenance.flag`)
2. Stop cron: `systemctl stop cron` or comment `/etc/cron.d/isp360`
3. Identify latest good backup: `ls -lt /var/backups/isp360/db/`
4. Run `backups/restore_database.sh <backup.sql.gz>`
5. Verify: table count, Super Admin login, sample tenant data
6. Re-enable cron; remove maintenance flag
7. Document incident in `audit_logs`

**Estimated RTO:** 2–4 hours

### 14.2 Scenario: Application Server Loss

1. Provision new VM (same OS/PHP spec)
2. Restore code from git release tag
3. Copy `/etc/isp360/env.php` from secure backup
4. Restore `/uploads/` from latest tar backup
5. Configure Nginx + PHP-FPM per Section 6
6. Point DNS to new server
7. Verify health endpoint and login

**Estimated RTO:** 2–4 hours (excluding DNS propagation)

### 14.3 Scenario: Full Site Loss (DB + App)

1. Provision app + DB servers
2. Restore MySQL from off-site backup
3. Deploy application + config
4. Restore uploads
5. Re-register payment callback URLs with gateways (M-Pesa, MoMo)
6. Verify FreeRADIUS NAS secrets match `nas` table
7. Run `radius_sync.php` and `router_health.php` manually
8. Notify tenants of maintenance window completion

**Estimated RTO:** 8 hours

### 14.4 Scenario: Ransomware / Compromise

1. Isolate affected servers (disconnect network)
2. Rotate all secrets: DB passwords, API keys, `ISP360_APP_KEY`, gateway credentials
3. Restore from known-good backup **predating** compromise
4. Force password reset for all staff users
5. Invalidate all `sessions`, `portal_sessions`, `api_tokens`
6. Security audit before re-opening

---

## 15. Upgrades & Migrations

### 15.1 Application Upgrade

```bash
cd /var/www/isp360
git fetch --tags
git checkout v1.1.0
composer install --no-dev  # if composer added later
touch maintenance.flag
# Run sql migrations if any
php install/migrations/migrate.php   # future
php cron/dashboard_metrics.php      # warm caches
rm maintenance.flag
```

### 15.2 Schema Migrations

- v1.0: Full schema via `sql/database_schema.sql` at install
- v1.1+: Incremental files in `install/migrations/YYYYMMDD_description.sql`
- Track applied migrations in `platform_settings.schema_version`

### 15.3 Zero-Downtime (Future)

- Blue/green: two app servers behind load balancer; migrate DB first
- Read replica: promote replica on primary failure (documented in HA appendix)

---

## 16. Environment Matrix

| Setting | Local | Staging | Production |
|---------|-------|---------|------------|
| `ISP360_APP_ENV` | local | staging | production |
| `display_errors` | On | Off | Off |
| SSL | Self-signed | Let's Encrypt | Let's Encrypt |
| Redis | Optional | Yes | Yes |
| Backups | Manual | Daily | Daily + off-site |
| Cron | Subset | Full | Full |
| Payment gateways | Sandbox | Sandbox | Production |
| Install wizard | Enabled | Disabled | Disabled |

---

## 17. Implementation Checklist (Wave 6)

- [ ] `install/install.php` + 6 step files
- [ ] `/etc/isp360/env.php` loader in `includes/config.php`
- [ ] Nginx vhost template in `docs/nginx/isp360.conf.example`
- [ ] PHP-FPM pool template
- [ ] `/etc/cron.d/isp360` template
- [ ] `backups/*.sh` scripts (4 files)
- [ ] `api/v1/health.php` implementation
- [ ] `layouts/maintenance.php`
- [ ] Cron lock wrapper pattern in all 20+ cron files
- [ ] Logrotate config
- [ ] DR runbook tested (tabletop + one restore drill)
- [ ] Security hardening checklist sign-off

---

## 18. Spec Review Checkpoint

With Phase 12 complete, all **12 specification documents** are delivered:

| Phase | Deliverable | Status |
|-------|-------------|--------|
| 0 | [PROJECT_RESET.md](PROJECT_RESET.md) | ✅ |
| 1 | [DATABASE_DESIGN.md](DATABASE_DESIGN.md) + `sql/database_schema.sql` | ✅ |
| 2 | [PROJECT_STRUCTURE.md](PROJECT_STRUCTURE.md) | ✅ |
| 3 | [UI_SPECIFICATIONS.md](UI_SPECIFICATIONS.md) | ✅ |
| 4 | [RBAC_SPEC.md](RBAC_SPEC.md) | ✅ |
| 5 | [MIKROTIK_INTEGRATION.md](MIKROTIK_INTEGRATION.md) | ✅ |
| 6 | [RADIUS_SPEC.md](RADIUS_SPEC.md) | ✅ |
| 7 | [WIREGUARD_SPEC.md](WIREGUARD_SPEC.md) + UI shell | ✅ |
| 8 | [PAYMENTS_SPEC.md](PAYMENTS_SPEC.md) | ✅ |
| 9 | [CAPTIVE_PORTAL_SPEC.md](CAPTIVE_PORTAL_SPEC.md) | ✅ |
| 10 | [CUSTOMER_PORTAL_SPEC.md](CUSTOMER_PORTAL_SPEC.md) | ✅ |
| 11 | [REPORTING_SPEC.md](REPORTING_SPEC.md) | ✅ |
| 12 | [DEPLOYMENT_SPEC.md](DEPLOYMENT_SPEC.md) | ✅ |

**Next step:** Spec review checkpoint per build plan — validate all deliverables against quality checklist before Implementation Wave 1.

---

## 19. Related Documents

| Document | Relevance |
|----------|-----------|
| [PROJECT_STRUCTURE.md](PROJECT_STRUCTURE.md) | Cron, install, backup file manifest |
| [RADIUS_SPEC.md](RADIUS_SPEC.md) | FreeRADIUS deployment |
| [MIKROTIK_INTEGRATION.md](MIKROTIK_INTEGRATION.md) | Router API firewall rules |
| [PAYMENTS_SPEC.md](PAYMENTS_SPEC.md) | Callback URLs, cron |
| [UI_SHELL_PATTERN.md](UI_SHELL_PATTERN.md) | Application entry points |
| [RBAC_SPEC.md](RBAC_SPEC.md) | Super Admin install user |
