1.1 Purpose

This manual provides comprehensive documentation for the GetSetLive Streaming Server platform. It covers the technology stack, architecture decisions, implementation details, testing procedures, deployment process, and operational guidelines.

The platform is designed to provide professional-grade live streaming capabilities with adaptive bitrate (ABR) transcoding, multi-protocol support, and enterprise security features.

1.2 Scope

This documentation covers:

• Complete technology stack with feature explanations
• System architecture and planning decisions
• Step-by-step implementation details
• API reference with curl examples and sample outputs
• Testing and validation procedures
• Production deployment guide
• Operational best practices and troubleshooting

1.3 System Overview

GetSetLive is a complete live streaming solution that enables:

2.1 SRS Media Server (v7.0.94)

What it is: Simple Realtime Server (SRS) is a high-performance, open-source media server written in C++ for live streaming.

Key Features:

• RTMP Ingest: Accepts live streams from OBS, FFmpeg, hardware encoders
• HLS Output: Generates HTTP Live Streaming segments for broad device compatibility
• HTTP-FLV: Low-latency Flash video streaming over HTTP
• WebRTC: Ultra-low latency peer-to-peer streaming
• HTTP Callback: Notifies backend on stream publish/unpublish events
• Cluster Support: Edge-origin architecture for scaling

Configuration Location:

/opt/streaming-server/configs/srs.conf

Service Management:

# Check SRS status
$ sudo systemctl status srs
● srs.service - SRS Media Server
     Loaded: loaded (/etc/systemd/system/srs.service; enabled)
     Active: active (running) since Thu 2025-12-05 00:00:20 UTC
   Main PID: 1147206 (srs)
     Memory: 45.2M

# View SRS version via API
$ curl -s http://127.0.0.1:1985/api/v1/versions
{
  "code": 0,
  "server": "vid-583u578",
  "data": {
    "major": 7,
    "minor": 0,
    "revision": 94,
    "version": "7.0.94"
  }
}

Why SRS was chosen:

• High performance: Handles 10,000+ concurrent connections
• Low memory footprint compared to alternatives
• Native HLS support without additional transcoding
• Active development and community support
• HTTP callback integration for custom authentication

2.2 FFmpeg Transcoder (v6.1.1)

What it is: FFmpeg is the industry-standard multimedia framework for encoding, decoding, and transcoding audio/video.

Key Features:

• H.264/H.265 Encoding: Hardware-accelerated video compression
• ABR Ladder Generation: Creates multiple quality renditions from single input
• AAC Audio: High-quality audio encoding
• HLS Segmenting: Generates .ts segments and .m3u8 playlists

ABR Quality Profiles:

FFmpeg Command Example:

# ABR transcoding command (simplified)
ffmpeg -i rtmp://localhost/live/stream_key \
  -map 0:v -map 0:a -c:v libx264 -preset veryfast \
  -b:v:0 5000k -s:v:0 1920x1080 \
  -b:v:1 2800k -s:v:1 1280x720 \
  -b:v:2 1400k -s:v:2 854x480 \
  -c:a aac -b:a 128k \
  -f hls -hls_time 2 -hls_list_size 10 \
  -master_pl_name master.m3u8 \
  /opt/streaming-server/hls/stream_key/playlist.m3u8

2.3 Node.js Backend (v20.19.5)

What it is: Node.js powers both the Streaming API and Authentication Service, providing RESTful endpoints for all platform operations.

Services:

Key Features:

• Express.js Framework: Fast, minimalist web framework
• JWT Authentication: Stateless token-based auth with RS256 signing
• Redis Integration: Session storage and real-time state
• SRS HTTP Callback: Handles stream events (on_publish, on_unpublish)
• System Monitoring: CPU, memory, disk, network statistics
• Service Management: Start/stop/restart systemd services

Service Management:

# Check API service status
$ sudo systemctl status streaming-api
● streaming-api.service - Streaming API Server
     Active: active (running)
     Memory: 78.5M

# View API logs
$ sudo journalctl -u streaming-api -f
Dec 05 10:30:15 server streaming-api: [INFO] Server listening on port 1987
Dec 05 10:30:16 server streaming-api: [INFO] Redis connected successfully

2.4 Caddy Web Server (v2.10.2)

What it is: Caddy is a modern web server with automatic HTTPS, serving as the reverse proxy and static file server.

Key Features:

• Automatic HTTPS: Auto-obtains and renews Let's Encrypt certificates
• Reverse Proxy: Routes requests to backend services
• WAF Integration: Coraza WAF module for security
• Static Files: Serves admin dashboard and HLS segments
• HTTP/2 & HTTP/3: Modern protocol support

Configuration:

# /etc/caddy/Caddyfile structure
stream.getsetlive.com {
    # WAF protection
    route {
        coraza_waf {
            load_owasp_crs
            directives `
                SecRuleEngine On
                SecRule REQUEST_URI "^/api/admin/" "id:1006,phase:1,pass,nolog,ctl:ruleEngine=Off"
            `
        }
    }

    # API routing
    handle /api/auth/* {
        reverse_proxy 127.0.0.1:1988
    }
    handle /api/* {
        reverse_proxy 127.0.0.1:1987
    }

    # HLS streaming
    handle /hls/* {
        root * /opt/streaming-server
        file_server
        header Access-Control-Allow-Origin "*"
    }

    # Admin dashboard
    handle /admin* {
        root * /opt/streaming-server/frontend
        try_files {path} /admin.html
        file_server
    }
}

2.5 Redis Database (v7.0.15)

What it is: Redis is an in-memory data store used for session management, caching, and real-time state.

Use Cases:

Verification:

# Test Redis connection
$ redis-cli ping
PONG

# View active sessions
$ redis-cli keys "session:*"
1) "session:admin:1733385600000"

# Check memory usage
$ redis-cli info memory | grep used_memory_human
used_memory_human:2.45M

2.6 Security Stack

2.6.1 Coraza WAF

Web Application Firewall integrated with Caddy, using OWASP Core Rule Set (CRS).

• SQL Injection protection
• Cross-Site Scripting (XSS) prevention
• Path traversal blocking
• Request rate limiting

2.6.2 CSF Firewall

ConfigServer Security & Firewall for network-level protection.

# Check CSF status
$ sudo csf -l | head -20
iptables filter table
Chain INPUT (policy DROP)
num  target     prot opt source     destination
1    ACCEPT     tcp  --  anywhere   anywhere   tcp dpt:https
2    ACCEPT     tcp  --  anywhere   anywhere   tcp dpt:1935

2.6.3 JWT Authentication

Token-based authentication with configurable expiry and refresh mechanisms.

# JWT Token Structure
{
  "header": {
    "alg": "HS256",
    "typ": "JWT"
  },
  "payload": {
    "username": "admin",
    "role": "admin",
    "iat": 1733385600,
    "exp": 1733472000
  }
}

3.1 System Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                         INTERNET                                        │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                    ┌───────────────┼───────────────┐
                    │               │               │
              RTMP :1935      HTTPS :443      WebRTC
                    │               │               │
┌───────────────────┴───────────────┴───────────────┴───────────────────┐
│                        CSF FIREWALL                                    │
└───────────────────────────────────────────────────────────────────────┘
                    │               │
                    ▼               ▼
            ┌───────────┐   ┌─────────────────┐
            │    SRS    │   │     CADDY       │
            │  :1935    │   │     :443        │
            │  :1985    │   │  + Coraza WAF   │
            │  :8080    │   └────────┬────────┘
            └─────┬─────┘            │
                  │         ┌───────┴────────┬──────────────┐
                  │         │                │              │
                  │         ▼                ▼              ▼
                  │   ┌──────────┐   ┌────────────┐   ┌──────────┐
                  │   │ Auth API │   │ Stream API │   │ Frontend │
                  │   │  :1988   │   │   :1987    │   │ /admin   │
                  │   └────┬─────┘   └─────┬──────┘   └──────────┘
                  │        │               │
                  │        └───────┬───────┘
                  │                │
                  │                ▼
                  │        ┌─────────────┐
                  │        │    REDIS    │
                  │        │    :6379    │
                  │        └─────────────┘
                  │
                  ▼
            ┌───────────────┐
            │ ABR Transcoder│
            │   (FFmpeg)    │
            └───────┬───────┘
                    │
                    ▼
            ┌───────────────┐
            │   HLS Output  │
            │   /hls/*.m3u8 │
            └───────────────┘

3.2 Data Flow

Stream Publishing Flow:

1. Encoder (OBS) connects to rtmp://stream.getsetlive.com/live/stream_key
2. SRS receives RTMP connection on port 1935
3. SRS triggers HTTP callback to streaming-api: POST /api/srs/on_publish
4. streaming-api validates stream key and IP whitelist
5. If valid, SRS accepts stream; ABR transcoder starts
6. FFmpeg creates HLS segments in /opt/streaming-server/hls/stream_key/
7. Segments available via https://stream.getsetlive.com/hls/stream_key/master.m3u8

Playback Flow:

1. Player requests https://stream.getsetlive.com/hls/stream_key/master.m3u8
2. Caddy serves master playlist with available quality levels
3. Player selects quality based on bandwidth
4. Player requests .ts segments for chosen quality
5. Caddy serves segments from /opt/streaming-server/hls/

3.3 Directory Structure

/opt/streaming-server/
├── auth/                    # Authentication service
│   ├── server.js           # Main auth server
│   ├── package.json        # Dependencies
│   └── node_modules/
│
├── streaming/               # Streaming API service
│   ├── server.js           # Main API server
│   ├── package.json
│   └── node_modules/
│
├── frontend/                # Admin dashboard
│   ├── admin.html          # Main dashboard page
│   ├── admin-app.js        # Dashboard JavaScript
│   ├── admin-styles.css    # Dashboard styles
│   └── index.html          # Login page
│
├── configs/                 # Configuration files
│   ├── streaming.config.json
│   ├── security.config.json
│   ├── abr.config.json
│   ├── whitelist_ips.json
│   └── srs.conf
│
├── hls/                     # HLS output directory
│   └── {stream_key}/
│       ├── master.m3u8     # ABR master playlist
│       ├── 1080p/
│       ├── 720p/
│       └── ...
│
├── waf/                     # WAF configuration
│   ├── config/
│   │   └── coraza-main.conf
│   └── logs/
│       └── audit.log
│
└── tmp/                     # Temporary files

4.1 Service Configuration

streaming.config.json

{
  "server": {
    "port": 1987,
    "host": "127.0.0.1"
  },
  "srs": {
    "api_url": "http://127.0.0.1:1985",
    "rtmp_port": 1935,
    "http_port": 8080
  },
  "hls": {
    "output_path": "/opt/streaming-server/hls",
    "segment_duration": 2,
    "playlist_size": 10
  },
  "redis": {
    "host": "127.0.0.1",
    "port": 6379
  }
}

security.config.json

{
  "authentication": {
    "jwt_secret": "your_secure_secret_here",
    "jwt_expiry": "24h",
    "session": {
      "cookie_max_age_seconds": 86400,
      "http_only": true,
      "secure": true
    }
  },
  "rate_limiting": {
    "auth_failures": {
      "max_attempts": 5,
      "ban_duration_seconds": 900
    },
    "api_requests": {
      "window_seconds": 60,
      "max_requests": 100
    }
  },
  "admin_dashboard": {
    "api_key": "64_character_secure_key_here"
  }
}

4.2 ABR Transcoding Setup

abr.config.json

{
  "enabled": true,
  "qualities": [
    {
      "name": "1080p",
      "width": 1920,
      "height": 1080,
      "video_bitrate": "5000k",
      "audio_bitrate": "192k"
    },
    {
      "name": "720p",
      "width": 1280,
      "height": 720,
      "video_bitrate": "2800k",
      "audio_bitrate": "128k"
    },
    {
      "name": "480p",
      "width": 854,
      "height": 480,
      "video_bitrate": "1400k",
      "audio_bitrate": "128k"
    },
    {
      "name": "360p",
      "width": 640,
      "height": 360,
      "video_bitrate": "800k",
      "audio_bitrate": "96k"
    },
    {
      "name": "240p",
      "width": 426,
      "height": 240,
      "video_bitrate": "400k",
      "audio_bitrate": "64k"
    }
  ],
  "encoding": {
    "video_codec": "libx264",
    "preset": "veryfast",
    "audio_codec": "aac"
  }
}

4.3 Authentication System

The authentication system uses JWT tokens with Redis session storage.

Login Flow:

1. Client sends POST /api/auth/login with username/password
2. Auth service validates credentials against security.config.json
3. If valid:
   - Generate JWT token with 24h expiry
   - Store session in Redis: session:{username}:{timestamp}
   - Return token + dashboardConfig (including API key)
4. Client stores token in sessionStorage
5. Subsequent requests include Authorization: Bearer {token}

API Key Authentication:

Admin API endpoints use X-API-Key header:
- Key is returned in dashboardConfig after login
- Validated against security.config.json
- Used for: /api/admin/* endpoints

5.1 Authentication APIs

curl -X POST https://stream.getsetlive.com/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "username": "admin",
    "password": "your_secure_password"
  }'

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6ImFkbWluIiwicm9sZSI6ImFkbWluIiwiaWF0IjoxNzMzMzg1NjAwLCJleHAiOjE3MzM0NzIwMDB9.abc123...",
  "role": "admin",
  "username": "admin",
  "expiresIn": "24h",
  "dashboardConfig": {
    "api_key": "pTuFDIJuTZJGDDfD0qbN@UJHtLsmrzZdvWjzmjd3MYRtpM0HSbc@-jyZ8g.VsGTm",
    "refresh_intervals_ms": {
      "services": 10000,
      "streams": 5000,
      "logs": 3000
    }
  }
}

{
  "error": "Invalid credentials"
}

curl -X POST https://stream.getsetlive.com/api/auth/verify \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIs..."

{
  "valid": true,
  "username": "admin",
  "role": "admin",
  "expiresAt": "2025-12-06T00:00:00.000Z"
}

curl -X POST https://stream.getsetlive.com/api/auth/logout \
  -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIs..."

{
  "success": true,
  "message": "Logged out successfully"
}

5.2 Stream Management APIs

curl -X GET https://stream.getsetlive.com/api/admin/streams/all \
  -H "X-API-Key: pTuFDIJuTZJGDDfD0qbN@UJHtLsmrzZdvWjzmjd3MYRtpM0HSbc@-jyZ8g.VsGTm"

{
  "success": true,
  "streams": [
    {
      "id": "vid-583u578",
      "streamKey": "admin_20251205133626",
      "app": "live",
      "clientId": "7x2k9p3m",
      "status": "publishing",
      "startTime": "2025-12-05T13:36:26.000Z",
      "uptime": "2h 15m 30s",
      "video": {
        "codec": "H264",
        "width": 1920,
        "height": 1080,
        "fps": 30
      },
      "audio": {
        "codec": "AAC",
        "sampleRate": 48000,
        "channels": 2
      },
      "bitrate": {
        "video": 3500,
        "audio": 128,
        "total": 3628
      },
      "bytes": {
        "received": 2847593472,
        "sent": 1423796736
      },
      "clients": {
        "publishers": 1,
        "players": 12
      }
    }
  ],
  "totalStreams": 1,
  "totalViewers": 12
}

curl -X POST https://stream.getsetlive.com/api/admin/streams/stop \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "streamKey": "admin_20251205133626"
  }'

{
  "success": true,
  "message": "Stream stopped successfully",
  "streamKey": "admin_20251205133626"
}

curl -X POST https://stream.getsetlive.com/api/admin/streams/kick-client \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "clientId": "7x2k9p3m"
  }'

{
  "success": true,
  "message": "Client disconnected",
  "clientId": "7x2k9p3m"
}

5.3 System APIs

curl -X GET https://stream.getsetlive.com/api/admin/system/stats \
  -H "X-API-Key: YOUR_API_KEY"

{
  "success": true,
  "timestamp": "2025-12-05T15:30:00.000Z",
  "cpu": {
    "usage": 23.5,
    "cores": 8,
    "model": "Intel(R) Xeon(R) CPU E5-2680 v4"
  },
  "memory": {
    "total": 16384,
    "used": 8547,
    "free": 7837,
    "usagePercent": 52.2
  },
  "disk": {
    "total": 200,
    "used": 45,
    "free": 155,
    "usagePercent": 22.5,
    "path": "/opt/streaming-server"
  },
  "network": {
    "bytesIn": 1547893248,
    "bytesOut": 8934572134,
    "connectionsActive": 156
  },
  "uptime": {
    "system": "45 days, 12:30:15",
    "services": {
      "srs": "22h 30m",
      "streaming-api": "45 days",
      "streaming-auth": "45 days"
    }
  }
}

curl -X GET https://stream.getsetlive.com/api/admin/services/status \
  -H "X-API-Key: YOUR_API_KEY"

{
  "success": true,
  "services": [
    {
      "name": "srs",
      "displayName": "SRS Media Server",
      "status": "running",
      "pid": 1147206,
      "memory": "45.2 MB",
      "cpu": "2.3%",
      "uptime": "22h 30m 15s"
    },
    {
      "name": "streaming-api",
      "displayName": "Streaming API",
      "status": "running",
      "pid": 234567,
      "memory": "78.5 MB",
      "cpu": "1.2%",
      "uptime": "45 days"
    },
    {
      "name": "streaming-auth",
      "displayName": "SRS Auth Service",
      "status": "running",
      "pid": 234568,
      "memory": "52.3 MB",
      "cpu": "0.5%",
      "uptime": "45 days"
    },
    {
      "name": "redis-server",
      "displayName": "Redis Database",
      "status": "running",
      "pid": 1234,
      "memory": "12.4 MB",
      "cpu": "0.1%",
      "uptime": "45 days"
    },
    {
      "name": "caddy",
      "displayName": "Web Server (Caddy)",
      "status": "running",
      "pid": 5678,
      "memory": "28.7 MB",
      "cpu": "0.8%",
      "uptime": "45 days"
    }
  ]
}

curl -X POST https://stream.getsetlive.com/api/admin/services/manage \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "services": ["srs", "streaming-api"],
    "action": "restart"
  }'

{
  "success": true,
  "results": [
    {
      "service": "srs",
      "action": "restart",
      "success": true,
      "message": "Service restarted successfully"
    },
    {
      "service": "streaming-api",
      "action": "restart",
      "success": true,
      "message": "Service restarted successfully"
    }
  ]
}

5.4 Security APIs

curl -X GET https://stream.getsetlive.com/api/admin/security/overview \
  -H "X-API-Key: YOUR_API_KEY"

{
  "success": true,
  "firewallStatus": "Active - CSF Enabled",
  "wafStatus": "Active",
  "firewallBlocksToday": 47,
  "wafBlocksToday": 23,
  "rateLimitedToday": 5,
  "activeSessions": 3,
  "lastUpdated": "2025-12-05T15:30:00.000Z"
}

curl -X GET "https://stream.getsetlive.com/api/admin/waf/events?level=blocked&limit=50" \
  -H "X-API-Key: YOUR_API_KEY"

{
  "success": true,
  "events": [
    {
      "timestamp": "2025-12-05T14:23:15.000Z",
      "ip": "185.220.101.45",
      "method": "GET",
      "uri": "/.git/HEAD",
      "ruleId": "920350",
      "message": "Path Traversal Attack",
      "blocked": true,
      "severity": "critical"
    },
    {
      "timestamp": "2025-12-05T14:20:30.000Z",
      "ip": "45.95.147.236",
      "method": "POST",
      "uri": "/api/login",
      "ruleId": "942100",
      "message": "SQL Injection Attack Detected",
      "blocked": true,
      "severity": "critical"
    }
  ],
  "count": 2,
  "totalBlocked": 23
}

curl -X GET https://stream.getsetlive.com/api/admin/csf/status \
  -H "X-API-Key: YOUR_API_KEY"

{
  "success": true,
  "status": "running",
  "version": "14.20",
  "testing": false,
  "rules": {
    "allow": 45,
    "deny": 1247,
    "tempBan": 12
  },
  "recentBlocks": [
    {
      "ip": "192.168.1.100",
      "reason": "Port scan detected",
      "time": "2025-12-05T14:15:00.000Z",
      "expires": "2025-12-05T15:15:00.000Z"
    }
  ]
}

6.1 Pre-Deployment Testing

Service Health Checks:

# Test all services are running
$ sudo systemctl status srs streaming-api streaming-auth redis-server caddy

# Test Redis connectivity
$ redis-cli ping
PONG

# Test SRS API
$ curl -s http://127.0.0.1:1985/api/v1/versions | jq .
{
  "code": 0,
  "data": {
    "version": "7.0.94"
  }
}

# Test Streaming API
$ curl -s http://127.0.0.1:1987/api/health | jq .
{
  "status": "ok",
  "timestamp": "2025-12-05T15:00:00.000Z"
}

# Test Auth Service
$ curl -s http://127.0.0.1:1988/api/health | jq .
{
  "status": "ok",
  "redis": "connected"
}

RTMP Stream Test:

# Publish test stream with FFmpeg
$ ffmpeg -re -f lavfi -i testsrc=size=1280x720:rate=30 \
  -f lavfi -i sine=frequency=1000 \
  -c:v libx264 -preset ultrafast -b:v 2000k \
  -c:a aac -b:a 128k \
  -f flv rtmp://localhost/live/test_stream

# Verify stream is active
$ curl -s http://127.0.0.1:1985/api/v1/streams/ | jq '.streams | length'
1

# Test HLS playback
$ curl -I https://stream.getsetlive.com/hls/test_stream/master.m3u8
HTTP/2 200
content-type: application/vnd.apple.mpegurl

6.2 Load Testing

# Simulate multiple viewers with Apache Bench
$ ab -n 1000 -c 100 https://stream.getsetlive.com/hls/test_stream/master.m3u8

# Monitor during test
$ htop  # CPU/Memory
$ ss -s # Connection stats
$ sudo journalctl -u srs -f  # SRS logs

6.3 Security Testing

# Test WAF is blocking malicious requests
$ curl -X GET "https://stream.getsetlive.com/api/test?id=1' OR '1'='1"
# Should return 403 Forbidden

# Test rate limiting
$ for i in {1..10}; do curl -X POST https://stream.getsetlive.com/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"test","password":"wrong"}'; done
# Should get rate limited after 5 attempts

# Verify JWT expiry
$ curl -X POST https://stream.getsetlive.com/api/auth/verify \
  -H "Authorization: Bearer expired_token_here"
# Should return 401 Unauthorized

7.1 System Requirements

7.2 Installation Steps

# 1. Install dependencies
sudo apt update && sudo apt upgrade -y
sudo apt install -y nodejs npm redis-server ffmpeg

# 2. Install Caddy with WAF module
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/caddy-stable-archive-keyring.gpg] https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main" | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy

# 3. Create directory structure
sudo mkdir -p /opt/streaming-server/{auth,streaming,frontend,configs,hls,waf,tmp}
sudo chown -R $USER:$USER /opt/streaming-server

# 4. Deploy application files
# (Copy auth, streaming, frontend directories)

# 5. Install Node.js dependencies
cd /opt/streaming-server/auth && npm install
cd /opt/streaming-server/streaming && npm install

# 6. Configure systemd services
sudo cp /opt/streaming-server/configs/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable srs streaming-api streaming-auth

# 7. Start services
sudo systemctl start redis-server
sudo systemctl start srs streaming-api streaming-auth
sudo systemctl start caddy

7.3 Backup Procedures

# Create backup
$ sudo /usr/local/bin/streaming-backup "Pre-deployment backup"
Creating backup: streaming-server_Pre-deployment_backup_20251205_153000.tar.gz
Backup completed successfully

# List backups
$ ls -la /opt/backups/streaming-server_*.tar.gz
-rw-r--r-- 1 root root 45M Dec 5 15:30 streaming-server_Pre-deployment_backup_20251205_153000.tar.gz

# Restore from backup
$ cd /opt
$ sudo tar -xzf /opt/backups/streaming-server_BACKUP_NAME.tar.gz
$ sudo systemctl restart srs streaming-api streaming-auth

8.1 Daily Operations

# Morning health check
$ sudo systemctl status srs streaming-api streaming-auth redis-server caddy | grep Active

# Check disk space
$ df -h /opt/streaming-server
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       200G   45G  155G  23% /

# Check active streams
$ curl -s http://127.0.0.1:1985/api/v1/streams/ | jq '.streams | length'

# View recent errors
$ sudo journalctl -u streaming-api -p err --since "1 hour ago"

8.2 Maintenance Tasks

# Weekly: Clean old HLS segments
$ find /opt/streaming-server/hls -type f -mtime +7 -delete

# Weekly: Restart SRS (memory leak mitigation)
$ sudo systemctl restart srs

# Monthly: Update SSL certificates (auto with Caddy)
$ sudo systemctl reload caddy

# Monthly: Review WAF logs
$ sudo tail -100 /opt/streaming-server/waf/logs/audit.log

8.3 Monitoring Commands

# Real-time log monitoring
$ sudo journalctl -u srs -u streaming-api -u streaming-auth -f

# Network connections
$ ss -tlnp | grep -E '(1935|1985|1987|1988|443)'

# Process resource usage
$ ps aux --sort=-%mem | head -10

# SRS statistics
$ curl -s http://127.0.0.1:1985/api/v1/summaries | jq .

9.1 Stream Issues

# Diagnosis
$ sudo systemctl status srs
$ sudo netstat -tlnp | grep 1935
$ sudo csf -l | grep 1935

# Solution
$ sudo systemctl restart srs
$ sudo csf -a YOUR_IP  # Whitelist if needed

# Diagnosis
$ ls -la /opt/streaming-server/hls/stream_key/
$ curl -I https://stream.getsetlive.com/hls/stream_key/master.m3u8

# Check SRS logs
$ sudo journalctl -u srs -n 50 | grep -i error

# Solution: Verify HLS output path permissions
$ sudo chown -R www-data:www-data /opt/streaming-server/hls

9.2 Authentication Issues

# Check auth service
$ sudo systemctl status streaming-auth

# Check Redis
$ redis-cli ping

# Check credentials in config
$ cat /opt/streaming-server/configs/security.config.json | jq '.credentials'

# View auth logs
$ sudo journalctl -u streaming-auth -n 50

# Check rate limit status
$ redis-cli keys "ratelimit:*"

# Clear rate limit for IP
$ redis-cli del "ratelimit:YOUR_IP:auth"

# Check config
$ cat /opt/streaming-server/configs/security.config.json | jq '.rate_limiting'

9.3 Performance Issues

# Check FFmpeg processes
$ ps aux | grep ffmpeg

# Reduce quality levels in config
$ nano /opt/streaming-server/configs/abr.config.json
# Remove 1080p or 720p quality

# Use faster preset
# Change "preset": "veryfast" to "ultrafast"

# Check memory usage
$ ps aux --sort=-%mem | grep srs

# Known issue: HTTP connection leak in SRS 7.0.94
# Solution: Weekly restart via cron
$ sudo crontab -e
# Add: 0 4 * * 0 systemctl restart srs

9.4 Quick Reference Commands

# Restart all services
$ sudo systemctl restart srs streaming-api streaming-auth

# Clear HLS cache
$ rm -rf /opt/streaming-server/hls/*

# View all logs combined
$ sudo journalctl -u streaming-api -u streaming-auth -u srs -f

# Check all service status
$ for svc in srs streaming-api streaming-auth redis-server caddy; do
    echo "=== $svc ===" && sudo systemctl status $svc | head -3
  done

# Emergency: Stop all streams
$ curl -X POST http://127.0.0.1:1985/api/v1/kick_all

10.1 Bottlenecks Encountered

# Evidence from logs
RTC: before dispose resource(HttpConn)(0x50d00004d410), conns=11, zombies=3
RTC: before dispose resource(HttpConn)(0x50d000069be0), conns=11, zombies=4
RTC: before dispose resource(HttpConn)(0x50d00006b030), conns=11, zombies=5

# Weekly SRS restart via cron (Sunday 4 AM)
$ sudo crontab -e
0 4 * * 0 /usr/bin/systemctl restart srs

# Reduce API polling frequency from 5s to 15s
# In streaming.config.json:
"srs_polling_interval_ms": 15000

# Changed FFmpeg preset in abr.config.json
# Before: "preset": "medium"
# After:  "preset": "veryfast"

# Result: 60% CPU reduction with acceptable quality trade-off
# Quality comparison (VMAF scores):
#   medium preset:   94.5 VMAF
#   veryfast preset: 91.2 VMAF (3.5% reduction, acceptable)

# 1. Reduced HLS window in srs.conf
hls_window 60;  # Reduced from 300 to 60 seconds
hls_fragment 2;

# 2. Added cleanup cron job
0 */6 * * * find /opt/streaming-server/hls -type f -mmin +360 -delete

# 3. Stream-end cleanup in server.js on_unpublish callback

# 1. Implemented Redis connection pooling in server.js
const redis = require('redis');
const client = redis.createClient({
    socket: {
        reconnectStrategy: (retries) => Math.min(retries * 100, 3000)
    },
    maxRetriesPerRequest: 3
});

# 2. Increased Redis maxclients in redis.conf
maxclients 50000

# 3. Added connection health check
setInterval(() => client.ping(), 30000);

10.2 Performance Optimizations Applied

10.2.1 HLS Configuration Tuning

# Current srs.conf HLS settings
vhost __defaultVhost__ {
    hls {
        enabled on;
        hls_fragment 2;
        hls_window 60;
        hls_dispose 60;
        hls_path /opt/streaming-server/hls;
        hls_m3u8_file [app]/[stream]/playlist.m3u8;
        hls_ts_file [app]/[stream]/[seq].ts;
    }
}

10.2.2 FFmpeg Transcoding Optimization

# Optimized FFmpeg command flags
-preset veryfast
-tune zerolatency
-g 48 -keyint_min 48
-sc_threshold 0
-b_strategy 0
-bf 0

10.2.3 HLS.js Player Optimization

// Disabled low latency mode for stability
const hls = new Hls({
    lowLatencyMode: false,    // Changed from true
    backBufferLength: 30,     // Increased from 10
    maxBufferLength: 60,      // Increased from 30
    maxMaxBufferLength: 120,
    liveSyncDurationCount: 3,
    liveMaxLatencyDurationCount: 10
});

10.2.4 API Polling Optimization

// streaming.config.json polling intervals
{
    "refresh_intervals_ms": {
        "services": 10000,
        "streams": 5000,
        "logs": 3000,
        "stats": 10000
    },
    "srs_polling_interval_ms": 15000
}

10.2.5 Caddy/WAF Performance

# WAF bypass for admin API (no WAF overhead for authenticated requests)
SecRule REQUEST_URI "^/api/admin/" "id:1006,phase:1,pass,nolog,ctl:ruleEngine=Off"

# Static file caching headers
handle /hls/* {
    header Cache-Control "public, max-age=2"
    header Access-Control-Allow-Origin "*"
}

10.3 Performance Monitoring

Key Metrics to Watch:

# SRS Memory (should stay under 500MB)
$ ps aux | grep srs | awk '{print $6/1024 " MB"}'

# FFmpeg CPU per stream
$ ps aux | grep ffmpeg | awk '{sum+=$3} END {print sum "%"}'

# Redis memory
$ redis-cli info memory | grep used_memory_human

# Active network connections
$ ss -s | grep estab

# HLS disk usage
$ du -sh /opt/streaming-server/hls/

Alerting Thresholds:

10.4 Lessons Learned

• Proactive Restarts: Scheduled restarts prevent memory-related outages
• Polling Trade-offs: Less frequent polling reduces load but delays status updates
• Preset Selection: Video quality vs CPU usage is a critical balance
• Connection Pooling: Essential for high-throughput API services
• Log Analysis: "zombie" count in SRS logs is an early warning indicator
• Buffer Tuning: Player buffering affects both latency and stability

Future Improvements Planned: