Project-Agent-trust-merge product-docs docs/testing-guide.md

Complete guide for testing the ARXsec.io multi-platform integration across Claude Code, ChatGPT, and custom SDKs.

Overview

This document covers testing for:

OpenAPI Foundation - Core specification and discovery endpoints
Claude Code MCP - Model Context Protocol integration
ChatGPT Integration - Custom GPT setup
Python SDK - arxsec package (PyPI)
JavaScript SDK - @arxsec/sdk (npm)
Example Applications - Security dashboard and API

Prerequisites

ARXsec.io account with API key
Agent ID (UUID) configured
Python 3.11+
Node.js 18+
Claude Code installed (for MCP testing)
ChatGPT Plus account (for Custom GPT testing)

Phase 1: OpenAPI Foundation Testing

1.1 Validate OpenAPI Specification

```bash

Get the OpenAPI spec

curl -H "Authorization: Bearer YOUR_API_KEY" \ https://api.arxsec.io/v1/tools/openapi > arxsec-openapi.yaml

Validate against OpenAPI 3.1 standard

pip install openapi-spec-validator python -m openapi_spec_validator arxsec-openapi.yaml ```

Expected Result: ✅ Spec is valid and follows OpenAPI 3.1 standards

1.2 Test Tool Discovery Endpoints

```bash

Test /v1/tools/schema

curl -H "Authorization: Bearer YOUR_API_KEY" \ https://api.arxsec.io/v1/tools/schema | jq '.total_count'

Should return a number > 100

Example output: 487

Test /v1/tools/categories

curl -H "Authorization: Bearer YOUR_API_KEY" \ https://api.arxsec.io/v1/tools/categories | jq 'keys'

Should return categories like: ["endpoint", "siem", "cloud", ...]

Test specific connector tools

curl -H "Authorization: Bearer YOUR_API_KEY" \ https://api.arxsec.io/v1/tools/connector/crowdstrike | jq '.operation_count'

Should return operation count for CrowdStrike (e.g., 8)

```

Verification Checklist:

[ ] /v1/tools/schema returns tools with total_count
[ ] /v1/tools/categories returns organized categories
[ ] /v1/tools/connector/{name} returns operations for specific connectors
[ ] All responses include proper authentication headers
[ ] Error responses (401, 403) handled correctly

Phase 2: Claude Code MCP Testing

2.1 MCP Server Setup

```bash

Install MCP server dependencies

cd arxsec-api/mcp-server pip install -r requirements.txt

Test server startup

export ARX_API_KEY=your-key export ARX_AGENT_ID=your-agent-uuid python server.py ```

Expected Output: `` {"event": "mcp.server.starting", "base_url": "https://api.arxsec.io"} ``

2.2 Add to Claude Code

Open Claude Code Settings
Go to "MCP Servers"
Add new server:

Name: arxsec-mcp-server
Path: /path/to/arxsec-api/mcp-server/server.py
Environment:

`` ARX_API_KEY=your-key ARX_AGENT_ID=your-agent-uuid ARX_API_BASE_URL=https://api.arxsec.io ``

Restart Claude Code

2.3 Test MCP Tools in Claude Code

Test 1: List Tools ``` User: "@arxsec show available tools"

Expected: Claude lists available connectors ```

Test 2: Query CrowdStrike ``` User: "@arxsec get last 5 CrowdStrike detections"

Expected: Claude calls crowdstrike_detections_read, returns results ```

Test 3: Send Slack Message ``` User: "@arxsec send message to #security-alerts: Test message from arx"

Expected: Claude calls slack_messages_write, confirms delivery ```

Test 4: Create Jira Ticket ``` User: "@arxsec create Jira ticket with title 'Test Incident' in project SEC"

Expected: Claude calls jira_issues_write, returns ticket key ```

Test 5: Generic Tool ``` User: "@arxsec execute pagerduty incidents:read with limit 10"

Expected: Claude uses generic execute_connector tool ```

Verification Checklist:

[ ] MCP server connects without errors
[ ] Claude can list tools
[ ] Specific tools execute correctly
[ ] Generic tool works as fallback
[ ] Error handling for denied operations
[ ] Approval escalation works (if policy requires)

Phase 3: ChatGPT Custom GPT Testing

3.1 Prepare OpenAPI Spec

```bash

Download or generate OpenAPI spec

curl -H "Authorization: Bearer YOUR_API_KEY" \ https://api.arxsec.io/v1/tools/openapi > arxsec-openapi.yaml ```

3.2 Create Custom GPT

Go to https://chatgpt.com/gpts/editor
Click "Configure"
Name: "ARXsec Security Agent"
Description: "Execute security operations across 100+ tools"
Instructions: Copy from docs/integration-guides/chatgpt.md

3.3 Add OpenAPI Action

Scroll to "Actions"
Click "Create new action"
Paste OpenAPI spec in schema field
Authentication: Bearer token
Token: Your ARX_API_KEY

3.4 Test ChatGPT Integration

Test 1: List Detections ``` User: "Get the latest CrowdStrike detections"

Expected: ChatGPT calls CrowdStrike API through arx Result: Returns detection list ```

Test 2: Create Incident ``` User: "Create a Jira ticket for a security incident"

Expected: ChatGPT creates ticket through arx Result: Returns ticket key ```

Test 3: Multiple Operations ``` User: "Get CrowdStrike detections AND send Slack alert"

Expected: ChatGPT chains operations Result: Both execute successfully ```

Test 4: Error Handling ``` User: "Disable a user in Okta"

Expected: If policy denies, ChatGPT receives error Result: Displays error or escalation URL ```

Verification Checklist:

[ ] Custom GPT loads successfully
[ ] OpenAPI spec validated
[ ] Authentication configured correctly
[ ] Tool execution succeeds
[ ] Error responses handled
[ ] Approval URLs provided when escalated

Phase 4: Python SDK Testing

4.1 Install SDK

```bash

From PyPI (when published)

pip install arxsec

From local source (for development)

cd arxsec-api/sdk/agentvault pip install -e . ```

4.2 Basic Functionality Test

```python

test_sdk_basic.py

import asyncio import os from arxsec import ARXClient

async def test_basic(): arx = ARXClient(api_key=os.environ["ARX_API_KEY"])

Test 1: CrowdStrike

print("Test 1: CrowdStrike detections...") cs = arx.crowdstrike() detections = await cs.get_detections(limit=5) assert detections.get("status") in ["success", "escalated", "denied"] print("✓ CrowdStrike test passed")

Test 2: Slack

print("Test 2: Slack messaging...") slack = arx.slack() result = await slack.send_message("#testing", "Test message") assert result.get("status") in ["success", "escalated", "denied"] print("✓ Slack test passed")

Test 3: Jira

print("Test 3: Jira issue creation...") jira = arx.jira() result = await jira.create_issue( project="TEST", summary="SDK Test Issue", description="Testing arx SDK" ) assert result.get("status") in ["success", "escalated", "denied"] print("✓ Jira test passed")

Test 4: Tool Discovery

print("Test 4: Tool discovery...") tools = await arx.get_tools() assert tools.get("total_count", 0) > 100 print(f"✓ Tool discovery passed ({tools['total_count']} tools)")

print("\n✅ All Python SDK tests passed!")

asyncio.run(test_basic()) ```

Run Tests: ``bash export ARX_API_KEY=your-key python test_sdk_basic.py ``

Expected Output: ``` Test 1: CrowdStrike detections... ✓ CrowdStrike test passed Test 2: Slack messaging... ✓ Slack test passed Test 3: Jira issue creation... ✓ Jira test passed Test 4: Tool discovery... ✓ Tool discovery passed (487 tools)

✅ All Python SDK tests passed! ```

4.3 Advanced Python SDK Tests

```python

test_sdk_advanced.py

import asyncio from arxsec import ARXClient

async def test_advanced(): async with ARXClient() as arx:

Test context manager

print("✓ Context manager works")

Test session context tracking

cs = arx.crowdstrike() result1 = await cs.get_detections(limit=1) result2 = await cs.get_hosts(limit=1) print("✓ Session context tracked")

Test error handling

try: await cs.contain_host(["invalid_id"]) print("✓ Error handling works") except Exception as e: print(f"✓ Exception raised: {type(e).__name__}")

asyncio.run(test_advanced()) ```

Verification Checklist:

[ ] SDK installs without errors
[ ] Client initializes correctly
[ ] All connector methods work
[ ] Async/await patterns function
[ ] Context manager works
[ ] Tool discovery returns data
[ ] Error handling is proper
[ ] Session context is tracked

Phase 5: JavaScript SDK Testing

5.1 Install SDK

```bash

From npm (when published)

npm install @arxsec/sdk

From local source (for development)

cd arxsec-api/sdk/javascript npm install -e . ```

5.2 Basic Functionality Test

```typescript // test-sdk-basic.ts import { ARXClient, ARXError } from "@arxsec/sdk";

async function testBasic() { const arx = new ARXClient({ apiKey: process.env.ARX_API_KEY, });

// Test 1: CrowdStrike console.log("Test 1: CrowdStrike detections..."); const cs = arx.crowdstrike(); const detections = await cs.getDetections({ limit: 5 }); console.assert(["success", "escalated", "denied"].includes(detections.status)); console.log("✓ CrowdStrike test passed");

// Test 2: Slack console.log("Test 2: Slack messaging..."); const slack = arx.slack(); const msg = await slack.sendMessage("#testing", "Test from SDK"); console.assert(["success", "escalated", "denied"].includes(msg.status)); console.log("✓ Slack test passed");

// Test 3: Tool Discovery console.log("Test 3: Tool discovery..."); const tools = await arx.getTools(); console.assert(tools.total_count > 100); console.log(✓ Tool discovery passed (${tools.total_count} tools));

console.log("\n✅ All JavaScript SDK tests passed!"); }

testBasic().catch((error) => { if (error instanceof ARXError) { console.error(ARX Error: ${error.message}); } else { console.error("Unexpected error:", error); } process.exit(1); }); ```

Run Tests: ``bash export ARX_API_KEY=your-key npx ts-node test-sdk-basic.ts ``

Verification Checklist:

[ ] SDK installs without errors
[ ] TypeScript types are available
[ ] Client initializes correctly
[ ] All connector methods work
[ ] Async/await patterns function
[ ] Error handling is proper (ARXError)
[ ] Tool discovery works

Phase 6: Example Applications Testing

6.1 Test Python Security Dashboard

```bash

Setup

cd examples/python-security-dashboard pip install -r requirements.txt export ARX_API_KEY=your-key

Start server

python app.py

In another terminal, test endpoints

curl http://localhost:8001/health curl http://localhost:8001/api/dashboard | jq '.' curl http://localhost:8001/api/threats/crowdstrike | jq '.count' curl http://localhost:8001/api/incidents | jq '.total' ```

Expected Endpoints:

✓ GET /health - Returns healthy status
✓ GET /api/dashboard - Returns aggregated security data
✓ GET /api/threats/crowdstrike - Returns CrowdStrike detections
✓ GET /api/threats/splunk - Returns Splunk alerts
✓ GET /api/incidents - Returns PagerDuty incidents
✓ POST /api/actions/contain-host - Requests host containment
✓ POST /api/actions/create-incident - Creates Jira incident

6.2 Test JavaScript Security API

```bash

Setup

cd examples/javascript-security-api npm install cp .env.example .env

Edit .env with your API key

Start server

npm run dev

In another terminal, test endpoints

curl http://localhost:3000/health curl http://localhost:3000/api/dashboard | jq '.' curl http://localhost:3000/api/threats/crowdstrike | jq '.total' curl http://localhost:3000/api/incidents | jq '.total' ```

Expected Endpoints:

✓ GET /health - Returns healthy status
✓ GET /api/dashboard - Returns aggregated security data
✓ GET /api/threats/crowdstrike - Returns CrowdStrike detections
✓ GET /api/threats/splunk - Returns Splunk alerts
✓ GET /api/incidents - Returns PagerDuty incidents
✓ GET /api/tools - Returns available tools

Phase 7: Integration Testing

7.1 End-to-End Workflow

```

Claude Code MCP:

List tools
Query CrowdStrike
Create Jira ticket
Send Slack alert

ChatGPT Custom GPT:

Same workflow as above

Python SDK:

Build security dashboard
Aggregate threats
Create incidents

JavaScript SDK:

Build REST API
Query multiple sources
Return aggregated data

```

7.2 Policy & Approvals Testing

Test Escalation: ```python

Send high-risk operation (should escalate)

cs = arx.crowdstrike() result = await cs.contain_host(["device_id"])

Expect: {"status": "escalated", "approval_url": "..."}

Check approval at: https://app.arxsec.io/approvals

```

7.3 Audit Trail Verification

All operations should appear in audit log: `` https://app.arxsec.io/audit ``

Verify:

[ ] Operations logged correctly
[ ] Timestamp is accurate
[ ] Agent and connector recorded
[ ] Risk score calculated
[ ] Approval status tracked

Testing Checklist

Phase 1: OpenAPI Foundation

[ ] OpenAPI spec is valid (3.1 standard)
[ ] /v1/tools/schema returns tools
[ ] /v1/tools/categories returns categories
[ ] /v1/tools/connector/{name} works for all connectors

Phase 2: Claude Code MCP

[ ] MCP server starts without errors
[ ] Integration added to Claude Code
[ ] Tool listing works
[ ] Specific tools execute
[ ] Generic tool works
[ ] Error handling is correct

Phase 3: ChatGPT Custom GPT

[ ] Custom GPT created successfully
[ ] OpenAPI spec uploaded
[ ] Authentication configured
[ ] Tools execute in ChatGPT
[ ] Errors are handled

Phase 4: Python SDK

[ ] Package installs from PyPI
[ ] All connector methods work
[ ] Tool discovery works
[ ] Context manager functions
[ ] Error handling works
[ ] Async/await patterns work

Phase 5: JavaScript SDK

[ ] Package installs from npm
[ ] TypeScript types available
[ ] All connector methods work
[ ] Tool discovery works
[ ] Error handling works
[ ] Async/await patterns work

Phase 6: Example Applications

[ ] Python dashboard starts
[ ] All Python dashboard endpoints work
[ ] JavaScript API starts
[ ] All JavaScript API endpoints work
[ ] Error handling in examples works

Phase 7: Integration

[ ] Full workflow works across all platforms
[ ] Approvals escalate correctly
[ ] Audit logging captures all operations
[ ] Drift detection alerts on unexpected behavior

Troubleshooting

OpenAPI Validation Fails

``bash pip install --upgrade openapi-spec-validator python -m openapi_spec_validator --version ``

MCP Server Connection Error

```bash

Check Python version

python --version # Should be 3.11+

Verify dependencies

pip list | grep mcp

Check environment variables

echo $ARX_API_KEY echo $ARX_AGENT_ID ```

SDK Import Errors

Python: ``bash pip install --upgrade arxsec python -c "from arxsec import ARXClient; print('OK')" ``

JavaScript: ``bash npm list @arxsec/sdk npm install --save-exact @arxsec/sdk ``

Authentication Failures

Verify API key: https://app.arxsec.io/api-keys
Check agent permissions: https://app.arxsec.io/agents
Regenerate key if expired

Operation Denied by Policy

Check agent configuration: https://app.arxsec.io/agents
Request permission for needed operations
Enable approval routing for safe escalation

Performance Testing

```bash

Test throughput (1000 operations)

for i in {1..1000}; do curl -s -H "Authorization: Bearer YOUR_API_KEY" \ https://api.arxsec.io/v1/tools/schema > /dev/null done

Expected: All requests succeed in < 5 minutes

Typical: ~50-100 requests/second

```

Security Testing

[ ] API key not exposed in logs
[ ] Credentials encrypted at rest
[ ] HTTPS enforced for all connections
[ ] CORS properly configured
[ ] Rate limiting functional
[ ] Audit trail immutable

Success Criteria

✅ All tests pass when:

OpenAPI spec is valid and discoverable
Claude Code MCP executes tools correctly
ChatGPT Custom GPT executes tools correctly
Python SDK installs and functions properly
JavaScript SDK installs and functions properly
Example applications work end-to-end
All operations are audited
Approvals escalate correctly
Error handling is robust
Performance is acceptable (< 2s per operation)

Support

For testing issues, contact:

Email: support@arxsec.io
Docs: https://arxsec.io/docs
GitHub: https://github.com/arxsec/arxsec-platform/issues

ARXsec.io Platform Testing & Verification Guide

Overview

Prerequisites

Phase 1: OpenAPI Foundation Testing

1.1 Validate OpenAPI Specification

Get the OpenAPI spec

Validate against OpenAPI 3.1 standard

1.2 Test Tool Discovery Endpoints

Test /v1/tools/schema

Should return a number > 100

Example output: 487

Test /v1/tools/categories

Should return categories like: ["endpoint", "siem", "cloud", ...]

Test specific connector tools

Should return operation count for CrowdStrike (e.g., 8)

Phase 2: Claude Code MCP Testing

2.1 MCP Server Setup

Install MCP server dependencies

Test server startup

2.2 Add to Claude Code

2.3 Test MCP Tools in Claude Code

Phase 3: ChatGPT Custom GPT Testing

3.1 Prepare OpenAPI Spec

Download or generate OpenAPI spec

3.2 Create Custom GPT

3.3 Add OpenAPI Action

3.4 Test ChatGPT Integration

Phase 4: Python SDK Testing

4.1 Install SDK

From PyPI (when published)

From local source (for development)

4.2 Basic Functionality Test

test_sdk_basic.py

Test 1: CrowdStrike

Test 2: Slack

Test 3: Jira

Test 4: Tool Discovery

4.3 Advanced Python SDK Tests

test_sdk_advanced.py

Test context manager

Test session context tracking

Test error handling

Phase 5: JavaScript SDK Testing

5.1 Install SDK

From npm (when published)

From local source (for development)

5.2 Basic Functionality Test

Phase 6: Example Applications Testing

6.1 Test Python Security Dashboard

Setup

Start server

In another terminal, test endpoints

6.2 Test JavaScript Security API

Setup

Edit .env with your API key

Start server

In another terminal, test endpoints

Phase 7: Integration Testing

7.1 End-to-End Workflow

7.2 Policy & Approvals Testing

Send high-risk operation (should escalate)

Expect: {"status": "escalated", "approval_url": "..."}

Check approval at: https://app.arxsec.io/approvals

7.3 Audit Trail Verification

Testing Checklist

Phase 1: OpenAPI Foundation

Phase 2: Claude Code MCP

Phase 3: ChatGPT Custom GPT

Phase 4: Python SDK

Phase 5: JavaScript SDK

Phase 6: Example Applications

Phase 7: Integration

Troubleshooting

OpenAPI Validation Fails

MCP Server Connection Error

Check Python version

Verify dependencies

Check environment variables

SDK Import Errors

Authentication Failures