open-appsec
WebsiteManagement PortalPlaygroundGitHub
  • open-appsec Documentation
  • What is open-appsec?
  • open-appsec Video Tutorials
  • Release Notes
  • Getting started
    • Getting Started
    • Start With Kubernetes
      • Install Using Interactive CLI Tool (Ingress NGINX)
      • Configuration Using Interactive CLI Tool
      • Install Using Helm
      • Install Using Helm - new flow (beta)
      • Configuration Using CRDs
      • Configuration Using CRDs - v1beta2
      • Configuration using CRDs - special options for Large Scale Deployments
        • Using appsec class for assigning separate custom resources to specific deployments
        • Using namespace-scoped custom resources
      • Monitor Events
    • Start With Linux
      • Install open-appsec for Linux
      • Using the open-appsec-ctl Tool
      • Configuration Using Local Policy File (Linux)
      • Local Policy File (Advanced)
      • Local Policy File v1beta2 (beta)
      • Monitor Events
    • Start with Docker
      • Install With Docker (Centrally Managed)
      • Install With Docker (Locally Managed)
      • Deploy With Docker-Compose (Beta)
      • Configuration Using Local Policy File (Docker)
      • Local Policy File (Advanced)
    • Using the Web UI (SaaS)
      • Sign-Up and Login to Portal
      • Agents Deployment
      • Connect Deployed Agents to SaaS Management Using Tool (K8s & Linux)
      • Connect Deployed Agents to SaaS Management Using Helm (K8s)
      • Connect Deployed Agents to SaaS Management (Docker)
      • Create a Profile
      • Protect Additional Assets
      • Monitor Events
    • Using the Advanced Machine Learning Model
  • Concepts
    • Agents
    • Management & Automation
    • Security Practices
    • Contextual Machine Learning
  • SETUP INSTRUCTIONS
    • Setup Web Application Settings
    • Setup Custom Rules and Exceptions
    • Setup Web User Response Pages
    • Setup Log Triggers
    • Setup Behavior Upon Failure
    • Setup Agent Upgrade Schedule
  • Additional Security Engines
    • Anti-Bot
    • API Schema Enforcement
    • Data Loss Prevention (DLP) Rules
    • File Security
    • Intrusion Prevention System (IPS)
    • Rate Limit
  • Snort Rules
    • Import Snort Rules
    • Write Snort Signatures
  • HOW TO
    • Configuration and Learning
      • Track Learning and Move From Learn/Detect to Prevent
      • Configure Contextual Machine Learning for Best Accuracy
      • Track Learning and Local Tuning in Standalone Deployments
      • Move From Detect to Prevent in K8s With Many Ingress Rules
  • Deployment and Upgrade
    • Load the Attachment in Proxy Configuration
    • Upgrade Your Reverse Proxy/API Gateway When an Agent is Installed
    • Integration in GitOps CD (K8s)
    • Build open-appsec Based on Source Code
  • Management Web UI
    • Track Agent Status
    • Delete or Reset Management Tenant (SaaS)
    • Disconnect an open-appsec agent from Central Management
  • Integrations
    • About Integrations With 3rd Party Solutions
    • CrowdSec
      • CrowdSec Bouncer Support
      • CrowdSec Intelligence Sharing Using open-appsec Parser/Scenario
    • NGINX Proxy Manager
      • Install NGINX Proxy Manager with open-appsec managed from NPM WebUI
      • Install NGINX Proxy Manager with open-appsec managed from central WebUI (SaaS)
      • Frequently Asked Questions
      • How to Migrate from an Existing NGINX Proxy Manager Deployment and Keep Configuration
    • NPMplus
    • Docker SWAG
      • Install Docker SWAG with open-appsec (locally managed)
      • How to connect locally managed Docker SWAG with open-appsec to WebUI
      • Install Docker SWAG with open-appsec (centrally managed)
      • Deploy Docker SWAG with docker-compose (beta)
      • Frequently Asked Questions
  • Prometheus
  • Troubleshooting
    • Troubleshooting
    • Troubleshooting Guides
      • Configuration contains ingress/asset with URL which already has asset attached to it in your tenant
      • HTTP Request to Port 80 Not Returning as Expected
      • Agent Fails to Recognize HTTP Transactions with NGINX
      • Agent Not Recognizing Initial HTTP Requests
      • Handling Large Requests (413 Responses)
      • open-appsec on Docker HTTP Transaction Handler Is Set To Ready
      • Traffic Recognition Issue on Single-Core Machine/Connection Timed Out
      • Installing open-appsec on CentOS 7
      • SELinux: checking status and disabling
      • Deploy open-appsec directly on the web server hosting the application to protect
      • object is locked or remote, and therefore cannot be modified
      • Failed to Register to Fog
  • references
    • Agent CLI
    • Event Query Language
    • Events/Logs Schema
    • WAF Comparison Project
Powered by GitBook
On this page
  • Phase 1 – Payload Decoding
  • Phase 2 – Attack Indicators
  • Phase 3 – Contextual Evaluation Engine
  • Additional Information

Was this helpful?

  1. Concepts

Contextual Machine Learning

PreviousSecurity PracticesNextSetup Web Application Settings

Last updated 3 months ago

Was this helpful?

open-appsec uses a Patented Contextual Machine Learning Engine that utilizes a three-phase approach for detecting and preventing web application and API attacks. In this section you will understand how these three phases deliver accurate results with a very low amount of false positives and how they protect the environment against known and unknown zero-day attacks with real-time protection.

Phase 1 – Payload Decoding

Effective machine learning requires a deep understanding of the underlying application protocols which is continuously evolving. The engine analyzes all fields of the HTTP request including the URLs, HTTP headers, which are critical in this case, JSON/XML extraction and payload normalization such as base64 and other decoding's. A set of parsers covering common protocols feeds the relevant data into phase 2.

For example, in the case of Log4Shell attacks, some exploit attempts were using base64 and escaping encoding so it was possible to pass a space character for applying parameters.

Phase 2 – Attack Indicators

Following parsing and normalization, the network payload input is fed into a high-performance engine which is looking for attack indicators. An attack indicator is a pattern of exploiting vulnerabilities from various families. We derive these attack patterns based on on-going off-line supervised learning of huge number of payloads that are each assigned a score according to the likelihood of being benign or malicious. This score represents the confidence level that this pattern is part of an attack. Since combinations of these patterns can provide a better indication for an attack a score is also calculated for the combination of patterns.

For example, in the case of Log4Shell and Spring4Shell attacks, open-appsec used several indicators from Command Injection / Remote Code Execution / Probing families that signaled payloads to be malicious in a very high score which was enough on its own, but to ensure accuracy and avoidance of false positives, the engine always moves to the third and last phase.

Phase 3 – Contextual Evaluation Engine

This contextual engine is using machine learning techniques to make a final determination whether the payload is malicious, in the context of a specific customer/environment, user, URL and field that in a weighted function sums up to a confidence score. If the score is larger than the threshold the request is dropped.

These are the factors that are considered by the engine:

Reputation factor

In each request, the request originator is assigned a score. The score represents the originator’s reputation based on previous requests. This score is normalized and used to increase or decrease the confidence score.

Application awareness

Often modern applications allow users to modify web pages, upload scripts, use elaborate query search syntax, etc. These provide a better user experience but without application awareness, these are detected as malicious attacks. We use ML to analyze and baseline the underlying application’s behavior.

Learn user input format

The system can identify special user input types that are known to cause false detection and apply ML to modify our detection process and allow legitimate behavior without compromising attack detection.

False detection factor

If there is an inconsistency in detection a factor is applied to the confidence score based on the reputation factor per detection location.

Supervised learning module

Optional module that shows administrators payload and ask them to classify them thus accelerating the learning process.

Additional Information

For further information on open-appsec's machine learning see also:

Configure Contextual Machine Learning for Best Accuracy
Track Learning and Move From Learn/Detect to Prevent