Entity query

Three-layer storage structure

USearch uses a tiered storage structure to ensure data is logically isolated and queries are efficient:

1. Workspace layer: The top-level isolation unit. Different workspaces are completely isolated from each other.

2. Domain layer: A domain-level classification used for grouping business logic.

3. EntityType layer: The specific entity type that contains the actual entity data.

Workspace: my-observability
├── Domain: apm
│   ├── EntityType: apm.service
│   ├── EntityType: apm.host  
│   └── EntityType: apm.instance
├── Domain: k8s
│   ├── EntityType: k8s.pod
│   ├── EntityType: k8s.node
│   └── EntityType: k8s.service
└── Domain: acs
    ├── EntityType: acs.ecs.instance
    └── EntityType: acs.rds.instance

Data storage features

• Guaranteed uniqueness: The __entity_id__ is unique within the same EntityType.

• Columnar storage: Supports table-structured data with multiple rows and columns. Supports statistical analysis using Structured Process Language (SPL).

• Index optimization: Optimized for retrieval scenarios using full-text indexes. Supports multi-keyword retrieval and ranking scores.

• Time-series support: Supports data queries and filtering based on time ranges. Supports reviewing the status of entities and relationships at any point in time.

1. Retrieval capabilities

-- Retrieve entities that contain "cart" from all domains
.entity with(domain='*', name='*', query='cart') 

-- Retrieve entities from entity types that contain "service" in all domains, where the properties contain "production"
.entity with(domain='*', name='*service*', query='production')

-- Retrieve entities from multiple entity types under a specific domain, where the properties contain "error" or "rate"
.entity with(domain='apm', name='apm.*', query='error rate')

-- Multi-keyword OR query
.entity with(query='kubernetes pod error', topk=50)

-- Field weights affect the score
.entity with(query='name:cart OR ports:8080', topk=20)

2. Scan capabilities

Reads raw data and uses SPL for further filtering and calculation:

-- Perform SPL calculations after scanning
.entity with(domain='apm', name='apm.service') 
| where cpu_usage > 0.8
| stats count() by cluster

Basic syntax structure

.entity with(
    domain='domain_pattern',     -- Domain filter pattern
    name='type_pattern',         -- Type filter pattern  
    query='search_query',        -- Query condition
    topk=10,                     -- Number of records to return
    ids=['id1','id2','id3']      -- Exact ID query
)

Parameter details

Domain and type filter patterns

Filtering supports fnmatch syntax:

-- Pattern matching examples
.entity with(domain='ac*')           -- Domains that start with ac
.entity with(domain='a*c')           -- Domains that start with a and end with c  
.entity with(name='*instance')       -- Types that end with instance
.entity with(name='k8s.*')           -- All types under the k8s domain
.entity with(domain='*', name='*')   -- All domains and types

1. Exact ID query

When you know the specific entity IDs, you can use the `ids` parameter for an exact query:

-- Query entities with specific IDs
.entity with(
    domain='apm', 
    name='apm.service', 
    ids=['4567bd905a719d197df','973ad511dad2a3f70a']
)

Scenarios:

• Query detailed information based on an entity ID from an alert.

• Verify the existence and status of a specific entity.

• Query information for a batch of known entity IDs.

2. Full-text search mode

-- Simple keyword search
.entity with(query='web application')

-- Multi-word OR relationship (default behavior)
.entity with(query='kubernetes docker container')

-- A full phrase match returns only `{"opentelemetry.io/name":"fraud-detection"}`
.entity with(query='opentelemetry.io/name-fraud-detection')

-- A normal search hits any word and returns all data
.entity with(query='opentelemetry.io/name cart')

Field-specific search

You can specify a search within a particular field:

-- Search in the description field
.entity with(query='description:"error handling service"')

-- Search in a custom property
.entity with(query='cluster_name:production')

-- Search in a label
.entity with(query='labels.team:backend')

Combining logical conditions

Supports `and`, `or`, and `not` logical operators:

-- AND condition: Must satisfy both conditions
.entity with(query='service_name:web AND status:running')

-- OR condition: Must satisfy either condition
.entity with(query='environment:prod OR environment:staging')

-- NOT condition: Must satisfy the left side but not the right side
.entity with(query='type:service NOT status:stopped')

-- Complex combination
.entity with(query='(cluster:prod OR cluster:staging) AND NOT status:maintenance')

Handling special characters:

• Queries that contain special characters, such as | and :, must be enclosed in double quotation marks.

• Example: query='description:"ratio is 1:2"'.

Relevance scoring

USearch uses a multi-factor scoring algorithm:

1. Term frequency: The frequency of a keyword in a document.

2. Field weight: The importance weight of different fields. For example, the name field has a higher weight than the description field.

3. Document length: Matches in shorter documents usually receive higher scores.

4. Inverse document frequency: Rare terms receive higher weights.

Sorting rules

By default, results are sorted in descending order of relevance score. If scores are the same, results are sorted by timestamp:

-- Default relevance sorting
.entity with(query='web service error', topk=20)

-- Custom sorting with SPL
.entity with(query='kubernetes pod')
| sort __last_observed_time__ desc| limit 50-- Multi-field sorting
.entity with(domain='apm', name='apm.service')
| sort cluster asc, service_name asc