Skip to main content

Action System

actions2

The SkyrimNet Action System allows NPCs to perform gameplay actions alongside natural conversation. Rather than treating dialogue and gameplay as separate systems, NPCs can decide during a conversation whether an action should accompany their response, resulting in more coherent and believable interactions.

Examples include:

  • Giving an item
  • Opening a shop
  • Starting combat
  • Following the player
  • Teaching a spell
  • Triggering custom mod events

Embedded Action Evaluation​

By default, actions are evaluated as part of the dialogue generation process (embedActionsInDialogue = true).

During dialogue generation, the LLM determines:

  • What the NPC should say.
  • Whether the NPC wants to perform an action.

If no action is appropriate, the conversation ends normally and no additional LLM call is made.

If the NPC intends to perform an action, the dialogue model outputs:

  • A top-level action category.
  • A natural-language intent describing what it wants to accomplish.

For example:

NPC:
"You look badly injured. Here, take this."

INTENT:
ACTION : {Give the player a healing potion.}

the action LLM will receive this intent , select the action from the category list and output:


ACTION: GiveItem Params {"akTarget: "Bob the Player" , "Item: Healing Potion"}

This approach significantly reduces latency because most conversations never require a second action-selection request.

Hierarchical Action Categories​

Actions can be organized into hierarchical menus using the customCategory YAML field.

Instead of exposing hundreds of individual actions to the dialogue model, only high-level categories are presented.

For example:

Inventory
Combat
Trading
Magic
Travel
Conversation

Once a category is selected, SkyrimNet performs a lightweight drill-down evaluation using only the actions contained within that category.

This provides several benefits:

  • Smaller prompts
  • Lower token usage
  • Faster evaluation
  • Better action selection
  • Unlimited scalability as more actions are added

Two-Stage Action Selection​

The action system uses a two-stage process.

Stage 1 — Dialogue Generation​

The dialogue model determines:

  • The NPC's spoken response.
  • Whether an action should occur.
  • The appropriate action category.
  • The NPC's intent.

Example:

NPC:
"I can sell you one if you'd like."

Category:
Trading

Intent:
Sell the player a healing potion.

Stage 2 — Action Drill-Down​

If an action was requested, SkyrimNet performs a second lightweight LLM call.

Unlike the dialogue prompt, this prompt contains only:

  • Eligible actions within the selected category.
  • The conversation.
  • The intent generated by the dialogue model.

The drill-down model's responsibility is not to decide whether an action should happen. Instead, it selects the specific action that best fulfills the provided intent.

For example:

Trading

- Sell Potion
- Sell Weapon
- Open Shop
- Buy Item

Intent:

Sell the player a healing potion.

Selected action:

Sell Potion

Intent-Based Selection​

Rather than requiring the dialogue model to choose a specific implementation, it simply describes the desired gameplay outcome.

For example:

Give the player a healing potion.

instead of:

Execute GivePotionAction(ID=17)

This separation allows the drill-down model to choose the most appropriate implementation from the available actions, including those added by mods.

Lightweight Drill-Down​

The second LLM call is intentionally minimal.

Unlike dialogue generation, it does not include:

  • Conversation memories
  • Event history
  • Location context
  • NPC biography
  • World knowledge

It only receives the information necessary to choose an action.

This reduces prompt size by roughly 60%, resulting in lower cost and faster responses.

Reduced Action Bias​

One concern with exposing actions directly to the dialogue model is that some language models become overly influenced by the tools they can see.

The hierarchical system minimizes this issue by exposing only broad categories during dialogue generation.

The dialogue model never sees the hundreds of specific actions available within each category, allowing conversations to remain natural while still enabling complex gameplay interactions when appropriate.

Legacy Action Evaluation​

The previous action evaluation mode is still available.

In that mode:

  1. The dialogue model generates a response.
  2. A second LLM call always decides whether an action should occur.

While fully supported, this approach is no longer recommended because it:

  • Requires an additional LLM request after every response.
  • Uses more tokens.
  • Increases latency.
  • Produces less accurate action selection than the embedded workflow.

Benefits​

The current action system provides:

  • Natural integration between dialogue and gameplay.
  • Lower latency by avoiding unnecessary LLM calls.
  • Approximately 60% lower token usage during action selection.
  • Improved action accuracy through intent-guided drill-down.
  • Better scalability with large numbers of actions.
  • Cleaner separation between conversational reasoning and gameplay implementation.
  • Easy extensibility for custom mods and native actions.

By combining embedded dialogue evaluation with hierarchical action categories and intent-guided drill-down, SkyrimNet enables NPCs to perform complex gameplay behaviors while maintaining fast, natural, and coherent conversations.