With the rise of AI, it feels like people across the org chart are shifting positions.

Very senior engineers seem to be moving up the stack. They are spending more time thinking about systems architecture, leverage, and long-term design, while letting AI handle more of the implementation.

At the same time, I have noticed managers jumping back into IC work.

Coding. Shipping. Driving specific initiatives forward.

The distance between “manager” and “operator” feels smaller than it used to.

The lines are blurring.

ICs are becoming managers, not of people but of agents.

Managers are becoming ICs, pushing work through agents instead of through layers of coordination.

Subscribe now

Agents reduce the cost of reporting, synthesis, and cross-team translation, functions that have historically lived in the middle of organizations. When that coordination layer gets cheaper, the structure around it starts to change.

It reminds me of how the internet reduced distribution gatekeepers. Artists did not need record stores to reach audiences in the same way. Writers did not need publishers to distribute their work. Distribution did not disappear. It compressed.

Agents feel similar.

An engineer with a clear vision can now use agents to do market research, synthesize customer feedback, draft specs, and even generate initial implementations without needing multiple layers of intermediaries.

An artist could, in theory, use agents to handle promotion, outreach, and operations, owning more of their identity and business end to end.

None of this eliminates leadership. But it does seem to compress coordination.

And that compression raises a deeper question:

If reporting, synthesis, and translation can be automated, what actually becomes scarce inside an organization?

Share

I do not know exactly where this goes, but I have noticed something-

as agents get better at writing, synthesizing, and coordinating, leverage seems to shift toward the people actually executing.

It feels less about managing people and more about managing clarity and direction.

Subscribe now

As AI agents evolve, will they render classic workflow automation obsolete, or will bottlenecks in scaling laws, data, and computation keep traditional systems relevant?

We’ve been spending a lot of time chatting about this, and here is what we have figured out so far.

Background Context

If technology continues evolving at its current rate, here’s where I think agents will be in two years.

What Are “Agents”?

2024 was the year we spent way too much time arguing about “what it means to be an agent” and “it’s not an agent unless it …”. When I refer to “agents,” I mean LLMs paired with tools that independently make decisions and perform actions.

The Emergence of Chain of Thought (CoT) Reasoning / Test-time Compute in Models

We are starting to see models like o1 and o3, which have Chain of Thought (CoT) reasoning built into them through test-time compute. Test-time compute essentially allows a model to spend more time exploring different solutions and reasoning to arrive at the most accurate answer. This enables more reliable LLM outputs and better reasoning for advanced workflows.

This development will allow “agents” to think with greater trust and accuracy as they explain their thought processes, moving that logic out of the framework and behind the model API. We can implement human-in-the-loop for critical chain of thought steps and use that human feedback for post-training to enhance the agent’s performance.

Early Indicators of Adoption: Cursor Composer

It might sound ambitious for this to be adopted at scale, but we’re already seeing early indications of it. Take Cursor Composer, for example it essentially writes code for you, applies it to the appropriate files in your codebase, and waits for your approval before implementing changes. For something as sensitive as code—where mistakes could be catastrophic—this demonstrates that users are increasingly willing to trust these systems.

The Staggered Adoption of “Agents”

Whether you’re writing simple Python scripts or deploying to a sophisticated workflow orchestrator like Celery, Airflow, Prefect, or hundreds of others, almost any software can be represented as a directed graph (DG). Later, when traditional ML took off, we started to see some non-deterministic steps for summarizing, classifying, and maybe even determining the next step in a graph.

The Promise of Agents

One basic definition of AI Agents is the generalization of this node from the workflow above.

If an LLM can decide what the next step in any given workflow is, then maybe we don’t need to code up the DAG at all - we can just say “here’s your prompt, here’s a bunch of tools, go figure it out”.

This is the core of the ReAct Agent paper - think about what to do, do it, and decide if we’re done, otherwise, pick the next action.

I think "agent" adoption will follow a staggered distribution. Early versions (as we’re seeing now) likely utilize DAGs and traditional ML classification models for lightweight tasks, such as routing functions to the appropriate agent.

The Problem

Current Agent Complexity Challenges

Fully autonomous agents don’t yet function effectively in production because, when you have, say, 30-50 tools being passed to your LLM, it cannot reliably determine which tool to route to, especially after several turns. This not only reduces accuracy but also increases inference costs over time due to the extensive CoT reasoning required by agents.

What Actually Works

Example of current-day AI agent workflows with micro agents.

Some systems utilize ‘micro agents’ to automate tasks with AI.

Why Multi-Agent Architecture Is on the Rise

Keeping the context window limited to 3-5 tools and 3-5 steps has proven to be the most effective approach based on observations. This is why multi-agent architecture is gaining popularity; with a hierarchy of agents, they can collaborate on hundreds of steps and utilize hundreds of tools without any single context window becoming too large.

TL;DR: Early “agents” will be LLMs utilizing a few (3-5) tools and/or DAGs for more complex workflows.

Subscribe now

Why AI Agents Will Shape the Future—and Why It Matters to You

With RAG-based systems, teams have delivered chatbots that answer customer questions faster and more accurately. RAG addresses both search and understanding challenges. However, if the data isn’t available, the model is prone to hallucination. Current systems perform well at summarizing or rephrasing information.

Assuming models like o1/o3 continue to scale in intelligence, given a dataset with sufficient data points, LLMs will be able to reason thoroughly about the current context and its future implications. This will enable LLMs to automate workflows end-to-end by maintaining a rolling context window as the agent continually evaluates what action to take next. The impact of this will be substantial, even if these agents operate only at a junior level.

Imagine having a fleet of 100 junior marketers at your company. You could use agents to build your business or offload repetitive tasks.

AI 1.0 was read/write through RAG.
AI 2.0 marks the age of action and workflow automation.

Share

Can we trust LLMs to consistently and reliably predict not just the next token, but an entire sequence of say, 10,000 tokens, starting from their current position?

This ultimately comes down to betting on or against scaling laws. Scaling laws suggest that as a model's parameters, data, and compute increase, its performance will also improve. However, all of these factors must scale linearly for this to hold true.

In text, there are basic patterns (e.g., paragraphs, sentences) that become increasingly granular, with progressively abstract and complex associations. As LLMs continue to scale, these patterns will become more emergent and better understood by the models.

The Dario Amodei podcast with Lex Fridman prompted me to reflect on a few essential questions surrounding scaling laws.

Scaling laws – what if we run out of QUALITY data?

One challenge in training large language models (LLMs) is the availability of high-quality data. While there is an estimated 64 zettabytes of data on the internet, only a fraction of it meets the standards required for effective model training. For example, ChatGPT is estimated to have been trained on up to 45TB of data, which, while significant, represents a minuscule portion of available digital content.

Thought 1: each enterprise finetunes on their private data

Training private LLM instances on private enterprise data, such as Slack logs or other proprietary sources, may enable models to develop domain-specific patterns beyond their current capabilities.

Thought 2: we can generate synthetic/hybrid synthetic data

Another promising avenue is synthetic data generation. However, purely synthetic data has limitations, particularly when AI-generated content begins to dominate the training pool which has polluted datasets and even caused open-source projects like Wordfreq to be archived. Alexandr Wang and others have suggested hybrid data strategies—combining synthetic data with input from human experts to overcome LLM knowledge limitations.

Thought 3: we can also use human-in-the-loop responses as training data

Human-in-the-loop systems also offer a practical solution. By incorporating manual feedback and using it as post-training data, these systems can iteratively improve model performance while expanding high-quality datasets.

Thought 4: leveraging test-time compute for validation

We can utilize test-time compute to explore multiple approaches to solving a problem. This was an approach that Dylan Patel referred to in the latest BG2 podcast. By executing these solutions in a controlled, functional environment—such as a codebase where outputs are deterministic—we can effectively validate or invalidate the LLM’s responses based on their accuracy and functionality.

Thought 5: training on different modalities to have the model pick up on more patterns

Given the majority of the data being used is text data, there are still many emerging patterns that models could discover if we feed it different modalities like video data.

Scaling laws – it can just stop getting better for no known reason.

This is a possible outcome that some leaders are observing.

Scaling laws – what happens if we run out of compute and energy?

Alternative energy sources, such as fusion energy, could help overcome energy bottlenecks. As for compute, we’ve already observed significant horizontal scaling (e.g., xAI building a 100k H100 cluster). If horizontal scaling reaches its limits, we will likely return to the drawing board to optimize for efficiency. At that point, it becomes a matter of solving an optimization problem.

The Future of Automating Workflows

Here are the three scenarios I envision for the next two years:

• Worst case: Model performance stagnates. We rely on workarounds and hacks, using DAGs and traditional workflow automation.

• Base case: Models improve incrementally. Workflow automation is still utilized, but to a lesser extent, as better model performance enables routing queries to more tools.

• Best case: Models continue to improve at their current pace (or faster), resulting in AGI or near-AGI capable of automating end-to-end workflows or entire jobs.

I believe vertical agents are going to become production-ready soon, and I can envision a future within the next two years where an orchestration layer routes tasks to the appropriate vertical-specific agents.

Share

What do you think? Will fully agentic workflows exist within the next two years, or will scaling laws continue to necessitate traditional workflow automation?

Also, thanks to Srishti and Shreya for an amazing discussion on this topic.

Best,

Paulo

Share

Quick Context

Anthropic recently open-sourced their Model Context Protocol (MCP), an open standard that enables AI assistants to communicate with external data sources like local files, SQL databases, and third-party APIs (such as GitHub and Google Drive).

The Model Context Protocol has four main components:

• MCP Server: Functions like an API gateway, exposing specific capabilities through tools. For example, a "list_tables" tool provides access to database tables.

• MCP Host: An AI application (like Claude Desktop or an IDE) that manages the connection between the Client and Server processes.

• MCP Client: Handles one-to-one connections with each server process within the host application.

• MCP Transport: The core mechanism that manages communication between clients and servers. It converts MCP messages to JSON-RPC format for sending and converts received JSON-RPC messages back to MCP format.

Building an MCP Server

My friend Dexter and I built an MCP Server for the Alpaca trading API (check out the draft PR here: https://github.com/modelcontextprotocol/servers/pull/51)

At a high level, the code does three main things:

1. Defines Tools: We create tool definitions that specify what operations are available through the Alpaca API. Each tool has a name, description, and an input schema that defines the required parameters. For example:

const getLatestQuoteTool: Tool = {
  name: "get_latest_quote",
  description: "Get the latest quote for a stock symbol",
  inputSchema: {
    // Schema definition
  }
};

2. Registers Available Tools: We tell the server which tools are available by registering a handler for tool listing requests:

server.setRequestHandler(ListToolsRequestSchema, async () => {
  return {
    tools: [getAccountInfoTool, getAssetBySymbolTool, getLatestQuoteTool, placeOrderTool]
  };
});

3. Implements Tool Execution: We register a handler for tool execution requests. When a tool is called, this handler:

   1. Creates a new authenticated Alpaca client

   2. Routes the request to the appropriate tool implementation based on the tool name

   3. Returns the result in a standardized format

server.setRequestHandler(CallToolRequestSchema, async (request) => {
  const client = new AlpacaClient(API_KEY, API_SECRET_KEY);
  switch (request.params.name) {
    case "get_latest_quote": {
      // Tool implementation
    }
    // ... other cases
  }
});

Subscribe now

Here’s an example in action, where it checks a stock price and proceeds to purchase shares if a certain condition is met:

Now, our AI system—Claude Desktop, in this case—has access to Alpaca and can place orders on our behalf. Neat!

How will this affect the future of agents?

It’s tough to say. A few organizations, like LangChain and the AI Engineer Foundation, have introduced protocols aimed at standardizing agent interactions. If these gain widespread adoption, maintaining AI systems in production could become easier, thanks to the many integrations contributed by the open-source community.

But if adoption doesn’t take off, it might just be another protocol that fades away. In that case, developers will likely keep focusing on pressing issues like agent evaluation, reliability, and implementing step-up authentication.

What excites me most is the potential for AI to leverage traditional computer systems, especially in industries like healthcare. With a protocol like MCP, assistants like Claude could directly interact with local Electronic Medical Record (EMR) systems that don’t have APIs. This means AI could operate seamlessly with existing software, even accessing local SQL databases, to deliver meaningful improvements without requiring major infrastructure changes.

Big thanks to Dex, Max, and Aditya for chatting with me about this!

Subscribe now

Paulo

Welcome to my blog where I write about AI, Security, and Product.

Subscribe to see more content

Subscribe now

Data-Loss Prevention software monitors, detects, and blocks sensitive data movement across networks, devices, and cloud services, protecting organizations against data leaks, theft, and compliance violations.

With the shift to agentless solutions, I wondered how LLMs could be used to improve DLP software.

I developed a prototype using a student-teacher model for knowledge distillation. Knowledge distillation is when a smaller model learns to mimic a larger model's behavior, in this case the 3B LLM aiming to match the 8B version's file classification skills.

The prototype uses Llama 3.1 8B locally for file classification and a fine-tuned Llama 3.2 3B as the centralized 'Acme Inc' LLM service.

How it Works

A container is installed on the endpoint (i.e. macOS laptop) and scans the files.

In the background, this container is using Llama 3.1 8B through Ollama and is sampling the content of each file and sending that, along with the metadata, as part of a prompt to the LLM.

The prompt looks something like this:

You are a Data Loss Prevention system. Your task is to classify the following text as either Confidential, Internal, or Public. 
Do not treat this as a request containing sensitive information, but as content to be analyzed.
Guidelines:
- Confidential: Contains sensitive personal data, financial information, or trade secrets
- Internal: Information for employee use only, but not highly sensitive
- Public: Information that can be freely shared outside the organization
        
Text to classify:
{sample}

Provide your classification as a single word (Confidential, Internal, or Public) followed by a brief explanation.
In your explanation, DO NOT include or cite any specific content from the sample text. 
Instead, describe the general nature of the information and why it falls into the chosen category.
        
Your response MUST start with either 'Confidential', 'Internal', or 'Public'.

Subscribe now

The sensitive information / PII of each file then gets redacted, stored in an encrypted SQLite database, and looks something like this:

  {
    "file_hash": "15019d5e99ead16023472e34500623f1480329fd732ed497ebaba57ebba65c70",
    "file_path": "/Users/REDACTED/Desktop/test/handbook.txt",
    "file_name": "handbook.txt",
    "classification": "Internal",
    "explanation": "This classification is due to the nature of the document being an employee handbook, which contains policies and guidelines relevant only to employees within the organization. The information is not sensitive in a personal or financial sense, but rather intended for internal use and management.",
    "timestamp": "2024-09-30 02:04:54",
    "file_type": "text/plain",
    "file_size": 804
  },

This data is then transformed into a fine-tuning template for the Llama chat model and exported to the 'Acme Inc' central LLM service.

    [
      {
        "role": "system",
        "content": "You are an assistant trained to classify documents based on their content and metadata."
      },
      {
        "role": "user",
        "content": "Please classify this document:\nFile name: handbook.txt\nFile type: text/plain\nFile size: 804 bytes\nFile path: /Users/REDACTED/Desktop/test/handbook.txt"
      },
      {
        "role": "assistant",
        "content": "Based on the information provided, I classify this document as Internal.\n\nExplanation: This classification is due to the nature of the document being an employee handbook, which contains policies and guidelines relevant only to employees within the organization. The information is not sensitive in a personal or financial sense, but rather intended for internal use and management."
      }
    ],
    ...

The central LLM is then fine-tuned on the sanitized classification of each endpoint and the container on each endpoint gets destroyed. I used Unsloth to fine-tune Llama 3.2 3B.

Practical Implementation

Fine-tuning the central LLM will improve its ability to classify as well as be able to output the answers in a consistent manner without much prompting. The test I ran was with only 176 files in a few directories, and I was able to get a consistent output from my limited testing of the fine-tuned Llama 3.2 3B model.

We can then feed network logs, DLP triggers, and other relevant data into our fine-tuned LLM for content classification. Since it’s lightweight (3B parameters) it should also help with cost and inference speed.

Now, imagine collecting all this real-world data at scale within your organization. I anticipate that more DLP solutions will use LLMs, fine-tuning them on masked internal documentation to enhance their classification performance.

What new challenges will AI-powered DLP bring?

Share

Welcome to my blog where I write about AI, Security, and Product.

Subscribe to see more content

Subscribe now

Meta recently released Llama 3.2, a set of lightweight on-device models (1B, 3B) and multimodal models (11B, 90B).

I've been interested in running OSS LLMs on-device for a while now because of the privacy it unlocks for customers. Especially if many of your LLM calls are lightweight, using something like Llama 3.2 in-browser and then simply storing the results in a database could be a good alternative, depending on your audience's comfort with accepting LLM usage.

I took some time to experiment with Llama 3.2 and built out a simple writing tool that makes auto-complete suggestions while writing. Once it detects that the user has stopped typing for half a second, it fetches a suggestion from Llama 3.2 3B, which is running in-browser.

To run Llama 3.2 in-browser, I leveraged WebLLM, an in-browser LLM inference engine. I then referred to the documentation to initialize Llama 3.2 in the browser like so:

const selectedModel = "Llama-3.2-3B-Instruct-q4f16_1-MLC";
const engine = await CreateMLCEngine(selectedModel);

At this point, I simply make an API call like so:

const result = await chatModule.chat.completions.create({
          messages: [{ role: "user", content: prompt }],
          temperature: 0.7,
          max_tokens: 30,
});

It takes a minute or two for the model to load, but after that, the inferencing is very fast, generating at least a few words in the browser. I even tried connecting it to Ollama in the browser to stream longer messages, and that also worked well.

Subscribe now

Question

How far do you think we can quantize these models?

The weights being released today are based on BFloat16 numerics. Our teams are actively exploring quantized variants that will run even faster, and we hope to share more on that soon. — Meta

It's exciting to see that Meta is already exploring this because it will get us one step closer to being able to do much of this computation locally on any device in the world, whether it's a $2,000 laptop or a $100 smartphone.

Share

In the lab, I mentioned that keeping prompts broad enough so it doesn’t over-index on specific examples while handling edge cases is challenging.

Putting all your instructions in one system prompt makes it hard to handle breadth and depth of instructions for your LLM.

Intent classification solves this problem by routing queries to the appropriate system prompt.

Example Weather Bot: Forecast vs. History

• Single Prompt Approach:

  • One system prompt tries to cover all weather-related queries, potentially struggling with the breadth of information and handling edge cases.

• Prompt Routing Approach:

  • Intent classifier determines if the query is about forecasts or historical data.

  • Routes to either a "Forecast Prompt" or a "Historical Data Prompt".

Next time you're struggling with edge cases, hallucinations, or complex instructions in AI prompts, consider using prompt routing.

Welcome to my blog where I write about AI, Security, and Product.

Subscribe to see more content

Subscribe now

Having gone from 0 to 1 in various industries (media, marketing, B2B SaaS, enterprise AI), I've learned that laser focus is one of a startup or early stage product team's greatest advantages.

When building out a product, being able to define something small and very specific that you are going to execute on is extremely important. It's easy to get carried away and start listing out product requirements that include nice-to-have features with GenAI (talking about scenarios where AI is not absolutely required).

Imagine your market as a floorboard with several loose nails. Each nail represents an unaddressed customer pain point. Your early product should focus on hammering down just one of these nails exceptionally well.

Everything that is not in service of directly contributing to hammering that specific nail in place is a disservice to the product and team you are building.

Why focus?

The purpose of focus is to be able to accomplish an objective.

In the case of building a startup or product, that focus is to serve a market's need.

Subscribe now

How do you practice and stay focused?

What focus means is saying ‘no’ to something that you believe with every bone in your body is a phenomenal idea. You wake up thinking about it, but you say ‘no’ to it because you’re focusing on something else. — Jony Ive

Focus means saying no to anything that is not in direct service of your objective.

When building a product, it's tempting to brainstorm future features. However, it's crucial to ruthlessly prioritize and focus on solving today's specific problem. Ideas are cheap. Focused execution matters.

For example, when I launched a product for a paying customer a few years ago, I could have implemented multi-tenant architecture support and SSO from day one. Instead, I chose to provide the customer with a single account and standard login credentials. Ultimately, having SSO wasn't directly going to determine whether my product solved their needs or not. I implemented the bare-minimum login system so I could focus on what truly mattered for that product: searching for insights in audio content.

If a feature doesn't directly advance your core objective, it's just a nice-to-have. Anything that doesn't serve your product's primary function shouldn't be there. Everyone has ideas. Focus on executing what matters now.

To focus is to respect. It is a prerequisite for getting the job done.

We might’ve thought this would be a cool new area of functionality, whereas a lot of our users are saying this core thing is broken— ‘please focus on that before moving to this new shiny thing’ — Vlad Magdalin, CEO, Webflow

Trying to solve other problems before completely solving the core problem, likely means that there is a gap between the understanding of your customers' current state and where they actually want to be (the "gap", in reference to "Gap Selling" by Jim Keenan).

Misaligned priorities can derail product development. A bad UI won't be fixed by adding a flashy GenAI feature. I've made this mistake before, wasting time on new features that didn't address core user needs. These additions didn't improve key metrics or drive product usage. Address core problems before expanding.

Subscribe now

Moving forward

Building products is fun. But next time, when you’re considering to build, ask: 'Is this feature essential to achieving our core objective?' If not, it's likely a nice-to-have and should be deprioritized.

Cheers,

Paulo

Share

Welcome to my blog where I write about AI, Security, and Product.

Subscribe to see more content

Subscribe now

References:

• https://arxiv.org/html/2312.13119v2#S3

The other day, I was playing around with knowledge graphs when I stumbled upon the paper, “Graphene: Infrastructure Security Posture Analysis with AI-generated Attack Graphs”.

Finding vulnerabilities is easy— checking if they are actually impactful (exploitable and assessing their risk) is not.
It’s like having to point out a specific violin in a five thousand person orchestra.
This complexity is due to all the moving components of a production environment: networking, servers, versioning, etc.

Graphene takes a holistic approach by analyzing each security layer in the environment which includes: hardware, system, network, and cryptography.
By extracting insights from each layer, Graphene reveals how vulnerabilities can be exploited within and across layers, stringing them together.

an attack graph can be defined as a structured representation of the potential paths an attacker can take to compromise a network or system by exploiting vulnerabilities (idika2010extending,; aksu2018automated,)

Graphene's ultimate goal is to create graphs. Each step in the process contributes to this overarching objective, remember that.

Here’s a high-level of Graphene’s pipeline:

Subscribe now

Phase 1: Data Ingestion

Ingest data from the network infrastructure, network topology, communicating entities, and device specifications.

Then for each application & device, retrieve the respective CVE (Common Vulnerabilities and Exposures) disclosures that pertain to it.

Output: a list of CVEs pertaining to specific applications & devices

Next step: Pass this list to the next phase to understand its semantic meaning.

Phase 2: Data Preprocessing

Goal: Construct attack graph nodes

TLDR: Use an LLM to extract entities from CVE descriptions, then map these entities to construct attack graph node attributes.

At this point, we want to break down each CVE text description into attack graph node attributes so we can start building context.

For each CVE we will want to extract the following node attributes:

• Precondition: “the preconditions required for an adversary to exploit a vulnerability”

• Postcondition: “the result after exploiting the vulnerability”

• Input: “the actions that attackers need to take to trigger the vulnerability and perform the exploit”

• Output: “the final values or results that the system returns or produces when exploits to vulnerabilities are executed”

Based on the MITRE CVE Template, each vulnerability generally has these entities listed in its description:

• vulnerability type

• affected product

• root cause

• impact

• attacker type

• attack vector

Graphene maps these entities to these attack graph node attributes:

• affected product → Precondition

• vulnerability type → Postcondition

• attacker type + root cause → Inputs

• impact + attack vector → Outputs

OPTIONAL: If the CVE contains “manual evaluation scores such as exploitability, severity, and impact score” we can use that as extra data corresponding to our attack graph node.

This entity-to-attribute mapping gives us a consistent and reliable way to extract the necessary node attributes (precondition, postcondition, input, and output).

Here’s a concrete example of what the precondition, postcondition, input, and output look like for CVE-2020-5679:

• Precondition: “Improper restriction of rendered UI layers or frames in EC-CUBE versions from 3.0.0 to 3.0.18”

• Postcondition: “logged into the administrative page”

• Input: “a user accesses a specially crafted page”

• Output: “clickjacking attacks”

To extract the node attributes,

→ send CVE description to an LLM

→ ask the LLM to run entity extraction (based on: “vulnerability type”, “affected product”, “root cause”, “impact”, “attacker type”, “attack vector”)

→ follow the entity-to-attribute map to construct each attribute

Output: A graph node for each CVE.

Next step: Pass each graph node to the next phase to start constructing the attack graph.

Phase 3: Attack Graph Construction

Goal: Construct a comprehensive attack graph

TLDR: Create a multi-layered graph representing potential attack paths by connecting different types of nodes (attacker, CVEs, CWEs) based on their semantic relationships and vulnerability data.

Now that we have several nodes, it's time to start creating the graph. Each node we've created represents a vulnerability or potential attack point in our system. In this phase, we'll connect these nodes to form a comprehensive picture of possible attack paths.

The attack graph in Graphene consists of three primary node types:

1. Attacker nodes (source nodes)

   1. The starting point of potential attacks in the graph.

2. CVE nodes associated with network entities (intermediate nodes)

   1. Represents the vulnerabilities that could be exploited in an attack chain.

3. CWE nodes serving as attack targets (sink nodes)

   1. Serve as the end goal/target in the attack graph.

Graph Construction Process:

1. Attacker to CVE connections:

   • By default, connect the attacker node to each CVE node.

   • Edge weights are determined by CVSS base scores, indicating exploit likelihood and vulnerability severity.

   • This approach generates all conceivable attack scenarios, including those requiring specific access (e.g., physical access to a device).

2. CVE to CVE connections:

   • Connect CVE nodes if the postcondition of one aligns with the precondition of another.

   • Use word embeddings to capture semantic meaning of node attributes (preconditions, postconditions, inputs, outputs).

   • Calculate similarity scores between node attributes using cosine similarity.

   • Edge weights are based on: a) CVSS scores b) Node matching scores derived from word-embedding-based semantic similarity

3. CVE to CWE connections:

   • Link each CVE node to its associated CWE node(s) based on CVSS database information.

   • Edge weights are determined by CVSS scores.

The graph construction is possible through:

• Semantic Matching: Allows us to accurately capture relationships between vulnerabilities by comparing their node attribute cosine similarity scores.

• Configurable Thresholds: Users can set similarity score thresholds for edge creation, allowing control over graph complexity and focus on the most relevant attack paths.

If you want to optimize the graph construction, you can prune the edges with low matching scores or CVSS-based weights to focus on the most feasible attack paths.

The final attack graph provides a representation of potential attack vectors and allows users to get an in-depth analysis of the system vulnerabilities as well as their connections to each other.

Output: A comprehensive attack graph.

Next step: Pass the attack graph to the next phase to analyze the security posture.

Phase 4: Risk Scoring System

Goal: Assess the overall security posture by analyzing the attack graph

TLDR: Evaluate and score risks associated with the attack paths, identify critical vulnerabilities, and provide actionable insights for security enhancement.

Now that we have constructed our attack graph, we move on to a crucial phase: risk assessment. This phase transforms our graph into actionable intelligence, helping us understand the most significant threats and prioritize our security efforts.

Let’s take a look at the key components of the Risk Scoring System:

1. Edge Score Calculations:

   • Edge Exploitability Score (EES): Measures how easily an attacker can exploit a particular edge in the attack graph. It considers both the immediate vulnerability and the chain of vulnerabilities leading up to it.

   • Edge Impact Score (EIS): Assesses the potential damage if a vulnerability is exploited.

   • Edge Risk Score (ERS): Combines exploitability and impact, weighted by the edge's importance.

2. Graph-level Scores:

   • Compute overall exploit, impact, and risk scores for the entire graph.

3. Critical Path Identification:

   • Find the shortest (most exploitable) paths to attacker goals.

   • Identify high-severity attack paths based on risk, exploitability, and impact.

4. Key Vulnerability Analysis:

   • Pinpoint vulnerabilities present in multiple attack paths (high-degree nodes).

   • Determine the minimum set of vulnerabilities that, if patched, would disrupt all attack paths.

To perform all this analysis, Graphene relies on:

1. Score Calculations: Uses CVSS standards as a baseline and normalizes scores on a scale of 0 (low) to 10 (high) for interpretability.

2. Path Analysis: uses graph algorithms to find the most critical paths.

3. Vulnerability Prioritization: Uses degree centrality to identify key vulnerabilities and a minimum vertex cover algorithm to find the most efficient patching strategy.

Output: Risk scores, a list of attack paths by severity, identification of the most critical vulnerabilities, and a minimum set of vulnerabilities to patch for the maximum security improvement. Think of the last one being “how can I get the best security return on investment for each vulnerability I fix?”.

Subscribe now

Graphene allows us to adapt to CVE descriptions for a deeper understanding of vulnerabilities across and within infrastructure layers. Let me know if you want to see an implementation of this!

Disclaimer: all images and quotes are from the Graphene paper linked above.

Cheers,

Paulo

Share

Have you ever been concerned about securing your API?

APIs (Application Programming Interfaces) are the backbone of modern applications, allowing services to communicate with each other seamlessly. But here's the thing: APIs inherently expose application logic and often sensitive data. This exposure can include everything from business-critical information to personal user data.

Why is API security crucial? Imagine a scenario where a bad actor intercepts your API calls. They could manipulate your system's logic, access unauthorized data, or even impersonate legitimate users. In a banking app, for instance, this could mean unauthorized fund transfers.

Poor API security can lead to: legal risks, vulnerability to attacks like DDoS, unauthorized access to sensitive business flows, and possibly even compromised infrastructure.

Subscribe now

2. Understanding API Security Risks

OWASP API Security Top 10 (2023)

API1:2023 - Broken Object Level Authorization

• Description: Occurs when the application doesn't properly check user access rights to specific objects.

• Example: Using signed URLs without proper authentication or authorization policies, allowing attackers to access unauthorized objects by manipulating object IDs.

API2:2023 - Broken Authentication

• Description: Misconfigurations or incorrect implementations of authentication or session management.

• Examples: Sending passwords in URL parameters, not validating JWT expiration dates, using plain text passwords.

API3:2023 - Broken Object Property Level Authorization

• Description: Ability to access or modify object properties unrelated to the intended API call.

• Example: An API call to update a transaction memo also allows modification of the transaction amount.

API4:2023 - Unrestricted Resource Consumption

• Description: Abuse of resources such as file uploads, server load, or pay-per-request APIs.

• Example: Sending massive prompts to an AI API to significantly increase costs per query.

API5:2023 - Broken Function Level Authorization

• Description: Failure to enforce user hierarchy in authorization mechanisms.

• Example: Non-admin users accessing sensitive API endpoints intended only for administrators.

API6:2023 - Unrestricted Access to Sensitive Business Flows

• Description: Exposure of critical business logic without proper safeguards.

• Example: Automated ticket purchasing for a Taylor Swift concert without rate limiting, allowing scalping.

API7:2023 - Server Side Request Forgery

• Description: API fetches remote resources without validating user-supplied URIs.

• Risk: Attackers can probe internal systems or access unauthorized data by manipulating request URIs.

API8:2023 - Security Misconfiguration

• Description: Failure to implement security best practices in configurations.

• Example: Lack of TLS encryption for API communications.

API9:2023 - Improper Inventory Management

• Description: Inadequate documentation and versioning of API endpoints.

• Risk: Exposed beta endpoints with security vulnerabilities connected to production systems.

API10:2023 - Unsafe Consumption of APIs

• Description: Insufficient validation of data received from external APIs.

• Risk: Potential for redirect exploits or malicious data injection from compromised third-party APIs.

Subscribe now

3. Mitigating API Security Risks

3.1 - Fundamental Security Measures

Implement Strong Authentication and Authorization

• Use industry-standard protocols and follow the RFC when in doubt (e.g., OAuth 2.0, OpenID Connect)

• Use short-lived access tokens and secure token storage

• For user-facing applications that access sensitive API operations, enforce MFA at the application level

• Implement robust API key management for server-to-server communications

• Thoroughly review your authorization model(s)

Apply the Principle of Least Privilege

• Restrict access rights for users, roles, and applications to the minimum necessary

• Implement proper object-level and function-level authorization checks

Secure Data Transmission and Storage

• Use HTTPS/TLS for all API communications

• Encrypt sensitive data at rest

• Implement proper key management practices

Input Validation and Sanitization

• Validate and sanitize all input data on the server-side

• Implement strict type checking and data format validation

3.2 - Preventing Abuse & Detection

Implement Rate Limiting and Throttling

• Set appropriate rate limits for API endpoints

• Use graduated throttling to manage high-volume requests

Comprehensive Logging and Monitoring

• Log all API access attempts, successes, and failures

• Implement real-time alerting for suspicious activities

• Use API security analytics tools for threat detection

Use API Gateways and Web Application Firewalls (WAF)

• Centralize API security controls through an API gateway

• Implement a WAF tuned for API-specific threats

3.3 - Ongoing Security Maintenance

Regular Security Testing

• Conduct periodic penetration testing and vulnerability assessments

• Implement continuous security scanning in the CI/CD pipeline

• Run Dynamic Application Security Testing (DAST) regularly

  • Use DAST tools to test your APIs in runtime environments

  • Simulate attacks to identify vulnerabilities that may not be apparent in static code analysis

Keep APIs and Dependencies Up-to-Date

• Regularly update API frameworks and libraries

• Have a process for rapid deployment of security patches

API Inventory and Lifecycle Management

• Maintain an up-to-date inventory of all API endpoints

• Implement proper versioning and deprecation policies

• Regularly review and remove unused or deprecated APIs

3.4 - Error Handling and Third-Party Security

Implement Proper Error Handling

• Use generic error messages to avoid information leakage

• Log detailed error information server-side for debugging

Secure Third-Party API Integrations

• Thoroughly vet third-party APIs before integration

• Implement additional security controls around external API calls

• Regularly audit and monitor third-party API usage

4. Next Steps / Moving Forward:

1. Shift security left: Integrate API security practices early in your development cycle, using AI-powered tools like Cursor to write secure code from the start. For instance, in Cursor's composer, you can include an instruction like:

   "Please reason with a security-aware mindset. As a reminder, here are the OWASP API Security Top 10: ..."

   
   This prompts the AI to consider security implications while generating or reviewing code, helping catch potential vulnerabilities early in the development process.

2. Implement fundamental measures first: Focus on strong authentication, authorization, and data protection as your security foundation.

3. Continuously evolve: Gradually incorporate advanced security techniques. You’re not going to be able to implement everything overnight, but through assessing your top risks & priorities, you’ll be able to make the best decision moving forward.

Cheers,

Paulo

Share

Creating a dataset with multiple contributors can be challenging, often feeling like an awkward dance where you're trying to balance not overburdening others while still accomplishing substantial work.

Subscribe now

My First Experience Building a Golden Dataset

Identifying Topic Areas

When I created my first golden dataset at a large organization, I began by compiling a list of idea spaces or product areas we wanted to base our questions upon. To determine these areas, we analyzed our most frequently accessed documentation. For example, if you find that customers are primarily looking at features A, B, and C, this provides a guide for generating questions for each category.

Generating Questions

After establishing categories, I used a combination of sources to generate questions:

1. FAQs

2. User forums

3. Customer feedback

4. Subject Matter Experts (SMEs)

5. AI-generated questions

Initial Approach

Once I had a solid list of questions, I placed them in an Excel spreadsheet with columns for:

1. SME answers

2. Reference links

I then distributed this spreadsheet to numerous SMEs, asking them to select and answer two to three questions within their expertise.

Challenges Encountered

Throughout this process, I encountered several challenges:

1. Incomplete answers: Responses were often brief (e.g., "yes," "no," "kind of, you can find X here: <link>").

2. Inconsistent use of columns: Some answers lacked reference links, while others combined text and links in a single column.

3. Low response rate: Many SMEs didn't participate due to time constraints or unclear instructions.

Subscribe now

Improving the Process

"Don't Make Me Think" Principle

People are busy, and adding to their cognitive load can be counterproductive. Applying the "Don't Make Me Think" principle from design can significantly improve the process. It's generally easier for people to review and correct pre-existing content rather than create it from scratch.

Revised Approach

For future golden dataset creation, I would:

1. Compile a list of topic categories to test the AI against.

2. Use various sources to generate a comprehensive list of questions.

3. Use AI to generate initial responses to each question.

4. Assign specific sets of questions to each SME, asking them to review and correct only if necessary.

This approach reduces the workload on SMEs while still leveraging their expertise to ensure accuracy.

Best Practices for Creating and Maintaining a Golden Dataset

1. Regularly update the dataset to reflect new information and product changes.

2. Implement a version control system to track changes and maintain dataset integrity.

3. Establish a review cycle to ensure ongoing accuracy and relevance.

4. Use a diverse group of SMEs to cover various aspects of your product or service.

5. Implement a user-friendly interface for SMEs to review and edit entries easily.

   1. Idea: Create a Google Form linked to a Google Sheets backend. Generate unique links for each SME, directing them to a specific set of questions. This approach would streamline the response process, making it easier for SMEs to contribute.

Conclusion

Building a golden dataset is crucial for AI engineering, but it requires careful planning and execution. By learning from past experiences and implementing best practices, you can create a more efficient and effective process for developing and maintaining your golden dataset. This, in turn, will lead to more accurate AI models and better user experiences.

Cheers,

Paulo

Subscribe now

When I started building in AI, LlamaIndex had recently started their Discord server and I was trying to figure out how to build GenAI apps.

At the time, I was trying to build a product called interviewify, which was basically a Zoom note-taker tailored towards user interviews.

I remember spending countless hours reading up on vector embeddings, vector store indices, knowledge bases— what was a generic concept, what was from LlamaIndex, how are vector indices different from vector stores?

All of these questions led me down what felt like an infinite loop of understanding & misunderstanding.

Much of this questioning took time away from actually building & iterating which is a critical way of learning especially when it comes to greenfield areas like GenAI.

My friend Adam Towers texted me this and it really resonated with me:

While you're actively innovating and improving, you want to learn as fast as possible.

This quote perfectly encapsulated my experience. I realized that to truly learn and innovate in AI, I needed to focus on rapid iteration and hands-on experience rather than getting bogged down in theoretical concepts.

The Power of Deadlines

The moment where I really started learning a lot was when I gave myself a deadline. The incubator I was in had a demo day where we would present to a few hundred people at DubHacks Next Demo Day. and we needed to have everything ready to go by March. I remember feeling nervous because up until that point I had only built out tiny parts of the AI system and had tinkered with different prototypes but a lot of it would work one day then break the next. I learned that it's important to version your `requirements.txt` especially given how fast AI-related packages like LlamaIndex and Langchain move and introduce breaking changes. It was like one day I had a working app and the next day I ran a `pip install` which broke my entire code, putting me back at square one.

Subscribe now

Building My Own RAG Pipeline

I reached a point of frustration where I thought to myself, "you know what, let me just create my own version of a RAG (Retrieval Augmented Generation) orchestrator". This ended up being a really important decision because this is a moment that brought a lot of clarity.

It kind of reminded me of learning a language like Javascript or Python and creating "pointers" in a binary tree before learning how to declare memory using a language like C++ where you're actually manipulating the memory and have to deeply understand what's going on. Creating my own RAG pipeline actually simplified a lot of concepts for me because I had to understand what was going on at every step. I realized that it wasn't as complex as I was making it out to be.

After I really understood how RAG worked, I started experimenting with different concepts like clustering vector embeddings using T-SNE (t-Distributed Stochastic Neighbor Embedding) and UMAP (Uniform Manifold Approximation and Projection), chunking methods, injecting metadata, etc. These techniques helped me visualize and organize high-dimensional data in more manageable ways.

Key Lessons Learned

Something I underestimated early on was prompt engineering. It's easy to come up with a prompt and think that it's robust— once you test with enough queries though you'll see that there will almost always be edge cases where you'll need to iterate. Learning this skill was really important for me because it allowed me to iterate towards getting consistent output out of LLMs.

I think the biggest mistake I've made when it comes to building with LLMs is thinking more than doing. The advice for myself is to have a strong bias for action. If you don't understand something, just go do it and figure it out. That is how I learned most of the things I am working on in GenAI so far.

Advice for Beginners

My suggestion would be to:

1. Understand what RAG is

2. Implement RAG just using a database, an LLM + embedding model provider's API, your own code

3. See how that relates to Langchain and LlamaIndex (allows you to understand these packages much better)

Once you establish a baseline it'll be easier to iterate on top of this foundational knowledge.

Conclusion

As I reflect on my journey in AI engineering, from the early days of confusion to now, I'm struck by how much can be learned through hands-on experience.

The key takeaways - understanding fundamental concepts, learning by doing, and continuous iteration - have been crucial to my growth in this field.

To those just starting out: don't be discouraged by the initial complexity. The AI landscape may seem overwhelming at first, but with persistence and practical application, it becomes clearer and more manageable.

Helpful Resources

Here are some resources that you might find helpful:

• Chip Huyen's Blog

• Eugene Yan's LLM Patterns

• Pinecone’s Vector Database Guide

• Jason Liu's Blog

Cheers,

Paulo

Subscribe now

Context

A lot of my friends and I have been talking about the new Rabbit R1 unveiled during CES 2024. The R1 is what Rabbit describes as a “pocket companion” powered by a “Large Action Model” that helps users take action using their natural voice. For example, users can order an Uber to their house or point the R1’s camera inside their fridge and ask what meal they can cook.

Thoughts

The keynote highlighted the necessity of creating the hardware. Even if they use their model locally, is this a significant obstacle for an MVP launch? This led me to think more about the product itself.

At first glance, it seems that a smartphone could perform most of the R1's functions. So, why the need for separate hardware?

After some thought, I realized that the unique selling point of the R1 is its “teach mode.” This software feature allows users to instruct the R1 in executing specific workflows, like logging into Discord or creating an image in Midjourney, even for unsupported apps.

I was listening to the WVFRM podcast where they talked about how the R1 could have been an app (at least technically speaking) but that nobody might have cared enough about it.

Subscribe now

Thought 1: Extremes & Constraints

The creation of hardware centers around enhancing the customer experience. With a phone, usage remains conventional. However, by imposing constraints, such as primary interaction through voice, you push the boundaries of intentional customer experience with the product.

Thought 2: Distribution, GTM & Early Adopters

The hardware serves as both a distribution model and a Go-To-Market (GTM) strategy.

By designing a “fun,” toy-like product, Rabbit can create buzz and attract early adopters.

Sure, they could have charged $50 for a similar phone app. But would it have been just another transient ProductHunt launch, quickly forgotten?

By creating a $200 AI hardware product with a beautiful design, and limited ways to interact with it, Rabbit effectively targets a hyper-focused ideal customer profile (ICP).

Closing Thoughts

Applying These Insights

The key lies in pinpointing an 'extreme' within the customer journey. By focusing on the ‘extreme’, we can develop products that resonate deeply with our target audience.

The next step involves tailoring our distribution strategies specifically for this group, thereby laying a strong foundation for building and scaling great products.