Be a Codex Prototyping Hero (for PMs/Designers)

Build a voice-enabled app that securely interacts with your email and calendar (GMail & GCal)

Hosted by Matthew Moore Ex-Google, Ex-Samsung, Multi-Exited Founder
Concepts

AI Assistant vs AI Agent

AI Assistant AI Agent
Human makes a task, subtask, AI uses tools, answers. Human makes a task, AI makes subtasks in a loop, eventually answers.
Concepts

Agent Harness makes AI Agents Possible

  • Start logic
  • Loop logic
  • Stop for human input logic
  • Tools & Skills (e.g. edit files, change code, generate images)

Examples: OpenClaw, Claude Cowork, Codex

Concepts

Skills, Tools, and Sub-Agents

Skills: Knowledge for a type of activity, with extra context and directions. Example: high-quality-code-comments

Tools: code on the computer that can be called with parameters. Example: query_database

Sub-Agents: collection of tools & skills with a new context window. Example: security-auditor

Getting our coding agent to work autonomously to the max

Concepts

The Ralph Loop

AI writes code → AI tests code → AI re-writes code → AI tests code ...

Ralph loop diagram

our goal: get the coding agent fully-autonomous

Concepts

Give the agent the keys to test & validate

To have it write code for us, we must:

  • Give it a good spec (PMs & Designers, Yeah!)
  • Remind it how to validate its work: Good harness & prompting

Examples:

  • make a screencast of the new features
  • Can you validate this behavior on your own, and iterate until you verify it works?

k enough theory.

LFG🚀🚀🚀🚀🚀!!!1

Setup

Initialize workspace using Terminal

mkdir -p ~/Projects/codexhero && cd ~/Projects/codexhero
cxharness
codex
code ~/.codex/config.toml

Copy this into ~/.codex/config.toml and save: 

web_search = "live"
[mcp_servers.openaiDeveloperDocs]
url = "https://developers.openai.com/mcp"

[sandbox_workspace_write]
network_access = true

[features]
unified_exec = true
multi_agent = true
apps = true

Back to terminal, paste:

cxhere -p 5173 firstfeature
Prompting

Make a React 19.2 App

In the codex prompt, write:

make a React 19.2 app with a single express server that asks my name and says hello name. show me a screencast of it working and run the server on :5173 so I can view it in my browser
OAuth

Google OAuth App

  • - Make new GCloud Project

  • - Add OAuth

  • - Configure OAuth Redirects

  • - Download Key, Make .env folder

  • - Configure Scopes

    https://www.googleapis.com/auth/gmail.modify, https://www.googleapis.com/auth/calendar.events

  • - Configure Test Users

  • - Enable Gmail & Calendar APIs

Prompting

GMail & GCal Prompt

Ask codex to connect to your email and calendar. 

Using a single express server running on :5173 and the json google client credentials in .env/, make it so I can connect my gmail and calendar with oauth redirect uri http://localhost:5173/auth/google/callback. it should ask for scopes: https://www.googleapis.com/auth/gmail.modify, https://www.googleapis.com/auth/calendar.events

The app should use a sqllite database for any storage.

Give me a way to authenticate on the homepage.
Realtime

Voice Mode App

  • - We'll use OpenAI Realtime API

  • - Make OpenAI Project

  • - Make Secret Key

  • - Copy to .env.local. Example line: 

    OPENAI_API_KEY=sk-proj-ab...AA

  • - Add $5 in credit

Prompting

Voice Mode Prompt

Setup your app to work with voice.

Use a single express server running on :5173 and the gpt realtime model to enable voice interaction on my app, using WebRTC. Support barge in.

Make tools to support:
- searching emails on gmail api
- reading out emails from gmail
- archiving emails
- reading out the day's event from google calendar api
- sending an email via the gmail api to all the invitees on a particular calendar event.

Add comprehensive evals for each kind of interaction specified. Make a well information-architected dashboard for viewing last results of these evals.

Make a screencast showing the audio chat, using codex TTS and TTS to create reasonable responses, capturing audio with PulseAudio in the environment.
Prompting

Make it recurring

In the codex prompt:

Using the single express server running on :5173, make an idempotent endpoint that makes a daily audio summary of my day to come, and summarize key emails that came in with deadlines; and it emails it to myself. Use OpenAI TTS. Call it the daily brief for <Day, Month> and phrase it like an upbeat podcaster.

Save past summary texts in the sqlite database, and have a UI that can generate the audio and play it in the browser, too. Make a screencast of that working.

Now in the non-cxhere prompt, run codex on your main computer in the project: 

codex

And then do this prompt:

commit and merge in changes from the last active git worktree into here

then, setup a script whereby every morning at 8am (up to 10am, if the computer is not on yet), it starts the server and calls the idempotent endpoint and emails it (and shuts the server back down when done)

send out a email if anything goes wrong with that endpoint, and output helpful logs if anything goes wrong with even sending an error email

this could involve using launchd depending on this computer's environment.
Closing

TYVM 🙏

If you're interested in other courses of mine, visit me at

maven.com/moorage