Inside the Pipeline: How a TikTok or YouTube URL Becomes a 7-Day Trip in Jettova

The simplest demo of 'AI plans your trip from a TikTok' uses the cover frame and the caption. It looks great on launch. Then someone pastes a video of ziplining over a Costa Rican jungle, captioned 'endless summer vibes,' and the AI returns a beach vacation in Aruba because the algorithm-optimized thumbnail showed a beach and the caption named the wrong vibe. The first failure mode of cover-frame-plus-caption is structural: TikTok's recommendation system optimizes the thumbnail for click-through, not destination accuracy, and captions describe feelings rather than activities.

This is the technical writeup of how Jettova's video-to-trip pipeline actually works — stage by stage, with the trade-offs we considered and the ones we didn't.

Stage zero: URL detection. The route accepts both TikTok and YouTube URLs because the user-facing affordance is 'paste a video' rather than 'pick a platform.' Two regexes test the URL on the way in; rejections happen before any network call so we don't spend money on the wrong shape of input. Instagram Reels are technically possible but operationally painful — Instagram aggressively blocks scrapers, and the reliable paths are paid services we'd rather wait to wire until demand justifies the cost.

Stage one: metadata fetch. Platform-specific. For TikTok we use TikWM, a public unauthenticated API that returns the no-watermark mp4 URL, cover image, caption, author handle, and duration. It's free, which is great for a prototype and bad for a load-bearing dependency — we wrapped the call in a three-try retry with 500ms / 1s / 2s exponential backoff so transient blips resolve on the second attempt. The failure mode we'd hate but don't have yet is sustained TikWM rate-limiting against Vercel's IPs; the fix queued behind that day is swapping to RapidAPI's TikTok endpoint, which is the same shape API for $0.001 per call.

For YouTube we use @distube/ytdl-core, an actively-maintained fork of ytdl-core that handles Shorts, regular videos, and the youtu.be short-URL form uniformly. It returns the same shape — a play URL, a thumbnail, a title, a channel handle, a duration — which makes the rest of the pipeline platform-agnostic. We use ytdl.chooseFormat with quality 'lowest' and filter 'videoandaudio' to pull the smallest combined stream available; we only sample 5 frames from it so HD bandwidth is wasted spend.

Stage two: duration cap. Long videos balloon the mp4 download plus ffmpeg work past our 60-second function budget on the server, and trip-inspiration content in practice fits inside a Short, a Reel, or a clip from a vlog rather than a 45-minute documentary. We hard-cap at 10 minutes and return a friendly error message that names the actual length when a longer video comes in. The cap doesn't really limit which videos people import — almost everything users paste is under 90 seconds — but it bounds the worst-case server cost.

Stage three: frame extraction. The mp4 is downloaded to /tmp on the serverless function (Vercel Functions explicitly support writes to /tmp despite the static-analysis tooling that occasionally flags it). ffmpeg runs with a single command: an fps filter set to count/duration (which evenly distributes the sample across the runtime), a scale filter to 512px wide (vision cost is proportional to image area, so wider is wasted), and -frames:v capped at the target count. For a 30-second video we get five frames at t=0, 6, 12, 18, 24. For a 5-second video, we get five frames at t=0, 1, 2, 3, 4. Edge case: very short videos with fps > 1 produce fewer frames than requested — ffmpeg handles this gracefully and we send whatever came back.

Why 5 frames specifically. Three is too few — a video of someone moving through a market and then sitting at a sushi counter needs at least one frame per setting. Seven is wasteful — vision tokens are billed per image area and the marginal information from the 6th and 7th frame is small. Five turns out to capture distinct scenes in nearly every travel video we tested while keeping the multi-image call cost around $0.005 with Claude Haiku 4.5.

Stage four: vision call. All five frames go to Claude Haiku as base64-encoded image blocks in a single messages.create call, alongside a text block with the caption, the author handle, and an explicit instruction set: look at each frame individually before answering; build the activities list from the union of what you see; when the caption and frames disagree, trust the frames; pick a real-feeling destination as 'City, Country.' The output is constrained by a tool_use schema that requires destination, vibe (one-line sentence), activities (2-5 strings), vibes from a canonical 12-vibe list, duration_days, confidence (high/medium/low), reasoning (one sentence), and a 7-day itinerary in the same shape as the normal Jettova trip itinerary.

The itinerary is generated in the same call rather than as a follow-up. We tried it as a follow-up (destination first, then itinerary in a second call) and the failure mode was a noticeable disconnect — the second call couldn't access the frame context as cheaply, and the itinerary stopped referencing what was actually in the video. Generating both in one pass keeps the model grounded on the same input and costs about the same as a separate call because the input frames dominate the token cost.

Schema-level rules in the itinerary tool definition: days 1-3 must reference specific things visible in the frames; days 4-7 round out a believable trip in the destination; every slot title must be a real-feeling named place (not 'a local café'); no title can repeat across days. These constraints sound aggressive but the model honors them at the Haiku 4.5 quality bar — under-constrained itineraries from the same model returned 'Day 3 lunch: local restaurant' which is the kind of generic that signals to users we didn't actually look at the video.

Stage five: response payload. The route echoes the frames back to the client as data: URLs in the response so the page can render them as a photo strip. This adds about 250 KB to the response — acceptable for a one-shot page render, not a hot path. The booking links (Expedia flights/cars, Booking hotels, Airalo eSIM, Viator activities) are built server-side from the destination plus a default date window (today+30 days, lasting durationDays). Outbound search URLs only; we don't fake live inventory because the affiliate behavior at click time is what matters, not in-line pricing.

Rate limiting. Three imports per hour for non-admins, enforced via Upstash sliding-window rate-limit. Identifier is user_id from the bearer token if signed in, IP if anonymous — so a single user across devices shares the same quota and an anonymous abuser can't multiply by switching browsers from the same connection. Each import costs about $0.005 in vision tokens plus a few cents of bandwidth, so 3/hour caps the unauthorized-abuse case at a few dollars a day before we'd notice, while staying tight enough not to throttle legitimate use ('I have five candidate videos for next month's trip').

Cold-start latency. Most of the wall time is the multi-image vision call (3-6 seconds for Haiku 4.5 with 5 images). Frame extraction is 1-2 seconds for a sub-minute video on the Vercel function. Metadata fetch is sub-second when the platform's downloader is healthy. Total typical latency is 4-10 seconds end-to-end; the rendering on the client adds maybe another second for the photo-strip layout. Most of the variability lives in the vision call; longer videos with more durational variety also tend to produce longer itinerary output, which slightly extends the call.

What we don't do that we could. We don't transcribe the audio — most TikTok travel content has either no narration or generic music. Transcribing would add another model call (Whisper or equivalent) and the benefit on Travel-genre content is small enough that we haven't pulled the trigger. We don't yet expose 'edit the itinerary' on the result page — for now the result is the read-only output, and editing routes back through the normal Plan flow with the destination prefilled. The two surfaces share the same itinerary schema, so unification is a small UI project and a planned upgrade.

Why the keyframes approach matters beyond accuracy. A user pasting a video has a single signal-strength expectation: the result page should prove the model actually watched the video, not just read the caption. The photo strip in the result (the cover plus the five extracted keyframes, in a 6-up grid) is the audit trail that makes the rest of the page credible. Without it, every wrong destination feels like the AI hallucinated. With it, you can see exactly what the model saw, and if the destination is wrong you can usually tell why from the strip. That moves the entire feature from 'magical and untrustworthy' to 'transparent and useful.'

If you want to try it: paste any public TikTok URL or any YouTube URL under 10 minutes into /plan-from-tiktok. The five-section result page (hero, photo strip, vibes, booking row, 7-day itinerary) renders in 4-10 seconds.