Documentation Index Fetch the complete documentation index at: https://mintlify.com/screenpipe/screenpipe/llms.txt
Use this file to discover all available pages before exploring further.
Frames Frames are individual screenshots captured by Screenpipe. The frames endpoints allow you to retrieve frame images, OCR data, metadata, and navigate through frames.
Get Frame Image Retrieve the JPEG image for a specific frame.
Endpoint Path Parameters ID of the frame to retrieve
Query Parameters If true, blur/redact any detected PII (credit cards, SSNs, emails) in the frame.
Example curl "http://localhost:3030/frames/12345" --output frame.jpg
With PII Redaction curl "http://localhost:3030/frames/12345?redact_pii=true" --output frame_redacted.jpg
Response Returns the frame image as JPEG binary data.
Content-Type : image/jpegError Responses Frame not found in database
Request timed out while retrieving frame
Frame unavailable - video file corrupted or missing
Get Frame OCR Data Retrieve OCR text positions with bounding boxes for a frame. Useful for creating text selection overlays on screenshots.
Endpoint GET /frames/{frame_id}/ocr
Path Parameters Example curl "http://localhost:3030/frames/12345/ocr"
Response Array of text positions with bounding boxes Show Text Position Object
X coordinate of top-left corner
Y coordinate of top-left corner
Example Response { "frame_id" : 12345 , "text_positions" : [ { "text" : "Hello World" , "x" : 100.5 , "y" : 50.2 , "width" : 120.0 , "height" : 20.0 }, { "text" : "Welcome to Screenpipe" , "x" : 100.5 , "y" : 80.3 , "width" : 180.0 , "height" : 20.0 } ] }
Get frame metadata including timestamp for deep link navigation.
Endpoint GET /frames/{frame_id}/metadata
Path Parameters Example curl "http://localhost:3030/frames/12345/metadata"
Response When the frame was captured
Example Response { "frame_id" : 12345 , "timestamp" : "2024-03-08T14:30:00Z" }
Error Responses Find Next Valid Frame Find the next frame with a valid video file on disk. Allows the frontend to skip directly to a valid frame instead of trying each one when frames fail to load.
Endpoint Query Parameters Current frame_id that failed to load
Search direction: forward or backward
Maximum number of frames to check (default: 50)
Example curl "http://localhost:3030/frames/next-valid?frame_id=12345&direction=forward&limit=50"
Response ID of the next valid frame found
Timestamp of the valid frame
Number of frames skipped to find this valid frame
Example Response { "frame_id" : 12350 , "timestamp" : "2024-03-08T14:31:00Z" , "skipped_count" : 5 }
Error Responses No valid frames found within the search limit
Use Cases Display Frame with Text Overlay
Retrieve the frame image
Get OCR data with bounding boxes
Render image with selectable text overlay
# Get frame image curl "http://localhost:3030/frames/12345" --output frame.jpg # Get OCR positions curl "http://localhost:3030/frames/12345/ocr"
Timeline Navigation
Get frame metadata for timestamp
Use timestamp for timeline scrubbing
If frame fails to load, find next valid frame
# Get metadata curl "http://localhost:3030/frames/12345/metadata" # If frame fails, find next curl "http://localhost:3030/frames/next-valid?frame_id=12345&direction=forward"
Privacy-Aware Display Automatically redact sensitive information:
curl "http://localhost:3030/frames/12345?redact_pii=true" --output safe_frame.jpg
Frame Management Tips Performance Best Practices :
Cache frame images on the client side
Use thumbnail sizes when displaying multiple frames
Request OCR data separately only when needed
Use next-valid endpoint for robust frame navigation
Enable PII redaction when displaying frames to users
Frame Availability : Frames may become unavailable if:
Video file has been deleted or moved
Storage corruption occurred
Frame was recorded but video encoding failed
Always handle 410 (Gone) responses gracefully. Integration Example TypeScript/React import { useState , useEffect } from 'react' ; function FrameViewer ({ frameId } : { frameId : number }) { const [ imageUrl , setImageUrl ] = useState < string >( '' ); const [ ocrData , setOcrData ] = useState < any >( null ); const [ error , setError ] = useState < string >( '' ); useEffect (() => { const loadFrame = async () => { try { // Load frame image const imageResponse = await fetch ( `http://localhost:3030/frames/ ${ frameId } ?redact_pii=true` ); if ( ! imageResponse . ok ) { if ( imageResponse . status === 410 ) { // Frame unavailable, find next valid one const nextResponse = await fetch ( `http://localhost:3030/frames/next-valid?frame_id= ${ frameId } &direction=forward` ); const nextData = await nextResponse . json (); // Retry with new frame_id return loadFrame ( nextData . frame_id ); } throw new Error ( 'Failed to load frame' ); } const blob = await imageResponse . blob (); setImageUrl ( URL . createObjectURL ( blob )); // Load OCR data const ocrResponse = await fetch ( `http://localhost:3030/frames/ ${ frameId } /ocr` ); const ocr = await ocrResponse . json (); setOcrData ( ocr ); } catch ( err ) { setError ( err . message ); } }; loadFrame (); }, [ frameId ]); if ( error ) return < div > Error : { error } </ div > ; if ( ! imageUrl ) return < div > Loading ...</ div > ; return ( < div className = "frame-viewer" > < img src = { imageUrl } alt = "Frame" /> { ocrData && ( < div className = "ocr-overlay" > { ocrData . text_positions . map (( pos , i ) => ( < div key = { i } className = "text-box" style = {{ position : 'absolute' , left : pos . x , top : pos . y , width : pos . width , height : pos . height , }} > { pos . text } </ div > ))} </ div > )} </ div > ); }
Next Steps