Image support

Cloud Vision Capabilities and Image Handling

Core Concepts:

• Functionality: Cloud allows users to include images in messages, enabling capabilities such as content identification, object counting, and image comparison.
• Image Block: Images are sent using an "image block" within a user message.
• Image Sources: An image block can contain either:
• Raw image data.
• A URL pointing to an externally hosted image.

Technical Limitations & Costs:

• Request Limit: A single request can include up to 100 images.
• Constraints: There are limitations regarding the height and width of individual images.
• Token Cost: Sending an image consumes tokens, and the charge is calculated based on the image's height and width in pixels.

Prompt Engineering for Images

Key Principle:

• Simple prompts often lead to inaccurate results when using images. Strong prompting techniques are essential for high accuracy.

Effective Prompting Techniques:

• Providing Analysis Steps: Guide Cloud through a multi-step process (e. g. , 1. Identify objects, 2. Count objects, 3. Verify the count, 4. Provide final result).
• One-Shot/Multi-Shot Prompting: Improve accuracy by alternating image parts and text parts in the message (e. g. , [Image 1] → "This image has X objects" → [Image 2] → "What is in this image? ").

Use Case Example: Fire Risk Assessment

• Goal: Automate fire risk evaluation using satellite imagery.
• Process: The prompt must be highly detailed, instructing Cloud to perform specific analyses:
• Find the primary residence.
• Analyze tree density.
• Assess fire service access paths.
• Check for overhanging branches.
• Assign a final fire risk rating (e. g. , 1 to 4).

Implementation Note:

• Images are typically converted into base 64 encoding before being included in the message structure.

Takeaways

• Cloud Vision uses an "image block" to enable capabilities like content identification and object counting, accepting either raw image data or external URLs.
• Strong prompting techniques are essential for accuracy, requiring guidance through multi-step processes or alternating image and text parts (one-shot/multi-shot prompting).
• Image processing consumes tokens, and the associated cost is calculated based on the image's height and width in pixels.
• Images are typically converted into base 64 encoding before being included in the message structure.