GPU-based Photomosaic

This demo showcases the creation of a GPU-accelerated photomosaic using p5.js, GLSL shaders, and the quadrille library. A photomosaic is composed of smaller images (tiles), each representing a portion of the overall picture. The quadrille sort method arranges tiles by luminance for accurate visual matching, while framebuffers store the sorted palette, enabling efficient GPU sampling and rendering with WebGL.

⚠️

The WEBGL mode in p5.js relies on GPU acceleration, which leads to higher energy consumption and a larger carbon footprint.

(press r to randomly pick another painting)

Strategy

The photomosaic construction relies on creating a palette from a collection of images. The quadrille sort method arranges the tiles by their lightness metric, typically LUMA, in ascending order. A pixelator downsamples the target image (texture2D(source, stepCoord)), and the resulting texel serves as a lookup key to sample the palette in the fragment shader.

  flowchart LR
    A[/mosaic shader/] --> E[texture mapping]
    B[create palette] --> E
    C[/paintings/] --> B
    D[/"source (target image or video)"/] --> E

Palette Creation

1. Loading Images

A collection of paintings is loaded into an array named paintings. These images form the tiles for the mosaic.

let paintings = [];
const numPaintings = 30;

function preload() {
  for (let i = 1; i <= numPaintings; i++) {
    paintings.push(loadImage(`p${i}.jpg`));
  }
}

2. Creating and Sorting the Palette

The createQuadrille function generates a palette from the paintings array. The sort method arranges tiles by their average LUMA, enabling a seamless gradient from darkest to lightest. Alternate lightness metrics can also be applied.

const SAMPLE_RES = 90;
let palette;

function setup() {
  // Initialize and sort the palette
  palette = createQuadrille(paintings);
  palette.sort({ ascending: true, cellLength: SAMPLE_RES });
  // Rendering the palette to a Framebuffer follows here
}

3. Rendering the Palette to a Framebuffer

The sorted palette is rendered onto a p5.Framebuffer. This framebuffer (fbo) becomes a uniform sampler2D palette in the shader, allowing fast GPU sampling during mosaic rendering.

let fbo;

function setup() {
  // palette creation omitted for clarity
  // Create framebuffer for palette storage
  fbo = createFramebuffer({
    width: SAMPLE_RES * palette.width,
    height: SAMPLE_RES,
    format: FLOAT
  });
  // Render the sorted palette onto the framebuffer
  fbo.begin();
  drawQuadrille(palette, {
    cellLength: SAMPLE_RES,
    outlineWeight: 0,
    x: -SAMPLE_RES * palette.width / 2,
    y: -SAMPLE_RES / 2
  });
  fbo.end();
}

The resulting fbo appears as follows:

This framebuffer acts as the palette reference for the shader, ensuring efficient tile sampling during the photomosaic generation.

Interaction

Random Image Selection: Press r to select a random painting from the palette as the source image.

function keyPressed() {
  if (key === 'r' && !video_on.checked()) {
    blender = random(paintings);
  }
}

Video Mode: A checkbox toggles between a static image and a video source. If video mode is active, the shader processes the live feed to generate a dynamic photomosaic.

video_on = createCheckbox('video', false);
video_on.changed(() => {
    blender = video_on.checked() ? video_src : random(paintings);
    video_on.checked() ? video_src.loop() : video_src.pause();
  });

References

Quadrille API

p5 API

createCanvas — Creates a drawing canvas on which all the 2D and 3D visuals are rendered in p5.js.
background — Sets the color used for the background of the canvas.
createFramebuffer — Creates a framebuffer object for off-screen rendering, allowing intermediate drawing results to be stored and efficiently sampled in GPU-based workflows.
shader — Sets the current shader for subsequent drawing operations, enabling GPU-accelerated rendering using custom vertex and fragment shaders.
setUniform — Updates the value of a uniform variable in the shader, allowing dynamic control over shader parameters during runtime.