Frequently Asked Questions

What is a single photon camera?

Single photon cameras are an emerging class of sensors capable of detecting and counting individual photons at ultra-high speeds, reaching up to 100 kHz. You can read more about their image formation model here.

How to register?

In order to submit to the benchmark, you first need to register. Please note that in order to download the dataset, no registration is needed. To register, first create an account and confirm your email address by clicking on the link we will send to you.

What format is the dataset stored as?

The training set is structured as pairs of ground truth images and their associated photoncubes, namely <SCENE-NAME>/<FRAME-IDX>.png and <SCENE-NAME>/<FRAME-IDX>.npy. The photoncubes are encoded as width-wise bitpacked numpy arrays, with unpacked dimensions of (1024, 800, 800, 3). To read these, you can use numpy like so:

import numpy as np 

# Load with memory mapping to avoid loading everything into RAM 
photoncube = np.load("datasets/train/attic/000000.npy", mmap_mode="r")
print(photoncube.shape)  # (1024, 800, 100, 3)

# Take first 10 binary frames 
bitplanes = np.unpackbits(photoncube[:10], axis=2)
print(bitplanes.shape)  # (10, 800, 800, 3)

The test set is structured the same way, with the exception that ground truth reconstructions are not made public.

How to submit my results to the benchmark?

To submit your results to the benchmark, you need to zip them up into a single submission file. Importantly, the structure of this zip file should exactly match that of the test set. That is, the folder structure inside the zip file should be <SCENE-NAME>/<FRAME-IDX>.png without any other directories. All test samples, and nothing but the test samples, are expected and will be accepted.

A naive implementation

The simplest way to reconstruct an image from single photon (binary) frames is to simply sum them up. We provide a simple implementation of this baseline here as a reference, feel free to run this locally to inspect the generated submission file, but please do not upload naivesum results, as these have already been evaluated here.

from pathlib import Path
from zipfile import ZipFile

import numpy as np
import imageio.v3 as iio
from rich.progress import track

from visionsim.emulate.spc import spc_avg_to_rgb
from visionsim.utils.color import linearrgb_to_srgb


def naivesum(
    testset: Path,
    tonemap: bool = False,
    invert_response: bool = False,
    batch_size: int = 1024,
):
    with ZipFile("submission.zip", "w") as zipf:
        for binary_path in track(list(testset.glob("**/*.npy"))):
            with zipf.open(
                str(binary_path.with_suffix(".png").relative_to(testset)), "w"
            ) as f:
                # Unpack and average last few binary frames
                pc = np.load(binary_path)
                pc = np.unpackbits(pc[-batch_size:], axis=2)
                im = pc.sum(axis=0) / batch_size

                # Optionally invert the single photon camera's response 
                # function and tonemap the image back to sRGB  
                im = spc_avg_to_rgb(im, factor=0.5) if invert_response else im
                im = linearrgb_to_srgb(im) if tonemap else im

                # Convert to uint8 and save into zipfile
                im = (im * 255).astype(np.uint8)
                iio.imwrite(f, im, extension=".png")

Note: We show how to directly save results into a zip here, but you can of course create the archive manually after the fact as well.

Can I benchmark my method without participating in the ongoing competition?

Of course! You can always submit your results for evaluation and not participate in the competition. However, to participate in the competition, you must not only submit your results for evaluation here, but also submit additional information about your method. Competition guidelines can be found here.

Further Questions?

Please contact us here.