proseminar/handout/handout.tex

%\newcommand{\CLASSINPUTtoptextmargin}{1.5cm}
\newcommand{\CLASSINPUTbottomtextmargin}{2.5cm}
\documentclass[conference, a4paper, hidelinks]{IEEEtran}

% BEGIN: DO NOT CHANGE HERE!
    \pagenumbering{gobble}
    \makeatletter
    \def\@maketitle{%
    \newpage
    \null{}
    \begin{center}%
        {\LARGE \textsf \@title{} \par} % Adjust the font size and style of the title
        \vskip 0.5em % Adjust the vertical spacing after the title
        {\small \textsf{Proseminar Differentiable Programming} \par}
        \vskip 0.5em % Adjust the vertical spacing after the subtitle
        {\small \textsf \@author{} \par }
        \vskip 0.5em % Adjust the vertical spacing after the author
        {\small \textsf \@date{} \par} % Adjust the font size and style of the date
    \end{center}%
    \par
    \vskip -1em % Adjust the vertical spacing after the title block
    }
    \makeatother
% END: DO NOT CHANGE HERE!

\title{Differentiable Monte Carlo Ray Tracing through Edge Sampling}
\author{Clemens Dautermann}
\date{\today} % Put the date of your talk here

% Bibliography setup
\usepackage[citestyle=numeric,bibstyle=numeric,hyperref,backend=biber]{biblatex}
\addbibresource{bibliography.bib}

\usepackage[english]{babel}
\usepackage{csquotes}
\usepackage{amsmath}

\usepackage{tikz}
\usetikzlibrary{calc,patterns,angles,quotes,shapes,arrows, positioning,overlay-beamer-styles}

\usepackage{subfig}

\usepackage{xcolor}
\usepackage{graphicx}

\usepackage{hyperref}
% Do not add anything here
\begin{document}

\maketitle

\begin{abstract}
    Differentiable Programming is a technique frequently used to solve optimization problems by minimizing some kind of error function. To do this though the error function needs to be differentiable with respect to the parameters that are to be optimized. This is usually not the case with ray tracing. This report will explain why this problem occurs and present the method to tackle it developed in~\cite{ACM:diffable_raytracing}.
\end{abstract}

\section{Introduction}

One of the main tasks in Computer Graphics is image synthesis. This means ``given a 3D scene, output an image depicting the scene``. Often it is required for the image to be as realistic as possible, meaning as close to a picture of the scene as if it was set in the real world as possible. This is most commonly achieved using the ray tracing algorithm and is a well studied problem. Doing this in a differentiable way however is much less trivial. This stems from the fact that the rendering integral (equation~\ref{eq:rendering_integral}) is not differentiable in certain well defined places. Because being able to ray trace an image differentiably has numerous applications a solution to this problem has been proposed in~\cite{ACM:diffable_raytracing}. This report will go into how exactly this problem arises, how it can be mitigated and what some of the applications of differentiable ray tracing are.

\section{Ray Tracing}
To formalize the problem of photo realistic image synthesis, an equation has been proposed by Kajiya in 1986~\cite{ACM:rendering_equation}. This equation captures physical light transport for a scene and if solved yields the color for a given point in the scene accounting for most physical light transport phenomena.
\begin{figure}[h]
    \centering
    \begin{equation}
        I(x,x^{\prime})=
        g(x,x')
        \left[
        \epsilon(x,x')+\int_S \rho(x,x',x'')I(x',x'')dx''
        \right]
        \notag
    \label{eq:rendering_integral}
    \end{equation}
    \caption{The rendering equation capturing physical light transport. It assigns  a value to the the Intensity of light transported from a point $x$ to a point $x^\prime$. The geometry term $g$ will be discussed later. The term $\epsilon$ accounts for the emissivity of the material at point $x$. The integral term represents all light scattered from any other point in the scene towards the point $x$. The integral domain $S$ contains all points in the scene.}
\end{figure}

This equation (equation~\ref{eq:rendering_integral}) is now widely recognized as ``the rendering integral``. It can not be solved analytically and is thus most commonly approximated using Monte-Carlo integration~-~i.e.\ ray tracing. Ray tracing works by backtracking light rays from the light sources in the scene and thus simulate physically realistic lighting.\\
To do this, rays are cast from the camera, through each pixel in the camera frustum. The intersection point with the scene geometry $x$ is calculated for each ray and material properties (e.g.\ color, emissivity etc.) are taken into account to calculate the pixel color. From this point more rays are drawn towards each light source. If the light source is visible, its light contributes to the pixel color as well. To account for indirect lighting the ray ``bounces around', yielding a color for some of the points scattering lights towards $x$. This approximates the integral term in equation~\ref{eq:rendering_integral}.\\
Differentiable ray tracing is the task of calculating the gradient of this process with respect to \emph{any} scene parameter.

\section{Problems with differentiability}
The geometry term $g(x,x^\prime)$ in equation~\ref{eq:rendering_integral} is the main problem when it comes to differentiating the rendering integral. This term is 1 iff $x$ is visible from $x^\prime$ and 0 otherwise.
\begin{figure}[h]
    \centering
    \include{diagrams/diagramm_occlusion}
    \caption{A simple occlusion scenario. An infinitesimal change in ray angle $d\omega$ is sufficient to determine whether the wall is visible or not.}\label{fig:occlusion_rays}
\end{figure}

\begin{figure}
    \centering
    \input{diagrams/geometry_term_diff.tex}
    \caption{The visibility of a point and the differentiation}\label{fig:visibility_term}
\end{figure}

As illustrated in figure~\ref{fig:occlusion_rays} an infinitesimal angle change $d\omega$ can lead to the blocker obstructing the wall or the wall being visible. The geometry term is thus a Heaviside step function which when differentiated yields a Dirac delta functional (see figure~\ref{fig:visibility_term}). Since the Dirac delta functional only differs from 0 in $\omega_0$ one point, the probability of sampling it when using uniformly distributed Monte-Carlo integration is 0.

\section{Proposed solution: Edge sampling}
The key observation is that these discontinuities only occur at the edges of meshes.
Thus the problem can be mitigated by not sampling a pixel uniformly, but importance sampling the edges of a mesh, hence capturing the Dirac delta spikes.

\begin{figure}[h]
    \subfloat[area sampling]{
        \input{diagrams/area_sampling.tex}
    }
    \hfil
    \subfloat[edge sampling]{
        \input{diagrams/edge_sampling.tex}
    }
    \caption{The figure depicts the sampling situation in a single pixel. We are interested in the gradient with respect to the gray triangle moving up. Area sampling does not account for this color change whereas edge sampling does. Primary occlusion is handled correctly.}\label{fig:area_vs_edge_sampling}
\end{figure}

Consider a situation as depicted in figure~\ref{fig:area_vs_edge_sampling}. When the gray triangle moves up, the white area decreases and the gray area increases. Thus white should contribute less to the pixel color. Area sampling does not account for this change, since all samples land on an area with the same color before and after movement. Edge sampling does account for this change because especially areas of discontinuity i.e.\ the edges are sampled. The samples will have a different colors after the triangle moves up, correctly capturing the gradient. Primary occlusion is handled correctly because the sample that intersected the blocker before, will also intersect the blocker after movement as it is independent of the triangle movement.

\subsection{Half spaces}

To understand why edge sampling correctly captures the gradient an explanation of what exactly happens at the point where a ray intersects an edge is required.

\begin{figure}[h]
    \centering
    \input{diagrams/halfspaces.tex}
    \caption{An edge separates the domain into two regions, whose contribution can be considered separately. The contribution of the upper half space increases while the contribution of the lower half space decreases with respect to the gray triangle moving right. Occlusion is handled correctly because the contribution of either half space is equal.}\label{fig:halfspaces}
\end{figure}

As seen in figure~\ref{fig:halfspaces} the edge separates the space into two half spaces. These two halfspaces can now be considered separately and a gradient can be calculated from the color difference. It is also evident that occlusion is handled correctly. This arises from the fact that where occlusion occurs, the half spaces' contribution to color is equal. From that it follows that the gradient will be 0.\\
Mathematically this yields that the Integral can be separated into two parts. One part that represents the discontinuous edges and one part that accounts for the original pixel integral over continuous regions.

\section{Results}


% BIBLIOGRAPHY
\nocite{*} % List all entries of the .bib file, even those not cited in the main body
\printbibliography{}


\end{document}