====== New Spectral Upsampling Methods ====== //This is the project proposal I wrote for Google Summer of Code 2020. The project can be found on their website [[https://summerofcode.withgoogle.com/projects/#6675798750134272|here]].//
New Spectral Upsampling Methods
for Colour Science for Python, participating under NumFOCUS
===== Personal information ===== **Name:** Paweł Ryszard Redman \\ **Country of residence:** Poland \\ **University:** Wrocław University of Science and Technology (Polish: Politechnika Wrocławska) \\ (Contact details redacted) I am a first year student pursuing a Master’s degree in Optics with a specialization in Optical and Photonic Engineering at the Wrocław University of Science and Technology. I received my Engineer’s degree in February 2020 at the same university, in the same field and specialization. If selected, I will be willing and capable of committing to full-time work on this project during the summer. I am aware of the potential that the summer holidays start later this year due to the ongoing situation, but I have been keeping up with my studies despite classes being cancelled and I am confident that any such extraordinary situation would not affect my work. ===== Abstract ===== Traditional rendering is based on the idea that light can be decomposed into three basic components: red, green and blue (RGB). In some areas this is a vast oversimplification of the physical reality; light exists on a continuous spectrum of wavelengths. Spectral rendering achieves better accuracy by taking that into account. The major drawback is that it cannot use widely available color data for surfaces such as photographs. The process of gathering usable spectral data is significantly more time- and resource-intensive. Spectral upsampling is the process of extrapolating spectral information from RGB color data. This is possible because, in practice, spectral reflectivities, which describe how a surface interacts with incident light on a wavelength-by-wavelength basis, tend to fit relatively simple patterns. The goal would be to add support for several newly published algorithms to [[https://github.com/colour-science/colour|Colour]], a color science library for Python. With the rising popularity of spectral rendering, the library would greatly benefit from the work. ===== Technical Details ===== Colour is a Python library, targeting both Python 2.7+ and Python 3.5+. Like many scientific projects, it relies on [[https://www.scipy.org/|SciPy]] for array operations, numerical algorithms (most importantly function minimization) and plotting. There is a number other dependencies, but they would not be relevant to this project in particular. There is also an extensive list of [[https://www.colour-science.org/contributing/|rules and guidelines]] for contributors regarding code style, documentation, unit testing and so on that I will have to follow in order to have my work accepted upstream. **How does spectral upsampling work?** Colour already has a submodule called colour.recovery, which deals with reflectance recovery. Currently two methods are implemented, [[https://doi.org/10.1080/10867651.1999.10487511|Smith (1999)]] and [[https://doi.org/10.1111/cgf.12676|Meng et al. (2015)]]. Because I can use the existing code and tests as reference, integrating new code into the codebase will be relatively simple. The priority would be to extend the submodule by adding support for [[http://lightmetrica.org/h-otsu/project/rgb2spec/|Otsu et al. (2018)]] and [[http://rgl.epfl.ch/publications/Jakob2019Spectral|Jakob and Hanika (2019)]], with implementations of [[https://geometrian.com/data/research/spectral-primaries/EGSR2019_SpectralPrimaryDecomposition.pdf|Mallett and Yuksel (2019)]] and [[https://doi.org/10.1145/3306346.3322964|Peters et al. (2019)]] being secondary goals. A tertiary goal that I thought up myself and is not a part of the original [[https://github.com/colour-science/GSoC/blob/master/2020/GSoC-2020-Project-Ideas.md#new-spectral-upsampling-methods|project proposal]], but was deemed a good idea by one of the mentors, is implementing the interpolation scheme described in Meng et al.’s article, which currently is missing from Colour. Given spectral distribution of the illumination, the process of converting the spectral reflectance of a surface to tristimulus values, which quantify the color perception of the surface, is straightforward. If the reflectance is represented as a vector with a suitable N-dimensional basis in the function space, the conversion is done by a simple linear map φ : ℝᴺ→ℝ³. Unless N = 3, which is rarely enough in real world scenarios, this mapping is non-invertible.((The original text contains an error and reads "invertible.")) In other words, for each color, there are infinitely many reflectances, which are then called metamers. Worse still, two reflectances that produce exactly the same color under one illuminant can look very different under another. Spectral upsampling solves this underdetermined problem by imposing additional constraints. The specifics vary from method to method, but, as an example, several articles propose formulating this as a multivariate numerical optimization problem. A solver, such as SciPy’s SLSQP or [[http://ceres-solver.org/|Ceres]], is tasked with finding a spectrum that matches the desired color as closely as possible, has a desired shape, varies smoothly between nearby colors, minimizes unwanted color shifts when changing illuminants and so on, all at the same time. This is usually very computationally expensive. All methods listed in this proposal also rely on performing all the heavy work ahead of time to achieve good performance in a spectral renderer. The most important part of an implementation is writing code that can exactly recreate published precomputed tables. Unit tests can be used to automatically test the validity and accuracy of the results. This is by far the most challenging task, as the methods can be rather contrived. The rest of the work is implementing interpolation, which is relatively simple, documenting the code and integrating it well into the existing codebase. ==== Schedule of Deliverables ==== === Community Bonding Period === To get to know the community and the codebase, I will use this time to try and get a minor pull request merged in. Implementing the missing part of Meng, et. al. is one possibility, but I will have to discuss this topic in further detail. In the meantime I will set up a blog and start posting there, as per NumFOCUS’s requirements. === Phase 1 === Implement [[http://rgl.epfl.ch/publications/Jakob2019Spectral|Jakob and Hanika (2019)]]: - Write code that minimizes the authors’ cost function at a single point. - Extend the solver to cover the entire sRGB gamut without discontinuities. Explore libraries other than SciPy if necessary (the authors used Ceres). - Create coefficient maps as described in the article matching the published values. - Write code that can sample the maps and compute the final spectral data. - Extend the method to other color spaces. The optimistic time frame for completion of all the points above is 2 weeks. Looking at the article, the method seems straightforward enough to do this. Each sub-goal, excluding the third, should take a day or two at most. I know from experience that trying to exactly match published results tends to be a tedious process and I expect to spend most of the time doing that. Once the code is finished, it will have to go through review. I assume that a week should be enough to get the code polished enough to be worth merging in. Assuming no unexpected delays or problems, I should be left with a bit over a week before the first evaluation to work on the next goal. === Phase 2 === Implement [[http://lightmetrica.org/h-otsu/project/rgb2spec/|Otsu et al. (2018)]]. - Implement the principal component analysis (PCA) the authors used. - Implement their clustering algorithm. - Make sure the results match the published values. The time frame here is harder to estimate. The method looks considerably more complicated than the previous. The article is also slightly vague (especially about PCA) and analysis of the included source code could take a while. I have to be conservative and assume I will need the entire four weeks or so to get this done, including the review (which I expect to take less time than the first one). === Final Week === The last few weeks will be the time to consider other spectral upsampling methods, discussing the topic with the mentors and the community. The specifics will depend on how fast I complete the goals of the first two phases and what is decided on. The hard goal would be to fully implement at least one more method. For example, Mallet and Yuksel’s is similar to Otsu, et. al.’s and I would not expect to have problems adding support for it before the final week. By the final week, I expect to have created several good and reviewed commits, worthy of being merged into Colour. The final week will be the time to make sure everything is ready for the final evaluation. ===== Development Experience ===== I have about a decade of experience as an amateur C programmer. I published my first open source project, a modification of the first-person shooter Tremulous, on GitHub back in 2013. Since then I’ve worked on many projects, including some minor paid work as a freelancer. I showcase the more interesting ones on my webpage. I started using Python a few years ago as a replacement for MATLAB, but I’ve also embraced is as a general-purpose language, with it pretty much replacing C as my language of choice in most cases. While I could use experience working with large Python codebases, which is partly why I am applying for this project, I think I am more than qualified as far as programming skills go. ===== Why this project? ===== I’ve been interested in computational physics for a number of years now, so to me picking a NumFOCUS project was a natural choice. Expecting to find an interesting numerical problem to tackle in e.g. SciPy, I was surprised to find out that Colour Science was participating. As a student of Optics, having written his Engineer’s thesis on the topic of color rendering, I thought it was the perfect project to contribute to for someone with my background. Specifically, I have been considering spectral rendering as a potential topic for my upcoming Master’s thesis, so taking on this project would provide me a lot of valuable experience.