CS 184 Final Project: GenshinNeRF

Abstract

In this project, we started with two objectives in mind: To inject custom shaders into Genshin Impact, and also to attach NeRF onto a game engine so that future applications can leverage this neural representation in use with alternate rendering methods, mesh generation, and scene reconstruction. For the first part of our project, in terms of generating custom shaders, we've decided to do a deep dive exploration on non-photorealistic shading methods, including three versions of Pastel Shading, Manga-Style Shading, and Cel Shading. As for the second part, Game Engines and NeRF have often been somewhat separated by their purpose. Afterall, who wants to wait to query some large model for rendering that could potentially take minutes? In this project, we've decided to create this unlikely fusion by dedicating NeRF to one particular usecase for a Game Engine, to potentially enable future feature sets that allows texture generation via Large Language Models, accurate scene reconstruction, and integration of visual large language models into the game-dev process. To enable these features we've decided to create a link between the game engine and NeRF studio synchronized in real-time. The game engine is secondary to this purpose, however, since the functionality is mainly based upon the application ability to create poses, so any game with a proper .dll injection to capture camera poses will work with our implementation.

Custom Shaders

Using a modified verson of HLSL (which is a modified version of the GLSL used in Project 4 of this class), we were able to create various non-photorealistic shading methods such as Pastel Shading, Manga-Style Shading, and Cel Shading.

Live Streaming to NeRF

Similar to LERF, we made a new method for nerf studiio which extended the nerfstudio codebase to allow live streaming on a TCP connection. Specifically, we altered the PyTorch code to allow for a TCP connection to be established, and then we created a C++ DLL that would be injected into the game engine to send the camera poses to the nerf studio.

Results

Upon completing the different shaders, the game runs very smoothly and with different visuals.

Video Demo of Different Shading

Final Presentation Video

Normal Shading

Genshin Impact without any custom shading

Square Kuwahara Shading

Genshin Impact with Square Kuwahara shading

Circular Kuwahara Shading

Genshin Impact with Circular Kuwahara shading

Anisotropic Kuwahara Shading

Genshin Impact with Anisotropic Kuwahara shading

Pencil Shading

Genshin Impact with Noir Pencil shading

Cel Shading

Genshin Impact with Cel shading

References

Shaders

  • Genshin-Impact-ReShade
  • Kyprianidis, J. E., Kang, H., & Döllner, J. (2010). Anisotropic Kuwahara Filtering on the GPU. In W. Engel (Ed.), GPU Pro - Advanced Rendering Techniques. AK Peters, pp. 247–264.

NeRF

Contributions

Levy

  • Worked out on injecting shaders into Genshin Impact
  • Created the oil pastel Shaders, the The cel/toon shader, and the pencil shader from modifying the Noir Shader
  • Created the design for integrating Godot with NeRF-studio
  • Implemented the Godot facing TCP connectivity interface with nerfstudio backend
  • Implemented the StreamDataset(IterableDataset) to query TCP stream and retrieve data
  • Modified the StreamDataManager to synchronize with the StreamDataset and Dataloader
  • Implemented the StreamDataloader to interact with the StreamDataset
  • Created the website, videos, slides, administrivia

Jason

  • Created the website layout, structure, and styling and wrote the shaders technical approach and main page
  • Helped implement the circular generalized kuwahara filter
  • Created the BlackWhite Noir Shader
  • Conducted researched into interesting different shader effects
  • Created the presentation video

Rahul

  • Created module/skeleton code for the Godot NeRF-integration that builds on top of the NeRF-studio framework
  • Modified StreamDataparser and StreamPipeline to use inheritance for easier implementation
  • Worked on the webiste
  • Wrote NeRF integration with Godot Engine in PyTorch/worked on this site.

Pierre

  • Did initial research into running NeRF studio
  • Generated NeRF outputs for the milestone
  • Researched data conventions for NeRF.