[REQUEST] Offscreen rendering inquiry

[Kaned1as]

Please confirm the following points:

• This question is NOT about the Android apps in the Play Store
• I have searched the project page to check if the question was already asked elsewhere

Topic

Development and Contributing

Your Request

Hello projectM team! I'm trying to understand what's wrong with my offscreen rendering pipeline. I launch it with the following command:

ffmpeg -i "/media/data/music/test.mp3" -f f32le -acodec pcm_f32le -ar 48000 - \
   | ./projectm-render-pipe \
   | ffmpeg -i "/media/data/music/test.mp3" -f ppm_pipe -r 60 -i - -map 0:a:0 -map 1:v:0 -y rendered.mp4

Here's what I get:

Here's what I expect to get:

As you can see there's a certain "glow" around the edges. Somehow when I render it offscreen it is not present. I'm kind of a rookie in graphics pipelines but I'm trying to implement offscreen rendering for ProjectM right now and before I submit a PR i need to get it right.

Here's my code that I use for offscreen rendering

#include "projectM-4/core.h"
#include <array>
#include <iostream>
#include <sstream>

#include <EGL/egl.h>
#include <GL/gl.h>
#include <projectM-4/projectM.h>
#include <vector>

static constexpr int kFixedFps = 60;
static constexpr int kAudioSamplerate = 48000;
static constexpr int kPbufferWidth = 1280;
static constexpr int kPbufferHeight = 720;
static constexpr int kBytesPerPixel = 3;

static const std::array configAttribs = {
        EGL_SURFACE_TYPE, EGL_PBUFFER_BIT,
        EGL_RENDERABLE_TYPE, EGL_OPENGL_BIT,
        EGL_BLUE_SIZE, 8,
        EGL_GREEN_SIZE, 8,
        EGL_RED_SIZE, 8,
        EGL_ALPHA_SIZE, 8,
        EGL_DEPTH_SIZE, 8,
        EGL_NONE
};    

static const std::array contextAttribs = {
      EGL_CONTEXT_MAJOR_VERSION, 3,
      EGL_CONTEXT_MINOR_VERSION, 3,
      EGL_CONTEXT_OPENGL_PROFILE_MASK, EGL_CONTEXT_OPENGL_CORE_PROFILE_BIT,
      EGL_NONE,
};

static const std::array pbufferAttribs = {
      EGL_TEXTURE_FORMAT, EGL_TEXTURE_RGB,
      EGL_TEXTURE_TARGET,  EGL_TEXTURE_2D,
      EGL_WIDTH, kPbufferWidth,
      EGL_HEIGHT, kPbufferHeight,
      EGL_NONE,
};

static auto screenshotToPpm(size_t width, size_t height, unsigned char *pixels) -> std::string {
    std::stringstream output;
    output << "P6 " << width << ' ' << height << ' ' << 255 << '\n';
    output.write((const char *) pixels, kBytesPerPixel * height * width);
    return output.str();
}

static void assertEglError(const std::string& msg) {
  EGLint error = eglGetError();

  if (error != EGL_SUCCESS) {
    std::stringstream exc;
    exc << "EGL error 0x" << std::hex << error << " at " << msg;
    throw std::runtime_error(exc.str());
  }
}


auto main(int argc, char *argv[]) -> int {
  // 1. Initialize EGL
  EGLDisplay eglDpy = eglGetDisplay(EGL_DEFAULT_DISPLAY);

  EGLint major = 0;
  EGLint minor = 0;
  eglInitialize(eglDpy, &major, &minor);
  assertEglError("initializing EGL");

  // 2. Select an appropriate configuration
  EGLint numConfigs = 0;
  EGLConfig eglCfg = nullptr;
  eglChooseConfig(eglDpy, configAttribs.data(), &eglCfg, 1, &numConfigs);
  assertEglError("choosing EGL config");

  // 3. Create a surface
  EGLSurface eglSurf = eglCreatePbufferSurface(eglDpy, eglCfg, pbufferAttribs.data());
  assertEglError("creating EGL surface");

  // 4. Bind the API
  eglBindAPI(EGL_OPENGL_API);
  assertEglError("binding OpenGL API");

  // 5. Create a context and make it current
  EGLContext eglCtx = eglCreateContext(eglDpy, eglCfg, EGL_NO_CONTEXT, contextAttribs.data());
  assertEglError("creating EGL context");

  eglMakeCurrent(eglDpy, eglSurf, eglSurf, eglCtx);
  assertEglError("making EGL context current");

  // from now on use your OpenGL context
  auto *projectm = projectm_create();
  projectm_load_preset_file(projectm, "file:///usr/share/projectM/presets/cream-of-the-crop/Waveform/Wire Circular/$$$ Royal - Mashup (191).milk", false);
  auto textures = std::vector<const char *> {"/usr/share/projectM/textures/cream-of-the-crop/"};
  projectm_set_texture_search_paths(projectm, textures.data(), 1);
  projectm_set_mesh_size(projectm, 64, 48);
  projectm_set_window_size(projectm, kPbufferWidth, kPbufferHeight);
  projectm_set_soft_cut_duration(projectm, 3);
  projectm_set_preset_duration(projectm, 30);
  projectm_set_easter_egg(projectm, 0.0);
  projectm_set_hard_cut_enabled(projectm, false);
  projectm_set_hard_cut_duration(projectm, 60);
  projectm_set_hard_cut_sensitivity(projectm, 1.0);
  projectm_set_beat_sensitivity(projectm, 1.0);
  projectm_set_aspect_correction(projectm, true);
  projectm_set_fps(projectm, kFixedFps);
  projectm_set_fixed_fps(projectm, kFixedFps);

  // buffer size for 60 fps should fit into 16ms
  // for 16000 bitrate that's 266.66 samples for each channel
  constexpr float samplesPerFrame = 2.0 * kAudioSamplerate / kFixedFps;    // 533.33
  constexpr auto bufferLen = static_cast<size_t>(samplesPerFrame) + 1;     // 534
  std::array<float, bufferLen> pcmBuffer{};

  constexpr int size = kBytesPerPixel * kPbufferHeight * kPbufferWidth;
  std::array<unsigned char, size> rgbBuffer{};
  size_t samplesRead = 0;
  for (size_t i = 0; ; i++) {
    glClearColor(0.0, 0.0, 0.0, 0.0);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
    
    // special arithmetic for cases where samplerate is not fully divisible by fps
    // without this we eventually get un-synced video and audio
    auto nextBatch = static_cast<size_t>(samplesPerFrame * i) - samplesRead;  // could be 533, could be 534
    std::cin.read(reinterpret_cast<char *>(pcmBuffer.data()), nextBatch * sizeof(decltype(pcmBuffer)::value_type));
    if (std::cin.eof()) {
      break;
    }

    samplesRead += nextBatch;
    // std::cerr << "Iteration " << i << " filled buffer size " << samplesToRead << " with data" << '\n';

    auto samplesToSubmit = std::min((unsigned int) bufferLen, projectm_pcm_get_max_samples());
    projectm_pcm_add_float(projectm, pcmBuffer.data(), samplesToSubmit, PROJECTM_STEREO);
    projectm_opengl_render_frame(projectm);

    glReadPixels(0, 0, kPbufferWidth, kPbufferHeight, GL_RGB, GL_UNSIGNED_BYTE, rgbBuffer.data());
    auto ppm = screenshotToPpm(kPbufferWidth, kPbufferHeight, rgbBuffer.data());
    std::cout << ppm;
  }
  

  // 6. Terminate EGL when finished
  eglDestroySurface(eglDpy, eglSurf);
  eglDestroyContext(eglDpy, eglCtx);
  eglTerminate(eglDpy);

  return 0;
}

I can also upload the full video or my small changes to timekeeping if you need it. I haven't touched any of GL-related code.

[revmischa]

you may be interested in the gstreamer plugin for graphics pipelines https://github.com/projectM-visualizer/gst-projectm

[Kaned1as]

@revmischa thank you, but as I can see it relies very heavy on gstreamer-provided OpenGL context, which I don't have 🥲

[revmischa]

@revmischa thank you, but as I can see it relies very heavy on gstreamer-provided OpenGL context, which I don't have 🥲

Yes but you could use gstreamer to replace your entire pipeline. It does internally what you're doing - piping ffmpeg outputs and inputs, but in a more natural and holistic fashion. Gstreamer is exactly for what you are trying to do here.

[Kaned1as]

@revmischa I meant that I won't have gstreamer OpenGL context because I eventually plan to use OSMesa on a completely headless machine, without audio and without any GPU, hardware acceleration or framebuffer 🥲

Sorry to bother you again, I will try to figure it out by myself

[kblaschke]

Not exactly sure why the glow is missing, as the rendering output should just contain whatever projectM draws.

I've opened a pull request yesterday adding an API call to render to a custom FBO, allowing to pass in a custom framebuffer with an in-memory texture attachment. This could possibly solve the issue, as you don't have to rely on the default framebuffer and surface. Edit: It's now merged to master.

I'll try out your code over the coming days when I've got some spare time and see what I can find.

[kblaschke]

Not really related to the issue, but as a note:
projectM currently uses the system clock to determine animation speeds in many different locations. If you render frames as fast as projectM can handle it, you'll probably end up with a different frame rate than the targeted 60 FPS, which will result in either slowed down animations (if encoding FPS are higher than real time) or sped up (if FPS are slower, e.g. projectM can't keep up with 60 FPS).

For libprojctM 4.2,we're also planning on adding a new API function to set the exact time of the next frame from the outside, so the rendering won't be tied to the system clock anymore. This way, you'll get stable animation speeds no matter how fast or slow you're encoding the video.

[Kaned1as]

@kblaschke yes! That's exactly what I did! I added a new API function to set fixed fps so that timekeeping is linear. You will probably need my patch to try the code above, I'll attach it in the evening once I'm home. Thank you for looking into this!

[Kaned1as]

Here's the patch for the timekeeper

diff --git a/src/api/include/projectM-4/parameters.h b/src/api/include/projectM-4/parameters.h
index 2d9c70c89..4df06c5b3 100644
--- a/src/api/include/projectM-4/parameters.h
+++ b/src/api/include/projectM-4/parameters.h
@@ -187,6 +187,21 @@ PROJECTM_EXPORT void projectm_get_mesh_size(projectm_handle instance, size_t* wi
  */
 PROJECTM_EXPORT void projectm_set_fps(projectm_handle instance, int32_t fps);
 
+/**
+ * @brief Set the current fixed frames per second.
+ *
+ * If this value is set to non-zero, timekeeping updates frames at a fixed rate.
+ * The application running projectM is expected to fill the audio buffer with
+ * the audio content up to expected for the current frame.
+ *
+ * E.g., a value of 60 fps means the application is expected to fill
+ * 16ms of audio each time render_frame is called.
+ *
+ * @param instance The projectM instance handle.
+ * @param fps The fixed FPS value projectM is expected to work with.
+ */
+PROJECTM_EXPORT void projectm_set_fixed_fps(projectm_handle instance, int32_t fps);
+
 /**
  * @brief Returns the current/average frames per second.
  *
diff --git a/src/libprojectM/PresetFactory.cpp b/src/libprojectM/PresetFactory.cpp
index 3c56876b2..bde847145 100644
--- a/src/libprojectM/PresetFactory.cpp
+++ b/src/libprojectM/PresetFactory.cpp
@@ -18,7 +18,7 @@ std::string PresetFactory::Protocol(const std::string& url, std::string& path)
 
     path = url.substr(pos + 3, url.length());
 #ifdef DEBUG
-    std::cout << "[PresetFactory] Filename is URL: " << url << std::endl;
+    std::cerr << "[PresetFactory] Filename is URL: " << url << std::endl;
 #endif
     return url.substr(0, pos);
 }
diff --git a/src/libprojectM/ProjectM.cpp b/src/libprojectM/ProjectM.cpp
index ae9390efa..98e5b95fb 100644
--- a/src/libprojectM/ProjectM.cpp
+++ b/src/libprojectM/ProjectM.cpp
@@ -354,6 +354,10 @@ void ProjectM::SetTargetFramesPerSecond(int32_t fps)
     m_targetFps = fps;
 }
 
+void ProjectM::SetFixedFramesPerSecond(int32_t fps) {
+    m_timeKeeper->SetFixedFps(fps);
+}
+
 auto ProjectM::AspectCorrection() const -> bool
 {
     return m_aspectCorrection;
diff --git a/src/libprojectM/ProjectM.hpp b/src/libprojectM/ProjectM.hpp
index b84e29335..cf9de3ad9 100644
--- a/src/libprojectM/ProjectM.hpp
+++ b/src/libprojectM/ProjectM.hpp
@@ -143,6 +143,15 @@ public:
      */
     void SetTargetFramesPerSecond(int32_t fps);
 
+    /**
+     * @brief Sets a new fixed frames per second value.
+     *        Timekeeping will use this value instead of OS time.
+     *
+     * @param fps The new frames per second value.
+     *
+     */
+    void SetFixedFramesPerSecond(int32_t fps);
+
     auto AspectCorrection() const -> bool;
 
     void SetAspectCorrection(bool enabled);
diff --git a/src/libprojectM/ProjectMCWrapper.cpp b/src/libprojectM/ProjectMCWrapper.cpp
index cbf5f6edc..e25a43beb 100644
--- a/src/libprojectM/ProjectMCWrapper.cpp
+++ b/src/libprojectM/ProjectMCWrapper.cpp
@@ -270,6 +270,13 @@ void projectm_set_fps(projectm_handle instance, int32_t fps)
     projectMInstance->SetTargetFramesPerSecond(fps);
 }
 
+void projectm_set_fixed_fps(projectm_handle instance, int32_t fps)
+{
+    auto projectMInstance = handle_to_instance(instance);
+    projectMInstance->SetFixedFramesPerSecond(fps);
+
+}
+
 void projectm_set_aspect_correction(projectm_handle instance, bool enabled)
 {
     auto projectMInstance = handle_to_instance(instance);
@@ -374,4 +381,4 @@ auto projectm_pcm_add_uint8(projectm_handle instance, const uint8_t* samples, un
 auto projectm_write_debug_image_on_next_frame(projectm_handle, const char*) -> void
 {
     // UNIMPLEMENTED
-}
\ No newline at end of file
+}
diff --git a/src/libprojectM/TimeKeeper.cpp b/src/libprojectM/TimeKeeper.cpp
index 612b0ab33..f0a5823c3 100644
--- a/src/libprojectM/TimeKeeper.cpp
+++ b/src/libprojectM/TimeKeeper.cpp
@@ -1,5 +1,6 @@
 #include "TimeKeeper.hpp"
 
+#include <math.h>
 #include <algorithm>
 #include <random>
 
@@ -16,11 +17,17 @@ TimeKeeper::TimeKeeper(double presetDuration, double smoothDuration, double hard
 
 void TimeKeeper::UpdateTimers()
 {
-    auto currentTime = std::chrono::high_resolution_clock::now();
+    double currentFrameTime = 0;
+    if (m_fixedFps != 0) {
+        m_secondsSinceLastFrame = 1.0 / m_fixedFps;
+        m_currentTime += m_secondsSinceLastFrame;
+    } else {
+        auto currentTime = std::chrono::high_resolution_clock::now();
+        currentFrameTime = std::chrono::duration<double>(currentTime - m_startTime).count();
+        m_secondsSinceLastFrame = currentFrameTime - m_currentTime;
+        m_currentTime = currentFrameTime;
+    }
 
-    double currentFrameTime = std::chrono::duration<double>(currentTime - m_startTime).count();
-    m_secondsSinceLastFrame = currentFrameTime - m_currentTime;
-    m_currentTime = currentFrameTime;
     m_presetFrameA++;
     m_presetFrameB++;
 }
diff --git a/src/libprojectM/TimeKeeper.hpp b/src/libprojectM/TimeKeeper.hpp
index 23d17df6c..d320f7f41 100644
--- a/src/libprojectM/TimeKeeper.hpp
+++ b/src/libprojectM/TimeKeeper.hpp
@@ -86,6 +86,10 @@ public:
         return m_secondsSinceLastFrame;
     }
 
+    inline void SetFixedFps(int32_t fps) {
+        m_fixedFps = fps;
+    }
+
 private:
     /* The first ticks value of the application */
     std::chrono::high_resolution_clock::time_point m_startTime{std::chrono::high_resolution_clock::now()};
@@ -107,6 +111,7 @@ private:
     double m_presetTimeA{};
     double m_presetTimeB{};
 
+    int m_fixedFps{0};
     int m_presetFrameA{};
     int m_presetFrameB{};

[kblaschke]

I've actually implemented it a bit differently, as per #740, allowing to pass in the actual frame time in fractional seconds. This is a bit more flexible, as it allows to set frame times with dynamic framerates, which may occur in video capturing or other non-deterministic rendering scenarios. In fixed-FPS use cases, the app can simply increment the frame time with the 1/FPS fraction.

The preset transition class also had to be fixed to use the TimeKeeper class. See PR #817 for details.

[kblaschke]

One other thing to note here is that the code example in the initial post uses EGL, which projectM currently doesn't support - it requires a GLX/GL Core context, as it internally uses the GLX vendor library for rendering. While it may work in general, some things will be broken. We have the exact same issue with recent Qt versions, as they also switched their GL rendering stack to EGL to support Wayland.

We already have issue #681 for this change, but it'll require some rework of how projectM uses OpenGL, plus potentially adding a build flag to choose between EGL and GLX. You can track the progress there for updates.

Since offscreen rendering support using a custom FBO/texture is now implemented in the master branch and the other potential issue is already tracked, I'll close this one.

[Kaned1as]

Understood! Thank you for spending your time on this!