Support Java direct OpenGL rendering?

Developer Tools
1

Hi!

I’m trying to update my app from an older CNSDK version to the latest. In the older version, I found a (hacky, internal-abusing) way to use an existing SurfaceView renderer with an InterlacedSurfaceView, making it (relatively) easy to use with my application. Source code for the LeiaRendererAdapter which made that possible.

The newer SDK version has a better-defined interface, so that hack isn’t working anymore. Specifically, applications can’t grab the texture id from an InputViewsAsset anymore, and can’t wrap the LeiaGLSurfaceView.Renderer (which is how it accessed the gl context). But now the only way I see to reuse that SurfaceView renderer is to make a separate gl context, to render into a Surface wrapping the exposed SurfaceTexture. That looks like it’ll take a lot of orchestration and overhead.

Would it be possible to get an API that let us make OpenGL calls directly to the texture backing that SurfaceTexture? I think that a method to construct an InputViewsAsset from a renderer would be very convenient, but I could be flexible.

1 Like
2

+1 I, too am curious about the potential to not have to reflect output from an existing Texture to the CNSDK one, if possible. Being able to register a source texture, or get a handle to a place to write to without requiring the additional wrapper object/class, that would be ideal in this lower-level, but not quite super low level case of emulator output / ffmpeg output, etc

3

I found a way to do this on the latest version, and it doesn’t even seem that hacky! InputViewsAsset lets you provide your own implementation, which gets run on the GL thread and just needs to draw to a texture and pass that texture’s ID to the interlacer. I rolled my own RendererImpl, which takes an instance of a simple Renderer interface and just makes it draw to a texture.

1 Like
4

That’s exactly the way you can do that via the Java API. There is an example inside the CNSDK package in the sdk-test project - CubeAsset. It implements basic OpenGL rendering.

The only other way to have OpenGL rendering without Surface overhead is to use C/C++ API directly, effectively integrating CNSDK inside your rendering pipeline (contrary to the InputViewsAsset approach where you plug in inside our rendering pipeline).

1 Like
5

I’m trying to use the InputViewsAsset approach but i’m getting this error:

15:57:40.858 D [EmulationFragment] Surface Created
15:57:40.859 D [EmulationFragment] Surface changed. Resolution: 2560x1600
15:57:40.859 I [ 7.169404] Frontend main/jni/native.cpp:Java_org_citra_citra_1emu_NativeLibrary_surfaceChanged:286: Surface changed
15:57:40.859 D [EmulationFragment] Starting emulation thread.
15:57:40.861 I [ 7.170158] Frontend main/jni/native.cpp:RunCitra:132: Citra starting…
15:57:40.863 I [LeiaSDK] [info] version: 0.7.28 (refs/heads/main - 1c3d3935a7689b416e48d5ebfeaddb3b9fd03fff)
15:57:40.864 W java.lang.ClassNotFoundException: Didn’t find class “com.epicgames.unreal.GameActivity” on path: DexPathList[[zip file “/data/app/~~QJKZIrSkXWpA4mnXvvgC6w==/org.citra.citra_emu.canary.debug–wZjUuY0cHO_AW-OWswU_Q==/base.apk”],nativeLibraryDirectories=[/data/app/~~QJKZIrSkXWpA4mnXvvgC6w==/org.citra.citra_emu.canary.debug–wZjUuY0cHO_AW-OWswU_Q==/lib/arm64, /data/app/~~QJKZIrSkXWpA4mnXvvgC6w==/org.citra.citra_emu.canary.debug–wZjUuY0cHO_AW-OWswU_Q==/base.apk!/lib/arm64-v8a, /system/lib64, /system_ext/lib64]]
15:57:40.864 W at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:259)
15:57:40.864 W at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
15:57:40.864 W at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
15:57:40.864 W at java.lang.Runtime.nativeLoad(Native Method)
15:57:40.864 W at java.lang.Runtime.nativeLoad(Runtime.java:1126)
15:57:40.864 W at java.lang.Runtime.loadLibrary0(Runtime.java:1080)
15:57:40.864 W at java.lang.Runtime.loadLibrary0(Runtime.java:1003)
15:57:40.864 W at java.lang.System.loadLibrary(System.java:1661)
15:57:40.864 W at com.leia.sdk.LeiaSDK.(LeiaSDK.java:362)
15:57:40.865 W at com.leia.sdk.views.InterlacedRenderer.GLThread_initInterlacer(InterlacedRenderer.java:68)
15:57:40.865 W at com.leia.sdk.views.InterlacedRenderer.onDrawFrame(InterlacedRenderer.java:130)
15:57:40.865 W at com.leia.sdk.views.LeiaGLSurfaceView$GLThread.guardedRun(LeiaGLSurfaceView.java:1468)
15:57:40.865 W at com.leia.sdk.views.LeiaGLSurfaceView$GLThread.run(LeiaGLSurfaceView.java:1195)
15:57:40.870 E SurfaceView[org.citra.citra_emu.canary.debug/org.citra.citra_emu.activities.EmulationActivity]#2(BLAST Consumer)2 connect: already connected (cur=1 req=1)
15:57:40.870 E eglCreateWindowSurface: native_window_api_connect (win=0xb400007ab5dac540) failed (0xffffffea) (already connected to another API?)
15:57:40.870 E eglCreateWindowSurfaceTmpl:681 error 3003 (EGL_BAD_ALLOC)
15:57:40.870 E eglQuerySurfaceImpl:865 error 300d (EGL_BAD_SURFACE)
15:57:40.870 A [ 7.180663] Frontend main/jni/emu_window/emu_window_gl.cpp:CreateWindowSurface:148: EmuWindow_Android_OpenGL eglCreateWindowSurface() returned error 12291
15:57:40.886 I [ 7.196341] Config main/jni/config.cpp:LoadINI:57: Successfully loaded /config/config.ini
15:57:40.913 E [ 7.223183] Service.FS core/file_sys/ncch_container.cpp:LoadHeader:157: 16384
15:57:40.913 I [ 7.223211] Config common/settings.cpp:LogSettings:83: Citra Configuration:

image
image818×313 16.3 KB

i could see if maybe i can destroy any existing instance :G

6

InputViewsAsset already gives you a valid EGL/gles context. You should not create EGL context/surface/display on the same thread. Just check the CubeAsset sample, you will see that it does not do any EGL calls.

7

in my case tho, the case of the citra emulator,

their code needs to set up their OGL context in a special way at a special time (aka, pre-determined)

so the less i have to dig into or modify any of that surface’s creation lifecycle, the better,
i’d like to just be able to hand the API a reference to a “live” surface if possible

8

So they set up their own SurfaceView? If so, the only way is to use CNSDK native API. CNSDK’s Java API’s only purpose is to provide a view implementation. Anything more sophisticated should use the native API.

1 Like
9

I gave a quick look at the project. You need to do the same thing we do in the Moonlight3D.

Here is the main commit that adds CNSDK into Moonlight - Add optional CNSDK Integration · LeiaInc/Moonlight3D@bad8551 · GitHub.

See app/src/3d/com/limelight/LeiaHelper.java. StreamView from this file corresponds to your LeiaSurfaceView.

1 Like
10

awesome, i’ll give this a look, thank you!

11

seems like this approach (in moonlight) still involves InputViewsAsset.createEmptySurfaceForVideo to kick things off

image
image1202×587 38.2 KB

which, i guess if i hook in earlier, to where the context is created, i can do it that way,
but, i’m just happening to be coming at it from the the other direction and could’ve saved some time using a setSurfaceTexture or setSurface instead of the setViewAsset route which requires a InputViewsAsset

it seems like maybe the deeper ask here would be having a InputViewsAsset constructor for an existing surface. maybe there is and i just ran out of coffee steam today

More Logs if helpful:

12

is it unexpected to see this error (i’m on latest lume ui, and leia media service, display config) seem up-to-date

image
image1412×569 73.9 KB

13

I did not say you don’t need it though.

That’s just wrong. You can’t even theoretically set a Surface for InterlacedSurfaceView. InterfacedSurfaceView is the one who give you a Surface.

As I can see from the code, you just need to remove binding.surfaceEmulation.holder.addCallback(this). And instead rely on the callback from createEmptySurfaceForVideo to give you a Surface. Which then will be pushed into emulationState.newSurface

1 Like
14

You can ignore that, it’s harmless. It’s fixed in the latest CNSDK. We should have a release soon.

1 Like
15

when i comment out the call to binding.surfaceEmulation.holder.addCallback(this)
the Leia SDK goes into 3d backlight mode, the face-tracking starts (camera notification light shows up) – but the game never loads :G

wonder if there’s some method call i need to mock

16

Did you just comment out binding.surfaceEmulation.holder.addCallback(this) and nothing else? The emulator needs a Surface. You should get it from the SurfaceTextureReadyCallback passed to createEmptySurfaceForVideo and pass it into emulationState.newSurface. At least that’s the only way I see how it could work at all.

1 Like
17 
val sL = LeiaHelper3D.SurfaceListener{ surface ->
            Log.debug("SurfaceListener onSurfaceChanged called") // Log debug warning
            emulationState.newSurface(surface)
        }
        binding.surfaceEmulation.setSurfaceListener(sL)

so i’ve added this

now it gets past “loading…” but i just see a black screen, which, is progress

two commits

Lp2 jan 2024 take2 by jakedowns · Pull Request #15 · jakedowns/citra · GitHub

Lp2 jan 2024 take2 by jakedowns · Pull Request #15 · jakedowns/citra · GitHub

18

Looks fine now. I suppose emulationState.newSurface(surface) is indeed executed, right? Can you share some logs?

1 Like
19

Could you also please point me to the place in the project where you tell it to render a side-by-side into the Surface?

1 Like
20

hey there! yeah ill get you some logs and links asap (got hit with a cold but ill be back online in a day or two)

thanks for your help! excited to get this updated for MWC