Notes from glTF Interactivity Extension
Khronos recently released a glTF interactivity specification requesting public comment. We had some concerns about the quality of the spec, and the availability of sample files to test on, so we reached out to Ben Houston and glTF 3D on Twitter/X to obtain clarification and determine the best way to contribute, going forward.
Core take-aways for us included:
- the emphasis of keeping the spec simple, or "boring". Rather than try to include every edge case, the use of events and translators add functionality without complicating things.
- The overall approach of the system having three components - Events and variables, the behaviour graph and the runtime object model - is very powerful.
From a community perspective, it was noted that taking in ideas from different members, especially more independent developers should be improved. There needs to be engagement in publically accessible ways. We were very pleased at the engagement in the X Spaces event, and hope to have more in future.
There is a Discord specifically for the Khronos glTF, but content from there does not often make it into (closed room) meetings. The participants agreed that more blogs about use cases would help with exposure, and allow discovery by interested parties.
Relevant links
- The blogpost requesting comment: https://www.khronos.org/blog/gltf-interactivity-specification-released-for-public-comment
- Khronos event showcasing the spec (Sept 2022) https://www.khronos.org/events/delivering-interactive-experiences-with-gltf (recording? transcript?)
- presentation at MSF: https://www.youtube.com/watch?v=EmpPw2VSNZs
- The React-based Digital Content Creation (DCC) app that is designed to streamline the process of authoring glTF assets with the KHR_interactivity extension: https://github.com/KhronosGroup/glTF-InteractivityGraph-AuthoringTool/tree/initial-work-merge
https://github.com/bhouston/behavior-grapharchived
The detailed notes of the discussion can be found below.
Key points from the conversation on Thursday and Friday, 20 and 21 June 2024
We had a dry-run on the Thursday to figure out how Spaces work, but ended up having quite a productive conversation anyway. These notes combine the conversations from both Thursday and Friday:
- The interactivity spec originated from Adobe's work on trigger-action lists for Adobe Aero and USDZ.
- There was debate between trigger-action lists, behavior graphs, and WASM approaches, with behavior graphs ultimately chosen.
- The spec development process involved studying existing systems like Unreal Blueprints and Unity Visual Scripting.
- Security considerations were a major factor in the design, leading to a more constrained system than arbitrary JavaScript or WASM.
- The goal was to create a "boring" standard that consolidates best practices rather than innovating.
- The spec introduces concepts like the glTF object model, events, and variables that can be built upon by future extensions.
- It's designed with a layered approach, allowing different levels of capability and security.
- The spec builds on the Animation Pointer extension for referencing parts of the glTF.
- Custom events allow GLTFs to send and receive messages, potentially enabling communication between nested GLTFs.
- There are ongoing efforts to finalize related extensions like audio and physics.
- The spec is designed to be flexible but with performance considerations in mind.
- Implementations are expected to limit execution time to maintain performance.
- There are concerns about the spec being incomplete, lacking examples, JSON schemas, and other expected components.
- The current implementation is limited in terms of data types and capabilities compared to full programming languages.
- There's discussion about potential future work, including adding more complex data types and operations.
- Google is working on implementing the spec for use in Google Maps and other products.
- The Godot engine team expressed interest in the spec but also raised concerns about other needed features, like consistent UUIDs for nodes across exports.
- There was discussion about the challenge of maintaining unique identifiers for nodes when optimizing or merging assets.
- The Blender team is exploring how to integrate the spec with their geometry nodes system.
- There are some concerns about the expansion of glTF's scope and potential performance implications.
- The importance of having multiple implementations before ratification was emphasized.
- The community was encouraged to contribute to the implementation efforts, particularly in projects like three.js.
- The spec is not intended to replace game engines but to enable interactivity for simpler use cases.
- There's a need for more examples and supporting materials to help people understand and implement the spec.
- The working group is considering how to best respond to community feedback and concerns about the ratification timeline.