I wonder how difficult it would be to make something similar that generated 3D models. Most of the examples look like they'd make good video game levels.
I wondered the same. There is some solid competition in this area right now, without AI assisted asset generation.
Unreal 5 has a new, free, 3d model library integrated as Quixel Bridge. [1]
Kitbash 3D, a company selling modular 3D sets used regularly in Beeple’s 2d provides mid-res, theme-based sets for customized use.
Neither take into account the idea of fully featured 3d objects being built from basic primitive using ML.
It makes sense that it will go this direction though, because it means designers can get unique 3D assets customized to the size and dimensions with less work.
Couple this with Apple’s photogrammetry in iOS 15 it seems original 3D assets available for training data will swell greatly.
Dungeon Alchemist seems really cool (I'm a backer), but I'm not entirely sure that it is related. DA is basically procedurally generated furnishing (with a few params), but it doesn't create 3D models from what I understand, it "just" shuffles around furniture.
Well, I think there is enough interesting research to put things in place. Not in single model. But, we have
0. This neural thing, of course, to create landscape-like 2D projections of a plausible scene.
1. Wave-function collapse models that synthesize domain data quite nicely when parametrized with artistic care - this is a "simpler" example of the concept. https://github.com/mxgmn/WaveFunctionCollapse
2. Fairly good understanding how to synthesize terrain. Terragen is a good example of this (although not public research, the images drive the point home nicely) https://planetside.co.uk/
So, we could use the source image from this as a 2D projection of an intended landscape as a seed to a wave-function collapse model that would use known terrain parametrization schemes to synthesize something usable (so basically create a Terragen equivalent model).
I think that's it plausibly more or less. But it's a "research" level problem still, I think, not something one can cook up by chaining the data flow from a few open source libraries together.
I think the theory's all there, it just needs reference material on the one hand and the work to be put in on the other. With the new Unreal 5 engine, I think there is a lot of room for technology where an artist sketches out a rock and tools come in to generate the small details - much like there's tools like speedtree and co nowadays to procedurally generate content.
