SceneTeller: Language-to-3D Scene Generation

ECCV 2024

Başak Melis Öcal¹, Maxim Tatarchenko², Sezer Karaoğlu¹, Theo Gevers¹

¹University of Amsterdam ²Bosch Center for Artificial Intelligence

Abstract

Designing high-quality indoor 3D scenes is important in many practical applications, such as room planning or game development. Conventionally, this has been a time-consuming process which requires both artistic skill and familiarity with professional software, making it hardly accessible for layman users. However, recent advances in generative AI have established solid foundation for democratizing 3D design. In this paper, we propose a pioneering approach for text-based 3D room design.

Given a prompt in natural language describing the object placement in the room, our method produces a high-quality 3D scene corresponding to it. With an additional text prompt the users can change the appearance of the entire scene or of individual objects in it. Built using in-context learning, CAD model retrieval and 3D-Gaussian-Splatting-based stylization, our turnkey pipeline produces state-of-the-art 3D scenes, while being easy to use even for novices.

Overview

Given a textual prompt in natural language delineating the desired spatial positions and orientations of objects within the scene, a 3D scene layout is generated using in-context learning. An initial 3D scene is assembled for the predicted layout, which is then fitted with a 3D Gaussian Splatting representation. This representation is then used to stylize the scene according to the user-provided text prompt, and subsequently render the final images of the scene.

Video

"A double bed is positioned in the middle of the room, a bit nearer to the upper left wall, with a perpendicular orientation. A wardrobe is placed at the middle of the bottom wall, somewhat nearer to the left wall, with no rotation. A nightstand is positioned at the top left corner, with a perpendicular orientation. Another nightstand is situated close to the bottom left corner, also with a perpendicular orientation. A cabinet is set at the middle of the right wall, slightly nearer to the bottom wall, oriented perpendicularly but in the opposite direction. Another cabinet is set near the top right corner, oriented perpendicularly but in the reverse direction. Make it Halloween themed."

[...]

"Centrally positioned along the bottom wall, with a slight bias towards the right wall, is a wardrobe with no rotation. On the upper wall, there's a double bed situated in the middle but nearer to the left wall and is oriented in the opposite direction. Nestled in both upper corners are nightstands, both facing backwards. Make it Barbie style."

[...]

"In the middle of the room, a double bed is placed, slightly more towards the bottom right wall, with its orientation being perpendicular and facing the reverse direction. A wardrobe is situated at the center of the top wall, a bit closer to the right wall, with its orientation facing backwards. Near the top right corner of the room, a table is situated, oriented perpendicularly but in the reverse direction. At the bottom left corner, a cabinet is positioned, perpendicular to the walls. Another table is situated near the bottom right corner, oriented perpendicularly but facing the opposite direction. Turn the double bed into golden."

[...]

"In the upper right corner, a multi seat sofa is situated with a backwards orientation. Positioned at the middle of the right wall, closer to the lower boundary, is a coffee table, also oriented backwards. A tv stand is set up at the bottom right corner, without any rotation. Towards the center of the left wall, a dining table is positioned, a bit nearer to the upper wall, and is oriented perpendicularly. A dining chair is positioned at the middle of the upper wall, a bit closer to the left wall, with an opposite perpendicular orientation. Another dining chair is found in the center of the room, leaning more towards the upper left wall, also oriented oppositely to the perpendicular. At the upper left corner, a dining chair is positioned, oriented perpendicularly. Lastly, another dining chair is situated at the middle of the left wall, nearer to the upper wall, and it is also perpendicularly oriented. Make it coastal style."

[...]

"A double bed is positioned in the middle of the room, a bit nearer to the top left wall, set at a right angle. In close proximity to the top left corner, a nightstand is situated, set at a right angle. Another nightstand can be found placed near the bottom left corner, also set at a right angle. In the bottom left corner, a wardrobe is positioned, with no particular rotation. Lastly, a shelf is set in the top right corner, with no rotation. Make it Pirates of the Caribbean style."

[...]

"The room is arranged with a double bed placed centrally, slightly nearer to the bottom-left wall and is aligned at right angles. A nightstand is situated adjacent to the upper left end, with a perpendicular orientation. Another nightstand can be found near the bottom left corner, also arranged perpendicularly. The wardrobe is positioned at the mid-point of the upper wall, skewed slightly to the left side, and is facing backwards. Make it Ice Age themed."

[...]

"The room's central space is occupied by a double bed, positioned a bit closer to the upper left wall and arranged in a perpendicular manner. A wardrobe is positioned at the midpoint of the upper wall, leaning more towards the left wall, and is facing backwards. Near the upper left corner, a nightstand is situated and is set up perpendicularly. A dressing table can be found closer to the lower left corner and is also oriented perpendicularly. A chair is positioned at the midpoint of the lower wall, leaning slightly more on the left side, and is tilted almost halfway between a horizontal and vertical orientation but in the reverse direction. A shelf is positioned near the lower right corner and is set up with a straight orientation, without any rotation. Make it Valentine's Day themed."

[...]

"The coffee table is situated in the middle of the upper wall, albeit a tad nearer to the right wall, and is aligned perpendicularly facing the opposite direction. Another coffee table can be found in the middle of the right wall, a bit nearer to the lower wall, and is facing towards the rear. Nestled in the upper right corner is another coffee table, positioned without any rotation. In the upper left corner, a tv stand is positioned perpendicularly. A multi seat sofa is positioned near the upper right corner, facing perpendicularly but in the reverse direction. Close to the lower left corner, a dining table is situated without any rotation. A dining chair can be found in the center of the lower wall, slightly nearer to the left wall, and is facing in its original direction. Another dining chair is positioned near the lower left corner, also without any rotation. A third dining chair is situated in the room's center, closer to the lower left wall, and oriented towards the rear. A fourth dining chair is positioned in the middle of the left wall, slightly closer to the lower wall, positioned towards the rear. Lastly, a console table can be found in the lower left corner, placed without any rotation. Turn the dining table into pink."

[...]

"The double bed is situated at the middle of the bottom wall, with a slight bias towards the right wall, and is not rotated. Positioned almost in the middle of the room, the stool leans slightly towards the upper right wall and is not rotated. The tv stand is placed at the middle of the top wall, a bit closer to the right wall and is oriented in reverse. The children cabinet is positioned in the upper right corner, set perpendicular but facing the opposite direction. The wardrobe finds its place at the bottom left corner, with no rotation. The coffee table is stationed at the bottom right corner, also with no rotation. Make it botanical themed."

[...]

"The wardrobe is situated at the midpoint of the left wall, a bit nearer to the top wall, and arranged in a perpendicular manner. The double bed is positioned in the middle of the room, just a little closer to the upper right wall, with a backwards orientation. The dressing table can be found at the upper right corner, facing backwards. A nightstand is placed near the upper left corner, also oriented in a backwards direction. Make it Cinderella style."

[...]

"Situated at the heart of the room, a double bed is positioned, leaning somewhat nearer to the upper right wall with a backward orientation. The wardrobe is positioned in the middle of the left wall, slightly closer to the upper wall, with a perpendicular orientation. Positioned near the upper left corner, a nightstand is oriented backwards. A tv stand is situated in the middle of the lower wall, leaning a bit to the right wall with no rotation in its orientation. Make it Garfield themed."

[...]

"A double bed is positioned centrally in the room, but slightly nearer to the bottom left wall, aligned in a perpendicular manner. A nightstand is placed at the mid-point of the left wall, leaning a bit towards the upper wall, with a perpendicular alignment. Another nightstand is situated at the bottom left corner, arranged perpendicularly. A wardrobe is stationed at the center point of the upper wall, a bit closer to the left wall, with a backwards orientation. Make it 101 Dalmatians style."

[...]

"The dining table is situated in the middle of the room, but slightly leaning towards the lower left wall, with its orientation reversed and perpendicular. One dining chair is positioned near the lower left corner, with its orientation being perpendicular. Another dining chair is placed in the middle of the left wall, a bit nearer to the lower wall, and is also perpendicular. Yet another dining chair is set up at the midpoint of the left wall, but it's slightly more towards the upper wall and is perpendicular. One dining chair is near the lower right corner, with its orientation reversed and perpendicular. Another dining chair can be found at the center of the right wall, a bit closer to the lower wall, and it is perpendicular but facing the opposite direction. Another dining chair is at the center of the right wall, but slightly more towards the upper wall and its orientation is perpendicular but reversed. One more dining chair is in the middle of the upper wall, closer to the left wall, and is oriented backwards. Finally, a dining chair is at the center of the lower wall, a bit closer to the left wall, and it is oriented without any rotation. Make it marble"

[...]

"Positioned at the top left corner is a wardrobe, standing perpendicularly. On the top right corner, a nightstand is placed perpendicularly but facing the opposite direction. Another nightstand is situated at the bottom right corner, also perpendicularly aligned but facing in the opposite direction. A double bed can be found slightly nearer to the upper wall, in the middle of the right wall, again oriented perpendicularly but in the opposite direction. A stool is centrally placed in the room, slightly nearer to the upper left wall, and is oriented perpendicularly. Turn the double bed into black coloured bed."

[...]

"At the heart of the room, a double bed is positioned a tad nearer to the top-right wall, its orientation perpendicular but in reverse. Adjacent to the top-right corner, a nightstand is placed, its orientation also perpendicular but in reverse. A wardrobe can be found centrally aligned with the bottom wall, subtly closer to the right wall, with no rotation in its orientation. Make it Starry Night Van Gogh painting style."

[...]

BibTeX

@article{ocal2024sceneteller,
  title     = {SceneTeller: Language-to-3D Scene Generation},
  author    = {{\"O}cal, Ba{\c{s}}ak Melis and Tatarchenko, Maxim and Karaoglu, Sezer and Gevers, Theo},
  journal   = {arXiv preprint arXiv:2407.20727},
  year      = {2024},
}