Applications of a Deep Neural Network to Illustration Art Style Design of City Architectural

1. Introduction

Style transfer is a technique designed to emulate the unique stroke style of a specific artist and apply it to an image, essentially converting a photograph into a synthetic work of art. Emerging from the field of computer vision, it extends its scope to encompass the broader field of texture synthesis. In 2015, Gatys et al. [1], inspired by convolutional neural networks, introduced a neural style transfer algorithm. This algorithm employs a pre-trained convolutional neural network to capture artistic features, discerning between style features and content features in images. Subsequently, it rearranges these features to generate a new image, with the goal of aligning the style features of the new image with those of a selected artistic reference and simultaneously matching the content features with those of the original content image. The method excels in seamlessly blending diverse artistic styles with images [2-7].

The neural style transfer model is an automated learning model which utilizes neural networks. In contrast to generative adversarial networks, which demand a substantial dataset, neural style transfer achieves image generation with just one style image and one content image. In terms of resolution, neural style transfer is constrained only by a minimum resolution limit, with no upper limit within the permissible range of computer performance, while generative adversarial networks typically impose stricter requirements on input and output image resolutions. Neural style transfer operates as a top-down generation method, updating pixels on the target image through self-updating. Numerous subsequent studies have enhanced or broadened the neural style transfer algorithm.

Presently, mainstream neural style transfer algorithms can be categorized into image-optimizationbased online neural methods and model-optimization-based offline neural methods, based on different image generation approaches. Image-optimization-based online neural methods generate stylized images by iterating over pixels on a noisy image. This category can be further subdivided into parametric neural methods with summary statistics [7-10] and non-parametric neural methods with Markov random fields (MRFs) [11], depending on the stylization method employed. Model-optimization-based offline neural methods encompass feed-forward stylization models [12-14] and methods based on generative adversarial networks [15-17].

Deep learning models, known for their ability to learn from vast datasets, have found application in architectural design generation projects, demonstrating impressive performance [18-21]. The neural style transfer model, requiring minimal training data and delivering satisfactory outcomes, proves particularly valuable in sparking design concepts during the initial phases of a project. For instance, Campo et al. [19] from the University of Michigan employed the neural style transfer algorithm in 2020 to transpose the urban texture of Barcelona onto a lunar landscape. They explored how humans categorize urban textures across different cities based on visual features, highlighting the algorithm's effective reflection of artificial intelligence's significance in architectural design. While the final outcome may not stand out remarkably, the migrated texture serves as a design reference, capturing Barcelona's original texture pattern to some extent.

Furthermore, Ozel from UCLA and fellow researchers [22] utilized the neural style transfer algorithm to integrate artistic pattern textures into the exteriors of building facades. By reconstructing 3D models from 2D images, they executed style transformation designs on urban building facades. This work underscores that architectural design often draws influence from various visual design disciplines. The neural style transfer algorithm enables the integration of content and elements from these visual artworks into the design of building facades, suggesting conceptual design ideas for interdisciplinary artistic collaboration.

The primary objective of this paper is to seek inspiration for the design of artistic features in urban architecture, based on the artistic perspective of children's illustrations and picture books and urban classic architectural aesthetic paradigms. This paper selects landmark buildings such as the Qingdao Guoxin Haitian Center, Qingdao Guest House, and the TAG Art Museum as the experimental targets. The experimental target architectural elements are used as content images, and classic featured children's illustrations and images from children's picture books are used as style images. The neural style network effectively transfers artistic stylization while preserving the distinct architectural characteristics inherent in the content images. This paper uses deep learning technology as a medium to explore the possibility of integrating architectural design styles and children's illustrations and offers a fresh perspective and valuable support for the artistic design of contemporary urban structures.

2. Neural Style Transfer Model

The structure of the neural style transfer model is predominantly based on a pre-trained convolutional neural network. This network extracts content features from the content image, processes style features from the style image, and restructures the target image to generate a novel synthesis. The resulting image exhibits stylistic characteristics similar to the input style image while retaining the structural content of the input content image. The loss function assesses the difference between the style features of the reconstructed image and those present in the style image. Additionally, it takes into account the distinction between the content features of the reconstructed image and those of the content image. The style and content differences are computed using a weighted sum method to control their respective influence ratios. The parameters of the convolutional neural network in the model remain fixed, and the loss function iteratively optimizes pixel values on the reconstructed image through backpropagation.

The architecture of the neural style transfer model is primarily structured around a pre-trained convolutional neural network (Fig. 1). This network plays a pivotal role in extracting content features from the content image and processing style features from the style image. The resultant target image is then reorganized to produce a new synthesis, aligning with the stylistic characteristics of the input style image while preserving the structural content of the input content image. The loss function critically evaluates the discrepancy between the style features of the reconstructed image and those of the style image. Additionally, it accounts for the difference between the content features of the reconstructed image and the content features of the content image. These style and content differences are computed using a weighted sum method, allowing for control over their respective influence ratios. Notably, the parameters of the convolutional neural network in the model remain constant throughout, with the loss function iteratively optimizing pixel values on the reconstructed image through the process of backpropagation.

Fig. 1.

Neural style transfer model framework [ 1].

In the realm of the neural style transfer model, fundamental concepts revolve around content features and style features, which significantly impact the model's generation outcomes. Content features serve as descriptors extracted from images through convolutional neural networks, providing insights into the elements and structures embedded in the visual content [23]. The widely adopted VGG19 in neural style transfer, boasting 19 convolutional and fully connected layers, exhibits the capability to extract diverse feature information across different layers. While shallow layers may capture basic straight-line edge features [24], deeper convolutional layers progressively unveil richer texture features. Consequently, even intricate feature information at an individual level can be discerned, contributing to the categorization of differences in images.

In contrast, style features are inherently complex, encapsulating elements such as brushstrokes, textures, structures, layouts, and colors—an intricacy that poses challenges in precise articulation. In the neural style transfer model, the style feature of the image is represented by the cross-correlation between each feature map of the convolutional layer.

The neural style transfer model achieves the transfer of desired style through continuous iterative optimization of style and content loss on the target image. The pioneering work of Gatys et al. [1] in 2015 introduced the application of the VGG network and the Gram matrix for style transfer. The activation functions of the VGG network were utilized to represent the content characteristics of the image, focusing on its overall structure and outline. Subsequently, the Gram matrix was employed to characterize its stylistic features. The essence of image style transfer lies in minimizing the disparity between the content and style features of the generated image and those of the input image.

The formulation of the content loss function, which precisely quantifies the disparity in content features, is:

Definition of style loss function:

Definition of Gram matrix:

The definition of the total loss function is:

Eliminating the need for extensive training on large-scale datasets, the neural style transfer model utilizes the pre-trained VGG19 model [25]. To generate results, a content image and a style image are provided as inputs to the model. Despite the numerous adjustable parameters, the weight ratio between style difference and content difference significantly influences the final outcome. These parameters determine the blend of content and style, with a higher weight intensifying the stylistic characteristics from the style image. Leveraging the deep feature extraction capabilities of the VGG19 model across various resolutions, higher-resolution input images can be selected for enhanced results.

The code for the neural style transfer model utilized in this study originates from an open-source GitHub project that implements neural style transfer using the TensorFlow deep learning library. The model described in the paper has been expanded to a certain extent, and default parameters have been optimized to produce satisfactory results, making it more user-friendly. Throughout the research process, the code ran on Python 3.6, with TensorFlow, OpenCV, NumPy, and SciPy installed as related libraries. Specifically, TensorFlow requires version 1.0, and NumPy should be version 1.6 or earlier. Compatibility issues with other libraries did not arise during the experiments. In this experiment, we used limitedmemory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) as the optimizer, because it has better performance than Adam optimizer on this task. However, L-BFGS optimizer also has a drawback, which is that it requires more memory space. To meet the memory demand of L-BFGS optimizer, we used a GTX 3090Ti graphics card with 22 GB of memory. In the experiment, we set the learning rate to 1 and performed 2,000 iterations.

3. Results and Discussion

3.1 Content Image Introduction

As shown in Fig. 2, the images of experimental buildings in this paper choose the Qingdao Guoxin Haitian Center (https://www.archdaily.cn), The Museum of the Former German Governor’s House (https://foursquare.com) and the TAG Art Museum (https://k.sina.com.cn). The Qingdao Guoxin Haitian Center is a landmark building in Qingdao. The main building is as high as 369 m. It has won the “Highest Award for Best High-Rise Building in Asia.” The Museum of the Former German Governor’s House is located in the center of the old city of Qingdao at the foot of the picturesque Signal Hill Park. It is a typical European castle-style building. The TAG Art Museum is situated on the southern shore of Tangdao Bay in the West Coast New District of Qingdao, within the international comprehensive art community known as Xihai Art Bay. Renowned French architect Jean Nouvel and landscape architect Gilles Clement collaborated in its design and construction. The entire art museum is surrounded by mountains and seas, reflecting the harmonious relationship between the coast, the sea and the natural landscape, as well as the concept of respecting the natural ecology.

Fig. 2.

Image of the experimental building area: (a) The Qingdao Guoxin Haitian Center, (b) the Museum of the Former German Governor’s House, and (c) the TAG Art Museum.

3.2 Style Image Introduction

The style images in this paper, as shown in Fig. 3, are selected from domestic and foreign illustration art works of different eras, styles and creative techniques. Some are commercial illustrations, some are children’s picture book illustrations and some are conceptual illustrations expressing personal thoughts. They are as follows: “Art Forms of Nature” by Ernst Haeckel [26], poster illustration by Ori Toor (https:// www.blackdragonpress.co.uk/blogs/news/melancholia-by-ori-toor), “Wishes” by Victo Ngai (http://www. letstalkpicturebooks.com/2021/07/wishes.html), conceptual illustration by Victo Ngai (https://victongai. com/science), “Monster” Quilt by Jillian Tamaki (https://embroideryasart.blogspot.com/2010/11/ jillian-tamaki-monster-quilt.html), “Strawberry” by Michael Sowa (https://www.etsy.com/listing/1305 720750/ring-around-the-strawberry-postcard), “Birds of Paradise” by Yue Wang, “Mixc World Launch” by Victo Ngai (https://victo-ngai.com/mixc-world-launch), and “The real boat” by Victoria Semykina (https://www.semykina.com/ships).

Fig. 3.

Illustration-based art style images: (a) “Art Forms of Nature” illustration by Ernst Haeckel, (b) poster illustration by Ori Toor, (c) “Wishes” by Victo Ngai, (d) conceptual illustration by Victo Ngai, (e) “Monster Quilt” by Jillian Tamaki, (f) “Strawberry” by Michael Sowa, (g) “Birds of Paradise” by Yue Wang, (h) “Mixc World Launch” by Victo Ngai, and (i) “The Real Boat” by Victoria Semykina.

3.3 Case 1 of Architectural Art Style Transfer: The Qingdao Guoxin Haitian Center

The Qingdao Guoxin Haitian Center is located on the bank of the Fushan Bay in the bustling commercial circle of the city center. It occupies the best height and perspective for viewing the golden coastline. In terms of architectural design, the architectural form of the “hexagonal matrix” and the design of laminated windows are adopted. The unique design concept echoes the urban landscape of mountains and seas, coastal landscapes and long-standing cultural heritage. The building facade is designed to form waves and look like sparkling sea waves from a distance, creating a fantastic and three-dimensional artistic conception.

As shown in Fig. 4(b), Ernst Haeckel was a multifaceted individual, encompassing roles as a German zoologist, naturalist, eugenicist, philosopher, physician, professor, marine biologist, and artist. Haeckel's published artistry comprises more than 100 intricate, multicolored illustrations depicting animals and marine creatures. These illustrations are compiled in his work “Kunstformen der Natur” (“Art Forms of Nature”), a book that left a lasting impact on the Art Nouveau artistic movement. Haeckel's illustrations use extremely precise graphics and subtle shading to present the forms of plants and animals, helping the world popularize microbes invisible to the naked eye and bringing art and science closer together. As shown in Fig. 4(c), Ori Toor is a successful American illustrator. His illustrations now possess an additional dimensionality, featuring an imperceptible depth that imparts a sense of multidirectional tilt to the scenes. His works are depicted with flowing lines and rich details, leading readers into their own world and generating unlimited imagination. Fig. 4(d) is taken from the book illustration created by the Hong Kong illustrator Victo Ngai for the children's picture book “Wishes”. The color of this work is harmonious and extremely full and full of tension. The flowing lines convey a toughness, show the dynamic beauty of the waves and have a strong Chinese style.

Fig. 4.

The rendering of the artistic style transfer of the Qingdao Guoxin Haitian Center. Among them, (a) the Qingdao Guoxin Haitian Center picture, (b) Ernst Haeckel's illustration work "Art Forms of Nature," (c) Ori Toor's poster illustration, (d) Victo Ngai's illustration work "Wishes," (e) the effect image of style transfer based on Fig. 4(b), (f) The effect image of style transfer based on Fig. 4(c), and (g) the effect image of style transfer based on Fig. 4(d).

Fig. 4(e), 4(f), and 4(g) are the three transferred images. Through deep learning, on the basis of the original architectural structure design, three illustration artworks of different styles and ocean-related themes are integrated. The appearance and color are more dazzling, presenting a visual effect with impact, liveliness, vitality and creativity. The integration of illustrations brings more possibilities to architectural design. Visual art is not limited to a certain style but is more people-oriented, fully respecting the emotional needs and diverse aesthetic needs of (children) young people. The Guoxin Haitian Center is a new generation of core business districts in Qingdao, which is gradually breaking the traditional old business district. This area has the conditions to create innovative buildings full of a sense of technology, fashion, and future. It has the responsibility and obligation to attract the younger generation so that the area will continue to prosper and the city will be more dynamic.

3.4 Case 2 of Architectural Art Style Transfer: The Museum of the Former German Governor’s House

The Museum of the Former German Governor’s House was first built in 1905 and completed by the end of 1907. It is the most representative German-style building in Qingdao. This is also a classic building that perfectly integrates artistic beauty, historical value and cultural connotation. As the leader of Qingdao's European architecture, this castle-style building that used to be the official residence of the German governor looks ancient and gorgeous because of its European royal style.

Fig. 5(b) is a conceptual illustration created for a magazine drawn by Ni Chuanjing, a well-known illustrator from Hong Kong, China. In this illustration, Victo draws on the pattern of Chinese blue and white porcelain and uses smooth and elegant lines and fresh and elegant blue to construct a pair of blue and white porcelain plates printed with ancient Chinese architecture, which has a unique and strong visual effect. Ni Chuanjing's illustrations contain strong and rich Chinese elements. These wonderful oriental elements are also imprints of her life when she was growing up in Hong Kong and are the result of the subtle influence of Chinese culture. Fig. 5(c) is the work “Monster Quilt” by the famous American illustrator Jillian Tamaki. This illustration is realized by means of embroidery. Jillian Tamaki endows imaginative and interesting illustration concepts between gorgeous embroidery threads and exquisite stitches and perfectly combines exquisite embroidery and illustration art in a “clumsy and enthusiastic” way. Fig. 5(d) is the playful work “Strawberry” created by German illustrator Michael Soaw. Michael Soaw's illustrations are known for their fantastical, surreal style. In the picture, the author uses fine and realistic techniques, warm and soft colors and subtle light and shadow to paint a picture full of childishness and a sense of humor, which is a work suitable for all ages.

With the accelerated development of Qingdao's urbanization and the upgrading of its industrial structure, many old buildings have lost their original purpose, an increasing number of old buildings have been demolished, remodeled, and different types of old buildings are moving in different directions. Through deep learning, old buildings and their surrounding natural and cultural environments may be rejuvenated in the future, achieving a reasonable and humane balance between old and new buildings. Fig. 5(e), 5(f), and 5(g) are the results of the fusion of different styles of illustration art works on the basis of the original architectural structure design through deep learning. Fig. 5(e) combines illustrations of the beauty of traditional Chinese art with typical old European buildings. Among the lush green hills, it presents a unique visual beauty and temperament of mountains and seas, as if inviting people to walk into a place full of the art museums with mysterious oriental charm. The buildings in Fig. 5(f) combines the embroidery texture of the illustration with the interesting illustration characters. The brilliant colors are more prosperous in the green ocean, symbolizing the vitality of life full of hope. In Fig. 5(g), the building presents the appearance of a strawberry. The overall style is full of fun and a sense of humor. The soft red buildings contrast with the surrounding green mountains and forests. Buildings and nature coexist harmoniously. It is unique and artistic. Green trees, blue sea, and blue sky bring infinite brilliance to the old town.

Fig. 5.

Artistic style transfer effect picture of the Museum of the Former German Governor’s House. Among them, (a) the picture of the Museum of the Former German Governor’s House, (b) the illustration work “conceptual illustration” by Victo Ngai, (c) the illustration work “Monster Quilt” by Jillian Tamaki, (d) the illustration work "Strawberry" by Michael Sowa, (e) the effect image of style transfer based on Fig. 5(b), (f) the effect image of style transfer based on Fig. 5(c), and (g) the effect image of style transfer based on Fig. 5(d).

3.5 Case 3 of Architectural Art Style Transfer: The TAG Art Museum

TAG Art Museum is located in “the seaside resort with the richest natural resources on the Chinese coastline” - Qingdao Phoenix Island National Tourist Resort. It is Jean Nouvel's first art museum architectural work in China. It will be officially opened to the public in 2021. The building and garden of the art museum echo the ecological concept of community art. Large-scale windows are opened to introduce natural landscapes, moving to different scenes, blurring the boundaries between indoors and outdoors, and letting art scatter in the art garden by the sea. Based on the locality, the TAG Art Museum emphasizes connection and fully respects the site spirit of the base, fully integrating the architecture with the human environment and the natural environment.

Fig. 6(b) is the illustration work "Birds of Paradise" by the author of this article, Yue Wang, which has won the Excellence Award from the New York Society of Illustrators. The work depicts many birds of different shapes facing the same direction against the background of billowing clouds, waiting for the opportunity to take off. The work is finished with acrylic drawing, with strong colors and delicate brushwork, which is full of meaning. Fig. 6(c) is from the picture book “The real boat,” written by Victoria Semykina, who is currently living in Italy. Victoria Semykina is good at drawing illustrations in the form of collages. The painting style is free and easy, and the lines have a sense of sketching. Combined with comprehensive materials of different textures, the shapes have exaggerated expressions. The boat in the illustration does unstick to the realistic perspective relationship but makes bold and interesting deformations, and a clever creative structure can be found in the seemingly random picture. In Fig. 6(d), “Mixc World Launch” is a commercial illustration by Victon Ngai. The lines of this work are clean and smooth, and the colors are gorgeous and dreamy. The complicated and uncomplicated painting style and fantasy elements make the whole picture span the constraints of time and space. It skillfully blends reality and dreams. It gives the audience unlimited fantasy space, making people deeply immersed in it and conceives possible stories.

Fig. 6.

Artistic style transfer rendering of TAG Art Museum in Qingdao. Among them, (a) the TAG Art Museum image, (b) Yue Wang's illustration “Birds of Paradise,” (c) the real boat by Victoria Semykina, (d) Victon ngai's illustration “Mixc World Launch,” (e) the effect image of style transfer based on Fig. 6(b), (f) the effect image of style transfer based on Fig. 6(c), and (g) the effect image of style transfer based on Fig. 6(d).

The TAG Art Museum is located on the bank of Tangdao Bay in the Qingdao West Coast New District, with flat terrain, beautiful scenery, and a pleasant environment. Relying on the Xihai Art Bay community, the TAG Art Museum plays its core role in gathering art resources. In the future, a relatively complete art community system will gradually be formed here, this will additionally establish a robust artistic ecological foundation for the museum's future development. Fig. 6(e), 6(f), and 6(g) show the transformed building appearances based on illustrations art based on the original architectural structure design through deep learning. Fig. 6(e) integrates the shapes of birds in nature into the architectural design. Against the backdrop of the blue seascape, the magical shapes and brilliant colors seem to be a real paradise for birds. Artistic expressions strengthen the image of the museum. Artistic attributes and environmental protection concept. The architectural design in Fig. 6(f) has the artistic effect of collage, the colors are fresh and elegant, reflecting the strong charm of the ocean, and the architectural layers are deep and distinct, full of a strong sense of rhythm. The architectural colors in Fig. 6(g) are in sharp contrast with the blue of the sea. On the premise of fully respecting the surrounding environment and site style, it highlights the visual enjoyment and artistic vitality of the building.

3.6 The Architectural Aesthetic Preferences of Children

To determine the architectural aesthetic preferences of children, the following aspects should be considered.

· Consider the age range of children: Children of different age groups have different aesthetic preferences for architecture. For example, preschoolers may prefer buildings with bright colors, simple shapes, and a warm and bright atmosphere, while teenagers may prefer novel, creative, and distinctive architecture.

· Observe children's behavior and reactions: By observing children's behavior and reactions in different architectural environments, some preliminary conclusions can be drawn. For example, if children frequently stay or enjoy observing a certain building, then this building may meet their aesthetic preferences.

· Communicate with children: Directly communicating with children and asking for their opinions and feelings about different buildings can help to better understand their aesthetic preferences. By engaging them in activities such as drawing or building with blocks, they can express their thoughts and ideas.

· Refer to children's culture: Architectural elements in children's culture are often liked by children. Referring to architectural elements that appear in children's culture, such as cartoon characters and fairy tales, can serve as a reference for designing buildings. By considering the above aspects comprehensively, it is possible to accurately understand the architectural aesthetic preferences of children.

4. Conclusion

The advancement of computer technology has led to the innovative integration of deep learning models in various architectural design projects. The strong logic and rational ways of thinking about architectural design form many architectural standards. Because the world in the eyes of children is different from that of adults, traditional standards often cannot be the most appropriate method in the design process of children-related urban buildings. Children's architectural design can combine children's psychological characteristics and aesthetic needs, provide children with a design style with game, exploration, aesthetic, and learning functions, and help children enhance their imagination and creativity. Architecture is not isolated; its cultural, conceptual, and aesthetic attributes are perpetually shaped by an array of visual arts, encompassing film, photography, sculpture, fashion, illustration, industrial design, and beyond. Among them, children's illustrators often stand from the perspective of children and successfully attract children's attention and love with bold and bright colors and unique and vivid shapes. Architectural design's culture, concepts, and aesthetics are in constant interplay with various visual and artistic disciplines, catering to diverse aesthetic audiences with distinct designs. The integration of architectural design and deep learning serves to enhance traditional workflows. Based on the artistic perspective of children's illustrations and picture books, this paper uses the neural style network to artistically stylize the target building using classic children's illustrations and images in children's picture books while preserving the distinctive architectural characteristics of the content images. The work of this paper explores the possibility of the integration of architectural design style and children's illustrations and it offers a fresh perspective and valuable support for the artistic design of contemporary urban structures.