Artificial Intelligence for Art Creation and Understanding (Multimedia Computing, Communication and Intelligence)

AI-Generated Content (AIGC) is a revolutionary engine for digital content generation. In the area of art, AI has achieved remarkable advancements. AI is capable of not only creating paintings or music comparable to human masterpieces, but it also understands and appreciates artwork. For professionals and amateurs, AI is an enabling tool and an opportunity to enjoy a new world of art. This book aims to present the state-of-the-art AI technologies for art creation, understanding, and evaluation. The contents include a survey on cross-modal generation of visual and auditory content, explainable AI and music, AI-enabled robotic theater for Chinese folk art, AI for ancient Chinese music restoration and reproduction, AI for brainwave opera, artistic text style transfer, data-driven automatic choreography, Human-AI collaborative sketching, personalized music recommendation and generation based on emotion and memory (MemoMusic), understanding music and emotion from the brain, music question answering, emotional quality evaluation for generated music, and AI for image aesthetic evaluation. The key features of the book are as follows: AI for Art is a fascinating cross-disciplinary field for the academic community as well as the public. Each chapter is an independent interesting topic, which provides an entry for corresponding readers. It presents SOTA AI technologies for art creation and understanding. The artistry and appreciation of the book is wide-ranging – for example, the combination of AI with traditional Chinese art. This book is dedicated to the international cross-disciplinary AI Art community: professors, students, researchers, and engineers from AI (machine learning, computer vision, multimedia computing, affective computing, robotics, etc.), art (painting, music, dance, fashion, design, etc.), cognitive science, and psychology. General audiences can also benefit from this book. Cover Half Title Series Page Title Page Copyright Page Dedication Table of Contents About the Editor Contributors Chapter 1: Explainable AI and Music 1.1 Introduction 1.2 Related Work 1.2.1 eXplainable AI and Generative Music 1.3 Case Study: Explaining Latent Spaces for Generative AI Music 1.3.1 Visualization 1.3.2 Real-Time Interaction 1.4 XAI for Music Challenges 1.4.1 Challenge: The Nature of Explanation 1.4.2 Challenge: AI Models, Features, and Training Data 1.4.3 Challenge: User Centered Design for XAI 1.4.4 Challenge: Interaction Design of XAI 1.5 Conclusions Acknowledgements Funding Notes References Chapter 2: AI-Enabled Robotic Theaters for Chinese Folk Art 2.1 Introduction: Background and Motivations 2.2 Robotic Theaters 2.3 Blending Robotic Theaters with Chinese Folk Art 2.3.1 Chinese Orchestra 2.3.2 Chinese Historical Theater 2.3.3 Xiàngshēng Theater 2.4 Creation and Implementation 2.4.1 Creation Workflow 2.4.1.1 Defining the Theme 2.4.1.2 Overall Performance Design 2.4.1.3 Character Design 2.4.1.4 Motion Design 2.4.1.5 Motion Matching 2.4.2 Technical Framework 2.4.2.1 Overview of the Technical Framework 2.4.2.2 Robot Actor System 2.4.2.3 Creation and Choreography System 2.4.2.4 Stage Performance System 2.4.3 Integration and Deployment 2.4.3.1 Venue, Space, and Infrastructure 2.4.3.2 Theater Integration 2.4.3.3 Performance Control 2.4.3.4 Safety and Robustness 2.5 Adopting AI Technology in Robotic Theater Creation 2.5.1 Overview of AI-Assisted Content Authoring 2.5.1.1 Music Composition 2.5.1.2 Lip Sync Processing 2.5.1.3 Motion Capture 2.5.2 Music Composing 2.5.2.1 MIDI Translation and Optimization 2.5.2.2 Real-Time Composition 2.5.2.3 Integrated Performance 2.5.3 Lip Synchronization 2.5.3.1 AI-Driven Robot Lip Synchronization 2.5.3.2 Sentence Breaking in Robotic Speech 2.5.3.3 Intelligent Speech Feedback in Improvised Performances 2.5.4 Motion and Facial Capture in Robotic Theater Creation 2.5.4.1 Facial Capture 2.5.4.2 Motion Capture 2.5.4.3 Deep Learning Model Training 2.5.4.4 Data Mapping 2.5.4.5 Animation Generation 2.5.4.6 Robot Driving 2.6 Discussion 2.6.1 Challenges and Opportunities 2.6.2 AI-Artist Collaboration 2.6.3 AI-Enabled Real-Time Interaction 2.6.3.1 Language Understanding 2.6.3.2 Generating Responses 2.6.3.3 Text-to-Speech (TTS) 2.6.3.4 Synchronizing Lip Movements and Expressions 2.6.3.5 Driving the Robot 2.6.4 From AI-Enabled to AI-Participated 2.7 Concluding Remarks References Chapter 3: AIBO: Or How to Make a ‘Sicko’ Brainwave Opera 3.1 Overview: An AI Brainwave Opera 3.2 Introduction 3.3 Creating the AI for the Opera 3.4 Sentiment Analysis 3.5 Signal Routing 3.6 Discussion 3.7 Conclusion Appendix (Figures 3.5–3.10) References Chapter 4: Cross-Modal Generation of Visual and Auditory Content: A Survey 4.1 Introduction 4.2 Typical Challenges of Cross-Modal Generation 4.2.1 Representation 4.2.1.1 Text Representation 4.2.1.2 Image Representation 4.2.1.3 Video Representation 4.2.1.4 Audio Representation 4.2.1.5 Multimodal Representation 4.2.1.5.1 Text and Vision 4.2.1.5.2 Text and Audio 4.2.1.5.3 Visual and Audio 4.2.2 Alignment 4.2.2.1 Text and Vision Alignment 4.2.2.2 Text and Audio Alignment 4.2.2.3 Vision and Audio Alignment 4.3 Tasks of Cross-Modal Generation 4.3.1 Visual Content Generation 4.3.2 Auditory Content Generation 4.3.2.1 Text-to-Speech 4.3.2.2 Text-to-music 4.3.2.3 Vision-to-Audio 4.4 Multimodal Datasets 4.4.1 Multimodal Vision-Related Datasets 4.4.2 Multimodal Audio-Related Datasets 4.5 Conclusions Acknowledgements References Chapter 5: Artistic Text Style Transfer 5.1 Introduction 5.2 Patch-Based Supervised Text Effect Transfer 5.2.1 Characteristics of Text Effect 5.2.1.1 Robust Normalized Distance Estimation 5.2.1.2 Optimal Scale Posterior Probability Estimation 5.2.2 Text Effect Transfer 5.2.2.1 Objective Function 5.2.2.2 Appearance Term: Texture Style Transfer 5.2.2.3 Distribution Term: Spatial Style Transfer 5.2.2.4 Psycho-Visual Term: Naturalness Preservation 5.3 Patch-Based Unsupervised Artistic Text Style Transfer 5.3.1 Text Style Transfer from Arbitrary Style Images 5.3.1.1 Guidance Map Extraction 5.3.1.2 Structure Transfer 5.3.1.3 Texture Transfer 5.3.2 Visual-Textual Presentation Generation 5.3.2.1 Color Transfer 5.3.2.2 Position Estimation 5.3.2.3 Shape Embedding 5.4 Deep-Based Supervised Text Effect Transfer 5.4.1 Transfer Effect by Disentangling Text and Effect 5.4.1.1 Network Architecture and Loss Function 5.4.1.2 One-Reference Text Effect Transfer 5.4.1.3 Semi-Supervised Text Effect Transfer 5.4.1.4 Joint Font Style and Text Effect Transfer 5.4.2 Transfer Effect with Decorative Elements 5.4.2.1 Decorative Element Segmentation 5.4.2.2 Text Effect Transfer 5.4.2.3 Structure-Based Decor Recomposition 5.5 Deep-Based Unsupervised Text Style Transfer 5.5.1 Static Text Style Transfer 5.5.1.1 Bidirectional Structure Transfer 5.5.1.2 Texture Transfer 5.5.2 Dynamic Text Style Transfer 5.5.2.1 Backward Structure Transfer with Frame Fusion 5.5.2.2 Frame Initialization Glyph Network 5.5.2.3 Frame Prediction Glyph Network 5.6 Text Effect Dataset and Evaluation 5.6.1 TE141K Dataset 5.6.2 Performance Evaluation 5.7 Conclusion References Chapter 6: Data-Driven Automatic Choreography 6.1 Introduction: Background of Automatic Choreography 6.1.1 Auto Choreography Based on Hand-Selected Movement Features 6.1.2 Auto Choreography Based on Classic Machine Learning 6.1.3 Auto Choreography Based on Deep Learning 6.2 Fundamental Knowledge of Automatic Choreography 6.3 Data-Driven Automatic Choreography 6.3.1 Dataset and Feature Extraction 6.3.1.1 Data Acquisition 6.3.1.2 Data Representation and Feature Extraction 6.3.2 Dance Movements Generation 6.3.2.1 Generative Model 6.3.2.1.1 LSTM 6.3.2.1.2 MDN 6.3.2.1.3 Motion Generative Model 6.3.2.2 Parameter Control 6.3.2.3 Motion Screening 6.3.3 Multi-Level Motion and Music Feature Matching 6.3.3.1 Global Music Feature Extraction 6.3.3.1.1 BPM 6.3.3.1.2 Note Density 6.3.3.2 Global Feature Matching 6.3.3.2.1 Motion Feature Extraction 6.3.3.2.2 Global Feature Matching 6.3.3.3 Local Feature Matching Considering Rhythm and Intensity Components 6.3.3.3.1 Music Feature Extraction 6.3.3.3.2 Motion Feature Extraction 6.3.3.3.3 Local Feature Matching 6.3.3.4 Transition Motion Generation 6.3.4 Experiments and Discussions 6.3.4.1 Qualitative Experiments 6.3.4.1.1 Parameter Control Method Experiment 6.3.4.1.2 Motion Screening Algorithm Experiment 6.3.4.2 Quantitative Experiments 6.4 Chapter Conclusion References Chapter 7: Toward Human-AI Collaborative Sketching 7.1 Introduction 7.2 Related Work 7.2.1 Non-Photorealistic Rendering 7.2.2 Sketch Animation 7.2.3 Style-Driven Image Generation and Embedding 7.3 Methodology 7.3.1 Learning a Latent Style Space 7.3.2 Encoding Geometry with GeoNet E G 7.3.3 Encoding Drawing with StyleNet E S 7.3.4 StyleGAN-Based Image Generator G 7.3.5 Learning for Drawing Generation 7.3.6 Reconstruction Loss and Perceptual Loss 7.3.6.1 Sparsity Loss 7.3.6.2 Interpolation Loss 7.3.6.3 Strokeness Loss 7.3.6.4 Adversarial Loss 7.3.7 Input Drawing Embedding 7.4 Implementation Details 7.4.1 Dataset Preparation 7.4.2 Data Collection for Evaluation 7.4.3 Network Architecture and Training Strategy 7.5 Experiments 7.5.1 Evaluation 7.5.1.1 Latent Style Space Embedding 7.5.1.2 Disentanglement between Style and Geometry 7.5.1.3 Style Interpolation 7.5.2 Ablation Study 7.5.3 Generalization 7.5.3.1 Unseen Frames 7.5.3.2 Unseen Animation 7.6 Applications 7.7 Conclusion 7.7.1 Limitations and Future Work References Chapter 8: MemoMusic: A Personalized Music Recommendation and Generation Framework Based on Emotion and Memory 8.1 Introduction 8.2 Related Work 8.2.1 Personalized Music Recommendation 8.2.2 Deep Learning-Based Music Generation 8.2.2.1 Music Generation Models 8.2.2.2 Conditional Music Generation 8.2.2.2.1 Music Generation Based on Low-Level Conditions 8.2.2.2.2 Music Generation Based on High-Level Conditions 8.2.3 Music Theory 8.3 The Framework and Algorithms of MemoMusic 8.3.1 The Framework of MemoMusic 8.3.2 Music Recommendation Based on Emotion, Memory, and Context 8.3.3 Music Generation Based on Emotion and Music Theory 8.3.3.1 Music Generation Model 8.3.3.2 Music Representation 8.3.3.3 Conditional MIDI Music Generation 8.4 The System of MemoMusic 8.4.1 The Interface of MemoMusic 8.4.2 The Database of MemoMusic 8.5 Experiment and Analysis 8.5.1 Dataset 8.5.2 Experiment Description 8.5.3 Experimental Results 8.6 Conclusion Notes References Chapter 9: Algorithmic Composition Techniques for Ancient Chinese Music Restoration and Reproduction: A Melody Generator Approach 9.1 Introduction 9.2 Computer Music Generation and Strategy of Musical Representation 9.3 Understanding Music Form, Morphology, and Its Cultural Significance 9.4 Baban Model 9.5 Database and the Encoding Process for Preprocessing 9.6 Training the Network and Melody Generation 9.7 Conclusion and Future Development Funding References Chapter 10: Understanding Music and Emotion from the Brain 10.1 Background 10.2 Cognitive Experiment Design 10.2.1 Music Listening Cognitive Experiment 10.2.2 Short-Time Sound Cognitive Experiment 10.3 Automatic Acoustic Event Detection 10.4 Feature Similarity Analysis 10.4.1 Representational Similarity Analysis 10.4.2 The Construction of Dissimilarity Matrix 10.5 Temporal Dynamics Analysis 10.5.1 Microstate Segmentation 10.5.2 Microstates Template Exploration 10.5.3 Microstates Back-Fitting 10.5.4 The Proposed Dual Threshold-Based AAHC 10.5.5 Microstate Class Spatial Topographies 10.6 Music Emotion Recognition 10.6.1 Music Feature Extraction 10.6.2 EEG Feature Extraction 10.6.3 Sparse Canonical Correlation Analysis 10.6.4 Experimental Process 10.7 Conclusion References Chapter 11: Music Question Answering: Cognize and Perceive Music 11.1 Introduction 11.2 Related Work 11.2.1 Music Perception 11.2.2 Question Answering 11.3 The Music Question Answering Dataset 11.3.1 Production of MQA Dataset 11.3.2 Dataset Analysis 11.4 Methodology 11.4.1 Conv-LSTM 11.4.2 Musicnn-LSTM 11.4.3 Musicnn-MALiMo 11.5 Experiments 11.6 Conclusion and Prospect References Chapter 12: Emotional Quality Evaluation for Generated Music 12.1 Background 12.2 Music Generation Methods 12.2.1 Structural-Property-Based Approach 12.2.2 Psychology-Based Approach 12.2.3 Data-Driven Approach 12.3 Emotional Music Generation Methods 12.3.1 Representative Emotion Model 12.3.2 Current Emotional Music Generation Networks 12.4 Quality Evaluation of Emotional Music Generation 12.4.1 Subjective Quality Evaluation 12.4.2 Objective Quality Evaluation 12.5 Objective Quality Evaluation for Generated Emotional Music Based on Emotion Recognition Model 12.5.1 Proposed Framework 12.5.2 Feature Selection 12.5.2.1 Analysis of Emotional Music Features Based on Chi-Square Test 12.5.2.2 Analysis of Emotional Music Features Based on Pearson’s Correlation Coefficient 12.5.3 Music Emotion Recognition Network 12.5.4 Experiments 12.5.4.1 Experiment on Emotional Music Feature Analysis 12.5.4.2 Experiment on Music Emotion Recognition 12.5.4.3 Experiment on Quality Evaluation of Emotional Music Generation 12.6 Conclusion References Chapter 13: A Deep Drift-Diffusion Model for Image Aesthetic Score Distribution Prediction 13.1 Introduction 13.2 Related Work 13.2.1 Computational Visual Aesthetics 13.2.2 Image Aesthetics Assessment 13.2.3 Score Distribution Prediction 13.2.4 Subjectiveness of Aesthetics 13.3 The Proposed DDD Model 13.3.1 Subjectiveness Analysis 13.3.2 The Deep Drift-Diffusion Model 13.4 Experiments 13.4.1 Datasets 13.4.1.1 AVA 13.4.2 Implementation Details 13.4.3 Evaluating Indicator 13.4.4 Score Distribution Prediction 13.5 Conclusions and Discussions Acknowledgements References