# Cybernetical Intelligence

## Engineering Cybernetics with Machine Intelligence

1. Edition October 2023

432 Pages, Hardcover*Professional Book*

**978-1-394-21748-9**

CYBERNETICAL INTELLIGENCE

Highly comprehensive, detailed, and up-to-date overview of artificial intelligence and cybernetics, with practical examples and supplementary learning resources

Cybernetical Intelligence: Engineering Cybernetics with Machine Intelligence is a comprehensive guide to the field of cybernetics and neural networks, as well as the mathematical foundations of these technologies. The book provides a detailed explanation of various types of neural networks, including feedforward networks, recurrent neural networks, and convolutional neural networks as well as their applications to different real-world problems. This groundbreaking book presents a pioneering exploration of machine learning within the framework of cybernetics. It marks a significant milestone in the field's history, as it is the first book to describe the development of machine learning from a cybernetics perspective. The introduction of the concept of "Cybernetical Intelligence" and the generation of new terminology within this context propel new lines of thought in the historical development of artificial intelligence. With its profound implications and contributions, this book holds immense importance and is poised to become a definitive resource for scholars and researchers in this field of study.

Each chapter is specifically designed to introduce the theory with several examples. This comprehensive book includes exercise questions at the end of each chapter, providing readers with valuable opportunities to apply and strengthen their understanding of cybernetical intelligence. To further support the learning journey, solutions to these questions are readily accessible on the book's companion site. Additionally, the companion site offers programming practice exercises and assignments, enabling readers to delve deeper into the practical aspects of the subject matter.

Cybernetical Intelligence includes information on:

* The history and development of cybernetics and its influence on the development of neural networks

* Developments and innovations in artificial intelligence and machine learning, such as deep reinforcement learning, generative adversarial networks, and transfer learning

* Mathematical foundations of artificial intelligence and cybernetics, including linear algebra, calculus, and probability theory

* Ethical implications of artificial intelligence and cybernetics as well as responsible and transparent development and deployment of AI systems

Presenting a highly detailed and comprehensive overview of the field, with modern developments thoroughly discussed, Cybernetical Intelligence is an essential textbook that helps students make connections with real-life engineering problems by providing both theory and practice, along with a myriad of helpful learning aids.

About the Author xix

About the Companion Website xxi

1 Artificial Intelligence and Cybernetical Learning 1

1.1 Artificial Intelligence Initiative 1

1.2 Intelligent Automation Initiative 4

1.2.1 Benefits of IAI 5

1.3 Artificial Intelligence Versus Intelligent Automation 5

1.3.1 Process Discovery 6

1.3.2 Optimization 7

1.3.3 Analytics and Insight 8

1.4 The Fourth Industrial Revolution and Artificial Intelligence 9

1.4.1 Artificial Narrow Intelligence 10

1.4.2 Artificial General Intelligence 12

1.4.3 Artificial Super Intelligence 13

1.5 Pattern Analysis and Cognitive Learning 14

1.5.1 Machine Learning 15

1.5.1.1 Parametric Algorithms 16

1.5.1.2 Nonparametric Algorithms 17

1.5.2 Deep Learning 20

1.5.2.1 Convolutional Neural Networks in Advancing Artificial Intelligence 21

1.5.2.2 Future Advancement in Deep Learning 22

1.5.3 Cybernetical Learning 23

1.6 Cybernetical Artificial Intelligence 24

1.6.1 Artificial Intelligence Control Theory 24

1.6.2 Information Theory 26

1.6.3 Cybernetic Systems 27

1.7 Cybernetical Intelligence Definition 28

1.8 The Future of Cybernetical Intelligence 30

Summary 32

Exercise Questions 32

Further Reading 33

2 Cybernetical Intelligent Control 35

2.1 Control Theory and Feedback Control Systems 35

2.2 Maxwell's Analysis of Governors 37

2.3 Harold Black 39

2.4 Nyquist and Bode 40

2.5 Stafford Beer 42

2.5.1 Cybernetic Control 42

2.5.2 Viable Systems Model 42

2.5.3 Cybernetics Models of Management 43

2.6 James Lovelock 43

2.6.1 Cybernetic Approach to Ecosystems 43

2.6.2 Gaia Hypothesis 44

2.7 Macy Conference 44

2.8 McCulloch-Pitts 45

2.9 John von Neumann 47

2.9.1 Discussions on Self-Replicating Machines 47

2.9.2 Discussions on Machine Learning 48

Summary 48

Exercise Questions 49

Further Reading 50

3 The Basics of Perceptron 51

3.1 The Analogy of Biological and Artificial Neurons 51

3.1.1 Biological Neurons and Neurodynamics 52

3.1.2 The Structure of Neural Network 53

3.1.3 Encoding and Decoding 56

3.2 Perception and Multilayer Perceptron 57

3.2.1 Back Propagation Neural Network 59

3.2.2 Derivative Equations for Backpropagation 59

3.3 Activation Function 61

3.3.1 Sigmoid Activation Function 61

3.3.2 Hyperbolic Tangent Activation Function 62

3.3.3 Rectified Linear Unit Activation Function 62

3.3.4 Linear Activation Function 64

Summary 65

Exercise Questions 67

Further Reading 67

4 The Structure of Neural Network 69

4.1 Layers in Neural Network 69

4.1.1 Input Layer 69

4.1.2 Hidden Layer 70

4.1.3 Neurons 70

4.1.4 Weights and Biases 71

4.1.5 Forward Propagation 72

4.1.6 Backpropagation 72

4.2 Perceptron and Multilayer Perceptron 73

4.3 Recurrent Neural Network 75

4.3.1 Long Short-Term Memory 76

4.4 Markov Neural Networks 77

4.4.1 State Transition Function 77

4.4.2 Observation Function 78

4.4.3 Policy Function 78

4.4.4 Loss Function 78

4.5 Generative Adversarial Network 78

Summary 79

Exercise Questions 80

Further Reading 81

5 Backpropagation Neural Network 83

5.1 Backpropagation Neural Network 83

5.1.1 Forward Propagation 85

5.2 Gradient Descent 85

5.2.1 Loss Function 85

5.2.2 Parameters in Gradient Descent 88

5.2.3 Gradient in Gradient Descent 88

5.2.4 Learning Rate in Gradient Descent 89

5.2.5 Update Rule in Gradient Descent 89

5.3 Stopping Criteria 89

5.3.1 Convergence and Stopping Criteria 90

5.3.2 Local Minimum and Global Minimum 91

5.4 Resampling Methods 91

5.4.1 Cross-Validation 93

5.4.2 Bootstrapping 93

5.4.3 Monte Carlo Cross-Validation 94

5.5 Optimizers in Neural Network 94

5.5.1 Stochastic Gradient Descent 94

5.5.2 Root Mean Square Propagation 96

5.5.3 Adaptive Moment Estimation 96

5.5.4 AdaMax 97

5.5.5 Momentum Optimization 97

Summary 97

Exercise Questions 99

Further Reading 100

6 Application of Neural Network in Learning and Recognition 101

6.1 Applying Backpropagation to Shape Recognition 101

6.2 Softmax Regression 105

6.3 K-Binary Classifier 107

6.4 Relational Learning via Neural Network 108

6.4.1 Graph Neural Network 109

6.4.2 Graph Convolutional Network 111

6.5 Cybernetics Using Neural Network 112

6.6 Structure of Neural Network for Image Processing 115

6.7 Transformer Networks 116

6.8 Attention Mechanisms 116

6.9 Graph Neural Networks 117

6.10 Transfer Learning 118

6.11 Generalization of Neural Networks 119

6.12 Performance Measures 120

6.12.1 Confusion Matrix 120

6.12.2 Receiver Operating Characteristic 121

6.12.3 Area Under the ROC Curve 122

Summary 123

Exercise Questions 123

Further Reading 124

7 Competitive Learning and Self-Organizing Map 125

7.1 Principal of Competitive Learning 125

7.1.1 Step 1: Normalized Input Vector 128

7.1.2 Step 2: Find the Winning Neuron 128

7.1.3 Step 3: Adjust the Network Weight Vector and Output Results 129

7.2 Basic Structure of Self-Organizing Map 129

7.2.1 Properties Self-Organizing Map 130

7.3 Self-Organizing Mapping Neural Network Algorithm 131

7.3.1 Step 1: Initialize Parameter 132

7.3.2 Step 2: Select Inputs and Determine Winning Nodes 132

7.3.3 Step 3: Affect Neighboring Neurons 132

7.3.4 Step 4: Adjust Weights 133

7.3.5 Step 5: Judging the End Condition 133

7.4 Growing Self-Organizing Map 133

7.5 Time Adaptive Self-Organizing Map 136

7.5.1 TASOM-Based Algorithms for Real Applications 138

7.6 Oriented and Scalable Map 139

7.7 Generative Topographic Map 141

Summary 145

Exercise Questions 146

Further Reading 147

8 Support Vector Machine 149

8.1 The Definition of Data Clustering 149

8.2 Support Vector and Margin 152

8.3 Kernel Function 155

8.3.1 Linear Kernel 155

8.3.2 Polynomial Kernel 156

8.3.3 Radial Basis Function 157

8.3.4 Laplace Kernel 159

8.3.5 Sigmoid Kernel 159

8.4 Linear and Nonlinear Support Vector Machine 160

8.5 Hard Margin and Soft Margin in Support Vector Machine 164

8.6 I/O of Support Vector Machine 167

8.6.1 Training Data 167

8.6.2 Feature Matrix and Label Vector 168

8.7 Hyperparameters of Support Vector Machine 169

8.7.1 The C Hyperparameter 169

8.7.2 Kernel Coefficient 169

8.7.3 Class Weights 170

8.7.4 Convergence Criteria 170

8.7.5 Regularization 171

8.8 Application of Support Vector Machine 171

8.8.1 Classification 171

8.8.2 Regression 173

8.8.3 Image Classification 173

8.8.4 Text Classification 174

Summary 174

Exercise Questions 175

Further Reading 176

9 Bio-Inspired Cybernetical Intelligence 177

9.1 Genetic Algorithm 178

9.2 Ant Colony Optimization 181

9.3 Bees Algorithm 184

9.4 Artificial Bee Colony Algorithm 186

9.5 Cuckoo Search 189

9.6 Particle Swarm Optimization 193

9.7 Bacterial Foraging Optimization 196

9.8 Gray Wolf Optimizer 197

9.9 Firefly Algorithm 199

Summary 200

Exercise Questions 201

Further Reading 202

10 Life-Inspired Machine Intelligence and Cybernetics 203

10.1 Multi-Agent AI Systems 203

10.1.1 Game Theory 205

10.1.2 Distributed Multi-Agent Systems 206

10.1.3 Multi-Agent Reinforcement Learning 207

10.1.4 Evolutionary Computation and Multi-Agent Systems 209

10.2 Cellular Automata 211

10.3 Discrete Element Method 212

10.3.1 Particle-Based Simulation of Biological Cells and Tissues 214

10.3.2 Simulation of Microbial Communities and Their Interactions 215

10.3.3 Discrete Element Method-Based Modeling of Biological Fluids and Soft Materials 216

10.4 Smoothed Particle Hydrodynamics 218

10.4.1 SPH-Based Simulations of Biomimetic Fluid Dynamic 219

10.4.2 SPH-Based Simulations of Bio-Inspired Engineering Applications 220

Summary 221

Exercise Questions 222

Further Reading 223

11 Revisiting Cybernetics and Relation to Cybernetical Intelligence 225

11.1 The Concept and Development of Cybernetics 225

11.1.1 Attributes of Control Concepts 225

11.1.2 Research Objects and Characteristics of Cybernetics 226

11.1.3 Development of Cybernetical Intelligence 227

11.2 The Fundamental Ideas of Cybernetics 227

11.2.1 System Idea 227

11.2.2 Information Idea 229

11.2.3 Behavioral Idea 230

11.2.4 Cybernetical Intelligence Neural Network 231

11.3 Cybernetic Expansion into Other Fields of Research 234

11.3.1 Social Cybernetics 234

11.3.2 Internal Control-Related Theories 237

11.3.3 Software Control Theory 237

11.3.4 Perceptual Cybernetics 238

11.4 Practical Application of Cybernetics 240

11.4.1 Research on the Control Mechanism of Neural Networks 240

11.4.2 Balance Between Internal Control and Management Power Relations 240

11.4.3 Software Markov Adaptive Testing Strategy 242

11.4.4 Task Analysis Model 244

Summary 245

Exercise Questions 246

Further Reading 247

12 Turing Machine 249

12.1 Behavior of a Turing Machine 250

12.1.1 Computing with Turing Machines 251

12.2 Basic Operations of a Turing Machine 252

12.2.1 Reading and Writing to the Tape 253

12.2.2 Moving the Tape Head 254

12.2.3 Changing States 254

12.3 Interchangeability of Program and Behavior 255

12.4 Computability Theory 256

12.4.1 Complexity Theory 257

12.5 Automata Theory 258

12.6 Philosophical Issues Related to Turing Machines 259

12.7 Human and Machine Computations 260

12.8 Historical Models of Computability 261

12.9 Recursive Functions 262

12.10 Turing Machine and Intelligent Control 263

Summary 264

Exercise Questions 265

Further Reading 265

13 Entropy Concepts in Machine Intelligence 267

13.1 Relative Entropy of Distributions 268

13.2 Relative Entropy and Mutual Information 268

13.3 Entropy in Performance Evaluation 269

13.4 Cross-Entropy Softmax 271

13.5 Calculating Cross-Entropy 272

13.6 Cross-Entropy as a Loss Function 273

13.7 Cross-Entropy and Log Loss 274

13.8 Application of Entropy in Intelligent Control 275

13.8.1 Entropy-Based Control 275

13.8.2 Fuzzy Entropy 276

13.8.3 Entropy-Based Control Strategies 277

13.8.4 Entropy-Based Decision-Making 278

Summary 279

Exercise Questions 279

Further Reading 280

14 Sampling Methods in Cybernetical Intelligence 283

14.1 Introduction to Sampling Methods 283

14.2 Basic Sampling Algorithms 284

14.2.1 Importance of Sampling Methods in Machine Intelligence 286

14.3 Machine Learning Sampling Methods 287

14.3.1 Random Oversampling 288

14.3.2 Random Undersampling 290

14.3.3 Synthetic Minority Oversampling Technique 290

14.3.4 Adaptive Synthetic Sampling 292

14.4 Advantages and Disadvantages of Machine Learning Sampling Methods 293

14.5 Advanced Sampling Methods in Cybernetical Intelligence 294

14.5.1 Ensemble Sampling Method 295

14.5.2 Active Learning 297

14.5.3 Bayesian Optimization in Sampling 299

14.6 Applications of Sampling Methods in Cybernetical Intelligence 302

14.6.1 Image Processing and Computer Vision 302

14.6.2 Natural Language Processing 304

14.6.3 Robotics and Autonomous Systems 307

14.7 Challenges and Future Directions 308

14.8 Challenges and Limitations of Sampling Methods 309

14.9 Emerging Trends and Innovations in Sampling Methods 309

Summary 310

Exercise Questions 311

Further Reading 312

15 Dynamic System Control 313

15.1 Linear Systems 314

15.2 Nonlinear System 316

15.3 Stability Theory 318

15.4 Observability and Identification 320

15.5 Controllability and Stabilizability 321

15.6 Optimal Control 323

15.7 Linear Quadratic Regulator Theory 324

15.8 Time-Optimal Control 326

15.9 Stochastic Systems with Applications 328

15.9.1 Stochastic System in Control Systems 329

15.9.2 Stochastic System in Robotics and Automation 329

15.9.3 Stochastic System in Neural Networks 330

Summary 331

Exercise Questions 331

Further Reading 332

16 Deep Learning 333

16.1 Neural Network Models in Deep Learning 335

16.2 Methods of Deep Learning 336

16.2.1 Convolutional Neural Networks 337

16.2.2 Recurrent Neural Networks 340

16.2.3 Generative Adversarial Networks 342

16.2.4 Deep Learning Based Image Segmentation Models 345

16.2.5 Variational Auto Encoders 348

16.2.6 Transformer Models 350

16.2.7 Attention-Based Models 352

16.2.8 Meta-Learning Models 354

16.2.9 Capsule Networks 357

16.3 Deep Learning Frameworks 358

16.4 Applications of Deep Learning 359

16.4.1 Object Detection 360

16.4.2 Intelligent Power Systems 361

16.4.3 Intelligent Control 362

Summary 362

Exercise Questions 363

References 364

Further Reading 365

17 Neural Architecture Search 367

17.1 Neural Architecture Search and Neural Network 369

17.2 Reinforcement Learning-Based Neural Architecture Search 371

17.3 Evolutionary Algorithms-Based Neural Architecture Search 374

17.4 Bayesian Optimization-Based Neural Architecture Search 376

17.5 Gradient-Based Neural Architecture Search 378

17.6 One-shot Neural Architecture Search 379

17.7 Meta-Learning-Based Neural Architecture Search 381

17.8 Neural Architecture Search for Specific Domains 383

17.8.1 Cybernetical Intelligent Systems: Neural Architecture Search in Real-World 384

17.8.2 Neural Architecture Search for Specific Cybernetical Control Tasks 385

17.8.3 Neural Architecture Search for Cybernetical Intelligent Systems in Real-World 386

17.8.4 Neural Architecture Search for Adaptive Cybernetical Intelligent Systems 388

17.9 Comparison of Different Neural Architecture Search Approaches 389

Summary 391

Exercise Questions 391

Further Reading 392

Final Notes on Cybernetical Intelligence 393

Index 399