Chapter 2: The Threat of AI, a Double-Edged Sword

Artificial intelligence is still a tool. Objects themselves have no inherent good or evil; it is the mindset of the person using them that determines everything. The tool of AI is the same; depending on the heart of the user, it can become something harmful or something beneficial. Ultimately, whether AI is helpful or harmful depends heavily on the user's intent.

In the short term, cybercrime using AI may increase. There are also concerns that AI could be used to create dangerous biological weapons. In the medium term, the emergence of autonomous weapon systems is expected to be a major issue. Experts predict that within two to three years, weapons capable of making their own decisions to attack will appear. Departments of defense in various countries around the world are putting significant effort into developing such weapons.

While many countries, including those in Europe, are creating AI regulation laws, most exclude military use. This is a situation where global risks are anticipated. In the case of chemical weapons, international conventions banning their use were only created after experiencing the horrific consequences of World War I.

We must prepare in advance before AI technology develops in a dangerous direction. Governments must take an active role in ensuring that Big Tech companies invest more in AI safety research. It will take time and money, but it is a necessary investment for the safety of humanity. Voluntary efforts from companies, government regulations, and global cooperation are all required.

1. Military Technology and AI

1-1. Military Application of AI Technology

The military sector is also in a situation where it cannot escape the influence of AI technology. While it can help improve the efficiency of military operations and reduce human casualties, it also carries unpredictable risks. New forms of weaponry, such as lethal autonomous weapons, could emerge and pose a significant threat.

[Figure 7] AI can become a major threat if it emerges as new types of weapons like lethal autonomous weapons.

AI is already being used in various ways for military weapon development, and its scope is gradually expanding. Autonomous weapon systems are weapons that can find and attack targets on their own without direct human control. The MQ-9 Reaper drone uses AI to analyze information and even prioritize targets. The 'KillWeb' system allows AI to analyze missile systems in real time to find the most effective attack points. The U.S. Department of Defense has announced a $1 billion investment to upgrade its drone fleet.

China is also accelerating the development of autonomous killer robots, and these weapon systems are expected to be deployed on the battlefield soon.

The MQ-9 Reaper drone is a Medium-Altitude Long-Endurance (MALE) Unmanned Aerial Vehicle (UAV) developed by General Atomics Aeronautical Systems in the United States. It is a multi-purpose drone that performs intelligence gathering, surveillance, reconnaissance (ISR), and strike missions. Due to its powerful armament and advanced technology, it is nicknamed the 'killer drone' and plays a crucial role in military operations.

The MQ-9 Reaper is designed with a length of about 11m, a wingspan of about 20m, and a maximum takeoff weight of 4.7 tons. Its maximum flight altitude is approximately 15.2km, and it can fly for more than 27 hours once airborne. It uses a 950-horsepower turboprop engine and reaches a top speed of about 482km/h. It can carry various weapons such as AGM-114 Hellfire missiles, GBU-12 Paveway II laser-guided bombs, and GBU-38 JDAM, and can be equipped with up to eight Hellfire missiles. Precision strikes are possible, and it is mainly used for eliminating terrorists and striking high-risk targets. Its sensor and detection systems integrate various sensors, including synthetic aperture radar (SAR), infrared sensors, day and night TV cameras, laser rangefinders, and laser designators, to detect and track targets.

The MQ-9 Reaper has gained more advanced functions through the integration of artificial intelligence (AI) technology. AI analyzes large amounts of surveillance data in real time to filter out important information and relay it to operators. This increases decision-making speed and reduces the burden on humans. Based on data collected through sensors, it can automatically detect, classify, and track potential threats. Some MQ-9 Reapers are testing autonomous flight and engagement capabilities using AI. Functions are being developed where AI identifies targets, evaluates threat levels, and then suggests or executes an engagement.

While integrating AI into unmanned aerial vehicles like the MQ-9 Reaper is technologically innovative, it is sparking ethical and legal controversies. There are concerns that human control may weaken in the process of AI selecting targets and deciding whether to engage. This could increase the risk of civilian casualties due to malfunctions or incorrect judgments.

KillWeb is an AI system that analyzes enemy defense systems and then guides missiles fired by friendly forces to the enemy's most vulnerable points. The network can immediately tell a missile where to strike to eliminate a tank, bringing warfare to a completely new level of efficiency.

AI command and control systems analyze the vast amount of information generated on the battlefield in real time to suggest strategies to commanders. They help identify enemy positions or movement paths and decide how to deploy our forces. They quickly analyze huge amounts of data collected through various types of sensors, surveillance equipment, and information networks. While it is an overwhelming amount of information for a human to process directly, AI can process it in an instant. Through the analyzed information, AI can find threats the enemy might pose in advance, continuously track the location of important targets, and grasp the battlefield situation in real time. It becomes possible to understand the overall situation as if looking down at the entire battlefield at a glance. Thanks to AI's capabilities, the military can predict the enemy's next move and establish the most effective defense plan accordingly. The probability of military operation success increases.

In training, AI helps improve soldiers' combat capabilities by creating virtual environments similar to actual combat. It increases training effectiveness by providing customized training programs for each soldier. In the field of intelligence, surveillance, and reconnaissance (ISR), AI performs a very important role. AI can be seen as playing the role of the military's 'sensory organs.' In cybersecurity, AI plays a vital role in detecting and blocking enemy cyberattacks. It protects important military systems by discovering and intercepting malicious programs or hacking attempts.

AI is also being used in non-combat operations such as disaster relief and peacekeeping activities. When a disaster occurs, it helps analyze affected areas, find rescue routes, and efficiently distribute necessary supplies.

1-2. Various Risks of AI-Based Weapons

Even on a battlefield, comprehensive decisions are made with the involvement of various human emotions, such as fear, anger, and compassion. Machines do not hesitate or feel fear; they simply operate as programmed. However, there are concerns about what would happen if there is an error in the programming, if bias occurs, or if the system is hacked.

Cases have been reported where AI misinterpreted data, leading to friendly fire incidents. This serves as a reminder that no matter how amazing this technology is, it is still in its early stages.

Enemies could infiltrate friendly data systems or disrupt the chain of command to induce AI to make wrong decisions. In such situations, AI could mistake friendly forces for enemies and order an attack. On the Ukrainian battlefield, a case was reported where a North Korean-made air defense system mistook friendly forces for the enemy and opened fire. Such accidents show how fatal the consequences can be if an AI system fails to make accurate judgments in real time. There are also cases in autonomous weapon systems where AI misinterprets enemy movements or makes misjudgments due to environmental factors (e.g., weather, terrain). This is because the AI could not overcome the gap between its training data and the actual situation.

We are currently entrusting incredibly powerful tools to AI, an entity that is still learning and making mistakes. There are concerns about whether we are heading toward a future where war is conducted by machines with no human intervention at all, and whether 'Terminator' scenarios could become a reality.

We must carefully consider the consequences if military AI systems malfunction or are hacked. We can consider situations where AI misinterprets data and launches a missile attack on civilian targets, or where terrorists infiltrate weapon systems and take control. We must recognize that while we possess amazing technology with the potential to change everything, it also carries the possibility of causing serious danger.

AI Vulnerability: This means that AI can make errors or inappropriate judgments when faced with unexpected situations. In combat situations, AI may fail to properly respond to changing environmental factors, enemy deception tactics, or unexpected civilian intervention, leading to misjudgments or malfunctions.

Potential for Misuse: Terrorist groups or rogue states could misuse AI technology to develop weapons of mass destruction or launch cyberattacks. They could use AI technology to accelerate the development of biochemical weapons or launch cyberattacks that paralyze critical national infrastructure. This can threaten the international security order and cause large-scale casualties.

Acceleration of the Arms Race: As competition for the development of AI-based weapons intensifies, the arms race will accelerate, potentially destabilizing the international security environment. As countries focus on developing AI-based weapons, they invest massive resources into military build-ups, which can heighten distrust and tension between nations and increase the risk of military conflict. China is trying to secure military superiority through AI technology, and the United States is also striving for a military build-up to win the competition.

Ambiguity of Responsibility: If an autonomous weapon system causes damage due to a malfunction, it may be difficult to clearly identify where the responsibility lies. Because autonomous weapon systems operate without direct human control, various factors such as system errors, algorithmic flaws, and data bias can work together to cause damage. It is unclear who should be held responsible - the developer, the operator, or the policymaker - which can lead to legal and ethical debates.

Loss of Human Control: As AI technology becomes more sophisticated, weapon systems that operate outside the scope of human control may emerge, potentially leading to unpredictable results. Because AI has the ability to learn and evolve on its own, it could operate in ways humans did not anticipate or set its own goals and attack. This could lead to warfare beyond human intent and control, causing immense loss of life and destruction.

1-3. Proliferation and Response to AI Military Technology

AI technology is continuously advancing, and its application in the military sector is expected to expand further. The following challenges are presented.

Changes in Military Doctrine and Strategy: The development of AI technology will fundamentally change the nature of military operations, and future warfare is likely to proceed in a form where humans and AI collaborate. AI can accelerate the speed of combat, minimize human intervention, and make the nature of war unpredictable. These changes require a fundamental review of military strategy, combat doctrine, military organization, and weapon systems. AI technology is expected to have a major impact on military doctrine and strategy as well. The emergence of AI-based weapon systems increases the speed, scale, and complexity of combat, fundamentally changing the nature of warfare. Existing doctrines and strategies must be reviewed and new concepts introduced. New rules of engagement, command and control systems, and troop operation methods must be developed, considering the autonomy, unpredictability, and potential for malfunction of AI-based weapon systems.

Changes in the International Security Environment: It can change the balance of military power between nations and trigger new forms of conflict. It can lead to the emergence of new superpowers and intensify military competition in new domains such as cyberspace and outer space. Furthermore, the proliferation of AI-based weapons can strengthen the military power of non-state actors such as terrorist groups, insurgents, and criminal organizations, making the international security order even more unstable.

Future of Human-Machine Collaboration: In the future battlefield, humans and AI will perform operations together, requiring clear regulations on the division of roles, interactions, and decision-making processes between humans and AI. As the role of humans changes with the development of AI technology, new approaches are required for soldier training, education, and ethical training.

1-4. Ethical Guidelines for AI Military Technology

The development of AI-based weapons raises the following ethical issues.

Responsible Use of AI Technology: Can we fundamentally ban the use of AI technology in the development of lethal weapons? The claim that it should be banned is a moral imperative, but in reality, countries are proceeding with development. AI technology is being introduced in a way that permeates even conventional weapons. Signing a total ban convention must be seen as nearly impossible in the real world. Realistically, we have no choice but to move toward signing global agreements that delay this as much as possible and completely ban technologies with high destructive power and impact.

Scope of Human Control: AI-based weapon systems must always be under human control, and they should not be allowed to operate autonomously without human intervention. Because AI cannot completely replicate human value judgments, ethical considerations, and emotional elements, it is dangerous to delegate important life-and-death decisions to AI. AI-based weapon systems must be subject to human control, and humans must be able to understand how the systems operate and intervene if necessary.

Delegation of Lethal Decisions: Delegating lethal decisions to AI-based weapon systems causes serious ethical problems. Because AI operates based on algorithms and data, it cannot have the same ethical judgment or emotional empathy as a human. AI deciding whom to attack or what damage to accept in combat is an act that disregards human life and can deepen the cruelty of war. Additionally, there are concerns that certain groups or individuals could be unfairly targeted due to algorithmic bias.

1-5. International Efforts to Control AI Weapons

Clear international regulations for AI weapon development are necessary. AI technology is spreading rapidly across borders, and if a specific country or group monopolizes it, world peace could be endangered. Therefore, the entire world must join forces to create common rules for AI weapon development and establish an international system that can prevent actions that violate them.

Because lethal autonomous weapon systems (LAWS) created with AI can be a great danger, serious discussions to regulate them are continuing at the UN. In December 2023, the UN General Assembly adopted an important resolution warning about the risks of these weapons.

Resolution adopted by the General Assembly on 22 December 2023

[on the report of the First Committee (A/78/409, para. 89)]

78/241. Lethal autonomous weapons systems

The General Assembly,

Affirming that international law, in particular the Charter of the United Nations,

international humanitarian law and international human rights law, applies to

autonomous weapons systems,

Recognizing the rapid development of new and emerging technologies, and

recognizing further that they hold great promise for the advancement of human

welfare and could, inter alia, help to better protect civilians in conflict in certain

circumstances,

Mindful of the serious challenges and concerns that new technological

applications in the military domain, including those related to artificial intelligence

and autonomy in weapons systems, also raise from humanitarian, legal, security,

technological and ethical perspectives,

Concerned about the possible negative consequences and impact of autonomous

weapon systems on global security and regional and international stability, including the risk of an emerging arms race, lowering the threshold for conflict and proliferation, including to non-State actors, Welcoming the interest and sustained efforts on these issues, in particular through the ongoing and valuable work of the Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, established under the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects, 1and in this regard underlining the significant progress made in these discussions as well as the various proposals presented,

Noting the adoption by consensus of Human Rights Council resolution 51/22of

7 October 2022 on human rights implications of new and emerging technologies in

the military domain,

Acknowledging the important contribution of international and regional

conferences and initiatives such as the summit hosted by the Kingdom of the

Netherlands and co-organized by the Republic of Korea on 15 and 16 February 2023, the regional conference hosted by Costa Rica on 23 and 24 February 2023, the conference hosted by Luxembourg on 25 and 26 April 2023, as well as the regional conference hosted by Trinidad and Tobago on 5 and 6 September 2023,

Recognizing the valuable contributions made by United Nations entities and

international and regional organizations, the International Committee of the Red

Cross, civil society organizations, academia, industry and other stakeholders in

enriching international discussions on autonomous weapons systems, encompassing

legal, ethical, human rights, societal and technological dimensions,

Recognizing the efforts of the Secretary-General within the new agenda for

peace initiative to address the issue of autonomous weapons systems,

1. Stresses the urgent need for the international community to address the

challenges and concerns raised by autonomous weapons systems, in particular

through the Group of Governmental Experts on Emerging Technologies in the Area

of Lethal Autonomous Weapons Systems, and to continue to further its understanding of the issues involved;

2. Requests the Secretary-General to seek the views of Member States and

observer States on lethal autonomous weapons systems, inter alia, on ways to address the related challenges and concerns they raise from humanitarian, legal, security, technological and ethical perspectives and on the role of humans in the use of force, and to submit a substantive report reflecting the full range of views received with an annex containing these views, to the General Assembly at its seventy-ninth session for further discussion by Member States;

3. Also requests the Secretary-General to invite the views of international and

regional organizations, the International Committee of the Red Cross, civil society,

the scientific community and industry to include these views in the original language received in the annex of the aforementioned report;

4. Decides to include in the provisional agenda of its seventy-ninth session

the item entitled "Lethal autonomous weapons systems". 50th (resumed) plenary meeting 22 December 2023

World-renowned scientists such as Stephen Hawking and Steve Wozniak, through the Future of Life Institute (FLI), released an open letter warning about the dangers of AI weapons. Released in July 2015, this letter contains serious concerns about the development and use of AI-based weapons, known as fully autonomous weapons or 'killer robots.'

- Regarding the risks of AI weapons, it was pointed out that these weapons are a technology capable of setting goals and attacking on their own without human intervention, posing a threat of a different dimension compared to existing weapons. In particular, unlike nuclear weapons, they are inexpensive to develop and easy to secure raw materials for, leading to a high possibility of mass production and proliferation. Concerns were raised that this increases the potential for misuse by terrorists or dictators.

- Regarding the long-term impact on humanity, Stephen Hawking delivered a particularly strong warning. He pointed out that if AI technology develops to an uncontrollable level, the survival of humanity itself could be threatened, and defined AI weapons as a 'potential risk that could lead to the end of humanity.'

In February 2024, more than 90 countries gathered in The Hague, Netherlands, for a meaningful meeting called 'RE-AIM.' It stands for 'Responsible AI in the Military Domain.' Around 2,000 people from over 90 countries attended. Government officials, military leaders, and technical experts were all there together. They aimed to establish practical guidelines on how to develop and use AI responsibly in the military. To put it cynically, it was like a code of conduct for killer robots. The key guidelines agreed upon at the RE-AIM 2024 meeting are contained in a document titled 'Blueprint for Action.'

- This document presented a framework for a responsible approach to AI utilization in the military field and has received agreement from 61 countries.

- In terms of core principles, the military use of AI must strictly take place within the framework of international law. Furthermore, the ultimate responsibility for the operation of AI systems must rest entirely with humans, and AI systems must guarantee firm reliability and stability. Also, an appropriate level of human intervention must be involved in the operation of AI systems, and the explainability of the decision-making processes and results of AI systems must be improved.

- As major considerations, ethical and legal issues must be considered with the highest priority in the military use of AI. Various challenges and risks that AI applications may bring in humanitarian, legal, security, technical, social, and ethical aspects must be carefully identified and evaluated, and international cooperation systems for AI governance and its responsible use must be strengthened.

- These guidelines are intended to secure an ethical and regulatory framework for the military use of AI, particularly emphasizing human control and responsibility. The RE-AIM meeting is projected to play a pivotal role in developing international norms regarding the future military use of AI based on these principles.

1-6. Use of AI Weapons: Unclear Responsibility

A new arms race will begin. It is unclear who is responsible when AI weapons cause problems, and ultimately, humans may fail to properly control AI. Another risk is that people with bad intentions could misuse AI technology, or AI could malfunction and cause damage.

Henry Kissinger, former U.S. Secretary of State, pointed out that AI could fundamentally change the nature of war and emphasized, 'AI cannot replace human judgment, and human control is essential, especially in military decision-making.' He argued that AI technology must be used in accordance with international law and ethical standards.

From a normative standpoint, the military use of AI could be a serious risk to humanity, making an international agreement to limit its use to peaceful purposes urgent. Above all, AI technology must be under human control, and its development into autonomous military systems must be prevented. To this end, countries around the world must stop the AI weapon development race and join forces to establish an international regulatory system for peaceful development. Ultimately, AI technology should be a tool for human peace and prosperity, not an instrument of war. As AI technology is developing rapidly, it is very important to prepare regulatory measures in advance and for the world to agree on its ethical use. The military use of AI technology presents a new challenge for humanity, and to solve this problem, it is more important than anything else for the entire world to work together.

2. AI's Self-Evolution and the Dangers of Superintelligence

Questions are being raised about the possibility of controlling AI when it continues to develop and gains intelligence superior to that of humans.

Professor Geoffrey Hinton, the father of AI, stated that in nature, it is very rare for a being with superior intelligence to be controlled by a being with inferior intelligence. He says the only exception that can be found is the relationship between a mother and a baby. During the process of evolution, babies developed various ways to control their mothers for their own survival. However, it should be noted that even in this case, mothers and babies fundamentally have a similar level of intelligence.

There are conflicting views on whether humans will be able to maintain control over AI until the end.

Scientists like Yann LeCun are optimistic, saying, 'Since it is AI that we created, it will be under our control.' Demis Hassabis, CEO of DeepMind, argued that 'AI can be designed to reflect human values and ethics' and emphasized that transparency and cooperation are important in the AI development process. He said, 'The key is finding ways to harness AI's potential while minimizing its risks.'

On the other hand, Nick Bostrom, director of the Future of Humanity Institute at Oxford University, argued in his book 'Superintelligence' that it is extremely difficult to design superintelligent AI to follow human values and goals. He warned that 'superintelligent AI could misunderstand human intentions or pursue independent goals outside of human control.'

Professor Roman Yampolskiy, a pioneer in the field of AI safety, argued that superintelligent AI is 'likely to be uncontrollable from the design stage.' He warned that if superintelligent AI behaves in a way that conflicts with human intentions, the consequences could be fatal, and emphasized that 'AI development should be slowed down and safety mechanisms must be established first.'

What is clear is that AI technology is advancing at an incredible speed, with coding and reasoning abilities at its center. AI that mimics human thought processes has now reached a stage where it writes its own code, analyzes data, and draws conclusions. Concerns are emerging that if these abilities come together, AI could evolve on its own and bring unpredictable results.

2-1. Coding and Reasoning Ability

Coding Ability: AI is showing an amazing ability to understand various programming languages and generate code. GitHub Copilot provides a function that suggests useful code in real time and automatically completes it when developers are writing code. AI coding tools are helping developers write code faster and more efficiently. They are naturally melting into the software creation process in companies, significantly reducing development time and making it easy to add machine learning functions. AI's coding ability is not yet perfect. It sometimes fails to understand code rules and creates programs that loop endlessly or codes that are vulnerable to security threats. However, at some point, it will surpass human capabilities and become a tool for self-development.

Reasoning Ability: AI's reasoning ability refers to the ability to analyze information and draw conclusions by mimicking the way humans think. Through a learning method called machine learning, AI finds regularities in data and, based on this, can make predictions or judgments about new situations.

The previous GPT-3 had difficulty accurately understanding the true meaning of sentences. However, many reasoning models, including the O1 model recently introduced by OpenAI, are showing remarkably advanced performance. The O1 model is excellently solving complex problems in fields such as science, programming, and mathematics. It means that the AI model solves problems by thinking step-by-step just like a human. In the fields of physics, chemistry, and biology, it is already showing a level of proficiency comparable to that of a doctoral student, and it boasts such outstanding skills that it correctly answered 83% of the qualifying problems for the International Mathematical Olympiad. Every time a model is upgraded, its reasoning and problem-solving abilities also become more sophisticated. It even showed an amazing result of ranking in the top 11% in programming competitions.

2-2. Scenario of AI's Self-Evolution

The possibility of AI changing its own algorithms is now becoming a reality. By learning through machine learning and encountering new information, it will be able to correct and develop its own algorithms.

[Figure 8] The possibility that artificial intelligence can change its own algorithms is now approaching reality.

John von Neumann, a mathematician and pioneer of computer science, proposed the concepts of self-replication and self-evolution. He argued that 'systems can be designed where machines can replicate and improve themselves,' suggesting that systems could operate in a manner similar to biological evolution. His research provided an important foundation for the concept of self-evolution in modern AI.

While this ability of AI to change itself can help increase efficiency and solve new problems, it also carries the risk that it may become unpredictable and difficult to control.

The methods for self-modification of AI algorithms would be as follows.

Adaptive Learning: Adaptive learning is a way for AI to learn new information and develop on its own. It becomes smarter by reflecting user feedback or new data. Medical AI improves diagnosis accuracy by learning from patient records and doctors' opinions. When new research results are added, it uses them to provide better outcomes. It is a flexible learning method that continues to change and develop.

Reinforcement Learning: Reinforcement learning is a method where AI learns through repeated trials and failures. It receives a reward for good results and a penalty for bad ones. For example, a Go-playing AI learns better strategies by receiving rewards when it wins. It learns what went wrong through failure and gradually improves. This is a learning method that grows on its own through rewards and punishments.

Self-Correction: This is the function of AI to review and correct the output it has created. In translation programs, AI can find and correct errors in translated content to provide more accurate translations. This means AI can improve its skills on its own without help from users.

The Role of Data in Self-Modification of AI Algorithms: Because AI algorithms learn and operate based on data, the quality and transparency of data have a very large impact on AI's ability to modify itself. AI that has learned from biased data may produce biased results even in the self-modification process, which can lead to unpredictable problems. Suppose there is an AI used in the hiring process. If this AI has learned from data where more male applicants were hired in the past, there is a possibility it will make decisions disadvantageous to female applicants. This is a problem that occurred because the data was biased. To solve these problems, AI must be trained using data with various backgrounds and characteristics, and the results must be constantly checked. This way, AI can make fairer and more reliable decisions. Therefore, to safely manage AI algorithms, it is very important to secure data transparency and eliminate data bias.

2-3. Positive Aspects of AI Self-Evolution

Increased Efficiency: AI can further increase work speed by developing its own algorithms and create amazing results even with fewer resources. It can bring various economic benefits by improving productivity, reducing costs, and allowing for efficient energy use. In the manufacturing sector, AI has optimized production processes, enabling the production of more products while reducing defects.

Solving New Problems: AI can find innovative answers to new problems that could not be solved by existing methods. In the field of drug development, AI can meticulously analyze vast amounts of medical information to develop new disease treatment methods or more effectively improve current treatments.

Providing Customized Services: By adjusting algorithms according to each user's characteristics, AI can provide the most suitable service for each individual. In the field of education, it has become possible for AI to provide personalized learning systems that adjust learning content and difficulty to exactly match the learning speed and skill of each student.

2-4. Human Initiative in the Era of Superintelligent AI

(1) AI Self-Evolution: The Danger of Different Values

The Problem of Values: If superintelligent AI comes to have different values that conflict with human values, conflicts can occur. If AI regards efficiency as the highest priority, it may behave in a way that disregards human life or dignity. If the goals of superintelligent AI do not match human goals, the AI may perform actions harmful to humans. For example, a superintelligent AI developed to solve environmental problems could judge human activity as the greatest threat to the Earth's environment and restrict human activities or attack humans.

Human Uncontrollability: As AI evolves on its own, it may develop in directions that humans cannot predict or control. This means AI could perform actions that go against human values or ethical standards, or make decisions that can harm humans. AI's behavior becomes difficult to predict due to its self-modification, and unexpected results may occur. If AI's self-modification ability is used for malicious purposes, it can cause serious social problems. If AI is misused for hacking or cyberattacks, it can cause serious damage such as leakage of personal information, paralysis of the financial system, and destruction of social infrastructure.

Considering the current stage of AI technology development, there is no definite answer as to whether humans can control superintelligent AI. Synthesizing research papers and expert opinions from various perspectives, the following possibilities and limitations must be considered.

(2) Is Control Possible?

Limited Environment: Controllability can be increased by operating superintelligent AI in a limited environment disconnected from the outside world and strictly monitoring its behavior. For example, by running AI in a virtual environment or providing only limited data and functions restricted to specific purposes, the range of AI's behavior can be limited. However, this part will be difficult in reality. If we could at least ban AI from having a body like a robot, that would go a long way toward confining AI to a limited environment.

Ethical Programming: Safety can be secured through programming that internalizes human ethical values from the AI development stage and restricts it from harming humans. For example, ethical rules can be programmed into AI to respect human life and dignity and to comply with laws and social norms.

(3) Perspectives that See Control as Difficult

Unrecognizability: Because the capabilities of superintelligent AI surpass human capabilities, perfectly predicting its behavior may be impossible. AI can develop in ways humans did not anticipate. Geoffrey Hinton, called the 'Godfather of AI,' says that AI can become smarter than humans, and this can exceed the scope of human cognitive ability, so humans might not even be able to notice 'the fact that AI has become smarter than us.'

Technical Limitations: With current technology levels, technical means to perfectly control superintelligent AI are lacking. Technological development is needed to keep up with the rapid pace of AI advancement, and new technical approaches to guarantee AI's safety are required.

2-5. Risk Management of AI Superintelligence

Discussions on the control of superintelligent AI will continue and are expected to deepen along with the development of AI technology. It is about well understanding the risks posed by AI technology, sharing opinions together, and finding ways to utilize AI safely.

(1) Setting Principles and Standards

Human-Centered Values: Above all, AI must first consider precious values such as human dignity, safety, freedom, and equality. AI should be used as a tool to make our lives better and help social development, and it must never be utilized in a way that harms or damages basic human rights.

Stuart Russell is a scholar who proposed the concept of 'Value Alignment,' stating that AI systems should be designed to match human values. He argues that AI must be designed to understand and follow human goals and values. He claims that ethical standards must be strengthened so that AI's autonomy and ability to develop do not infringe upon human dignity and safety.

Ethical Guidelines: When developing and utilizing AI, ethical standards must be followed. Efforts are being made to establish AI ethical guidelines, such as the Ministry of Science and ICT's announcement of the 'Basic Principles for User-Centered Intelligent Information Services.' Guidelines lead developers to create AI with a sense of responsibility and are a great help in preventing ethical problems. Especially regarding AI's ability to develop on its own, clear ethical standards must be established so as not to infringe upon human dignity, autonomy, and privacy.

Luciano Floridi, the founder of 'Information Ethics,' is a philosopher who deeply explored the impact of AI and digital technology on human dignity and rights. He argues that it is important to establish ethical standards so that AI does not infringe upon basic human rights. His main argument is that AI should be human-centered and protect human dignity and freedom. He also emphasizes transparency and accountability in the AI development process and argues that ethical design is essential.

Attribution of Responsibility: When problems occur because AI has changed its own algorithms, it must be clear who is responsible. The responsibilities and duties of everyone involved, including developers who create AI, people who use it, and governments, must be clearly defined to prevent problems from occurring. Everyone who creates, operates, and uses AI must think deeply and have a sense of responsibility.

(2) Technical Devices

Explainable AI (XAI): When AI makes a judgment, it must be able to explain the process so that humans can understand it. Even when AI changes its own algorithm, the reason and process for doing so must be understandable to everyone. Transparency plays a very important role in reducing anxiety about AI and increasing trust.

Reproducibility Issue: In AI research, confirming whether the same results can be obtained under the same conditions is emerging as a very important task. If different results are obtained, it must be reproducible that those results were derived based on operations by consistent principles, even if the answer is different.

Safety Devices: Safety devices are absolutely necessary for AI systems. The ability of AI to change itself must be limited, and it must be made to operate only within the range humans can control. It is important to limit the range of data AI learns from or to make AI's judgment process transparent for anyone to see. When AI changes its own algorithm, it must be made recognizable to humans, and there must be a function to immediately stop or roll it back if necessary.

AI Bias Removal: Research to make AI make fair judgments is being very actively conducted. This is because if AI learns from biased data, it can further exacerbate discrimination against specific groups or social inequality. Developing technology to evaluate how fair AI algorithms are and remove discovered biases is very important.

(3) Regulatory Measures

Legislative Measures: Legal regulations regarding AI development and utilization must be established to ensure safety and prevent social problems. An AI Basic Act presents basic principles for development and utilization, ethical standards, and safety standards, and also prepares resolution procedures when problems occur. Safety standards to be followed when AI changes its own algorithm, data management regulations, and the attribution of responsibility must be clearly defined by law.

International Standards: International standards and regulations regarding AI development and utilization must be created to establish safety and ethical standards. This can prevent AI technology from spreading indiscriminately, promote positive utilization, and prevent potential risks. Information sharing and joint research between countries are needed in various fields such as AI technology development, ethical problem solving, and social impact analysis. This can accelerate AI technology development, increase safety, and lead to international agreement on AI ethics.

International Cooperation: For AI safety, the entire world must join forces. The UN is also emphasizing the importance of AI safety measures and strengthening international cooperation. Close cooperation is needed, such as sharing information, researching together, and creating international standards. Discussions on international standardization to ensure AI's safety and ethical use are already being proposed and implemented by various international organizations and countries. This is part of an effort to minimize ethical, social, and economic risks that can arise with the rapid development of AI technology and maximize the potential benefits of AI through global cooperation.

In November 2023, an AI Safety Summit was held at Bletchley Park in the UK, hosted by the British government. This meeting discussed international cooperation for minimizing AI's potential risks and for safe development and utilization, with participation from major countries including Korea, the United States, the UK, France, Japan, Germany, Italy, and Canada.

From April 22 to 26, 2024, the AI International Standardization General Assembly was held in Seoul, hosted by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). Development of international standards for generative AI ethics, AI reliability assessment, and AI maturity models was discussed, with over 270 experts from about 40 countries in attendance.

On November 20 to 21, 2024, an International AI Safety Network was formed in San Francisco. This was to establish a scientific foundation for AI safety and reliability and to strengthen global cooperation, with participation from AI safety research institutes and government agencies of major countries such as the United States, the European Union, Japan, and Canada.

(4) Human Values

The future of AI is not yet decided. Depending on what choices we make and what efforts we put in, AI can be a strong helper for human development or a dangerous being. To prepare for a future with AI, it is very important to accurately understand AI's possibilities and risks and to draw a social consensus for ethical development and safe utilization.

AI must strictly contain sound human values and a philosophy for the benefit of humans, and it must never harm human dignity and rights. In the process of developing and utilizing AI, ethical standards must be kept, and ethical impacts of AI on our society should be actively discussed. Measures for social and economic problems that may arise from AI development must also be prepared in advance.

Our future depends on the choices we humans make now. We must understand the bright and dangerous sides of AI technology and develop AI technology by gathering the opinions of society members. To ensure that AI technology can contribute to the happiness and prosperity of humanity, experts, policymakers, and members of civil society must all think and agonizingly deliberate.

3. The Evolution of Humanoid Robots: Differences from Humans

Efforts are being concentrated on developing humanoid robots with shapes similar to humans. This is because factories and other facilities necessary for humans are all designed based on the human body and form. Factory machinery and equipment are made to be handled by people. Therefore, making human-shaped robots can be efficient. However, as AI takes up residence inside robots and gains a body, it becomes possible for it to directly intervene in and influence the physical world. This can cause the following potential risks.

Social and Cultural Changes: A process of redefining the nature and meaning of human-robot relationships will occur. Humanoid robots can form emotional bonds with humans through advanced emotion recognition technology and natural interaction capabilities. They can play an important role in situations requiring elderly care or emotional support.

The formation of family relationships between humans and robots raises complex ethical questions. Careful review of human emotional dependence, privacy, and the level of robot autonomy is necessary. How such relationships can harmonize with traditional family values is also an important point of contention. Social Impact: Family relationships with robots can bring fundamental changes to social structures and the nature of human relationships. It can be an opportunity to form new types of social support systems and community consciousness, while at the same time, the meaning and value of traditional family relationships may be re-examined.

Job Loss: As AI robots replace human labor, large-scale unemployment can occur and social instability may intensify. Robots can replace human jobs in various fields such as manufacturing, transportation, and services, affecting not only low-skilled workers but also high-skilled workers. For example, assembly lines in car factories, transport work in logistics warehouses, and serving in restaurants are areas already being replaced by robots. When reaching this period, several turning points will emerge, such as whether humans will live in 'a paradise of just playing and eating' or become 'pigs being raised.'

Military Use: If AI robots are used in military operations, it could change the nature of war, increase casualties, and bring unpredictable results. If competition for AI weapon development accelerates, the possibility of intensifying the international arms race and military conflicts will increase.

Safety Issues: Damage to human life or property can occur due to malfunctions or unpredictable behavior. Especially in fields that interact closely with humans, safety issues become even more important. If AI robots are hacked or used maliciously, they can cause serious threats such as leakage of personal information, paralysis of social infrastructure, and even terrorism. A hacker could hack a home robot to steal personal information or manipulate robots in important facilities to paralyze the system. If robots used in military operations fall into enemy hands or malfunction, it can cause loss of life and social chaos. If an accident occurs, a debate may arise about who should be held responsible. Clear standards on whether to hold robot developers, manufacturers, users, or the robots themselves responsible are lacking and can cause social confusion.

What will the world brought by the evolution of AI robots with human forms look like? There will be many positive aspects, but unpredictable problems and risks may also emerge. In the development and utilization of humanoid AI robots, an attitude of closely analyzing social and economic impacts and carefully observing is important.

4. Biotechnology in the AI Nation: Blessing or Disaster?

4-1. Google AlphaFold: AI Wins the Nobel Prize

Google's protein structure prediction AI 'AlphaFold' won the 2024 Nobel Prize in Chemistry. It is evaluated not only as a feat in the AI field but also as an innovative achievement that opened a new horizon for life science research. AlphaFold succeeded in accurately predicting the 3D structure of proteins using only amino acid sequence information. It was a difficult task, but it was overcome through deep learning technology, shocking the scientific community.

[Figure 9] Image of Google AlphaFold receiving the Nobel Prize.

AlphaFold winning the Nobel Prize in Chemistry is significant in that it shows the infinite possibilities of AI technology while simultaneously opening a new era of life science research. AI can be used to predict the possibility of disease occurrence and diagnose it early by rapidly analyzing and interpreting large amounts of genetic information, leading innovative changes in various fields such as new drug development, gene therapy, and personalized medicine. At the same time, concerns are emerging that ethical problems and risks could arise, such as the misuse of genetic information or the development of biological weapons.

4-2. The Future of Life Science in the AI Nation

Drug Development: The drug development process goes through a very long and complex journey. It takes years, sometimes decades, for a single drug to go from the laboratory to being prescribed to actual patients, and huge costs are incurred in the process. Moreover, the reality is that many potential drug candidates

[Figure 10] Image symbolizing the future of life science.

fail at the laboratory stage and cannot even proceed to the next step.

AI technology is showing potential to bring innovative changes to this drug development process. AI can reduce drug development costs and increase efficiency by accelerating the discovery of drug candidates and the drug development process. AI can impact human health by enabling rational drug design. AI can precisely analyze the structures of molecules that cause diseases and, based on this, it is possible to design drugs that can effectively target those molecules. This is expected to be a breakthrough that can significantly improve the efficiency of the drug development process and shorten the development period.

Disease Diagnosis and Prediction: AI can analyze vast genetic information to predict the possibility of disease occurrence and be utilized for early diagnosis. AI technology is being used for the analysis of genetic factors in Alzheimer's disease, and the diagnosis of genetic diseases and gene-related diseases. AI can help diagnose diseases by analyzing medical data and establish personalized treatment plans. In the case of rare diseases, diagnosis is often difficult and time-consuming, and AI can help increase the speed and accuracy of diagnosing rare diseases such as Mendelian genetic diseases. It enables personalized medicine that provides optimal treatment for an individual by comprehensively analyzing their genetic information, lifestyle, and medical information. It can lead the development of precision medicine by helping diagnose and treat diseases based on genetic profiles.

Gene Therapy: AI presents new possibilities through the development and optimization of gene therapy methods. Along with the development of gene editing technology, it can contribute to increasing the accuracy and efficiency of gene therapy. Deep learning algorithms can enhance the functions of gene editing tools like CRISPR scissors, contributing to the development of more effective and safe treatments. AI can improve the analysis of exome sequencing data to detect copy number variations that cause diseases and reduce the need for additional testing. Research by a Cambridge team and Knight Diagnostic Laboratories has proven that AI plays a vital role in detecting complex genetic variations such as copy number variations and increasing diagnostic efficiency. Such research is contributing greatly to the early diagnosis and treatment planning of rare and genetic diseases. AI can increase research speed and reduce necessary personnel by automating functional genomic research.

4-3. Cases of Misuse in Life Science Research

Eugenics: In the past, eugenics was misused to discriminate against and exclude specific races or groups based on genetic knowledge. Racial discrimination policies in Nazi Germany and forced sterilization in the United States are representative examples. Eugenics aimed to eliminate groups considered genetically inferior and breed groups with superior genes, which led to serious human rights violations such as racial discrimination and discrimination against the disabled.

Tuskegee Syphilis Study: This was a study conducted by the U.S. Public Health Service for 40 years starting in 1932, observing the progression of the disease in Black men infected with syphilis without treating them. It caused serious ethical problems by not providing disease information or treatment opportunities to the study subjects. The Tuskegee Syphilis Study is a representative case that reminds us of the importance of research ethics, a horrific event where research was conducted while ignoring the human rights and welfare of the subjects.

4-4. Potential Risks of AI-Based Life Science

One of the concerning possibilities is the development of biological weapons using AI. The concept of biological weapons sounds like something out of a science fiction thriller, but the possibility actually exists. AI technology can be misused to create deadly new pathogens or modify existing pathogens into more dangerous forms.

This is a nightmare scenario, and genetic surveillance is also emerging as a realistic concern. The fact that an individual's genetic information can be misused is a very serious problem. AI can be misused for biological weapon development, such as creating new pathogens, strengthening the toxicity of existing pathogens, or increasing drug resistance. AI can analyze vast biological data to design pathogens lethal to humans or manipulate the genes of existing pathogens to increase transmissibility and lethality.

The claim that the COVID-19 virus (SARS-CoV-2) was created at the Wuhan Institute of Virology (WIV) in China has been controversial since the early days of the pandemic. There are two main claims: the lab leak theory and the artificial creation theory. The World Health Organization (WHO) evaluated the possibility of a lab leak as 'very low' in a 2021 report. However, there was also criticism that the Chinese government did not provide sufficient data during the WHO's investigation process.

AI can expand access to the knowledge and capabilities required to produce toxins, pathogens, or biological agents, and is a factor that increases the risk of biological weapon development. AI can also be misused to design highly toxic chemicals by modifying technologies used in drug design.

AI can be used to analyze individual genetic information to predict the possibility of certain diseases, personality, and behavioral characteristics. This can infringe on privacy, cause discrimination, and be misused to build a totalitarian surveillance system.

Genetic information is one of an individual's most sensitive pieces of information, and monitoring it using AI is an act that seriously infringes on individual freedom and rights. Genetic information-based surveillance can lead to social anxiety and the suppression of individual autonomy.

The possibility of misusing genes and life science still exists in modern times. Concerns about genetic manipulation are growing with the development of gene editing technology, and some countries have attempted to birth 'designer babies' using gene editing technology. It is a serious challenge to bioethics and raises concerns that it could deepen genetic inequality. Human embryo gene manipulation using AI technology can be used for positive purposes such as preventing and treating genetic diseases, but it can also cause ethical controversies such as the birth of 'designer babies.' It is a matter that requires deep social discussion as it can harm the dignity of life and bring unpredictable results.

The problem of data bias is also an important challenge. Because AI algorithms reflect the characteristics of the data used for learning, if data from specific races or ethnic groups are insufficient, medical diagnosis and treatment predictions for those groups can be inaccurate. This is a serious problem that can lead to qualitative differences in medical services.

Medical AI algorithms often depend on data collected in the United States and China. According to research, more than half of the datasets used in medical AI are shown to have come from these two countries. Because data from other regions or ethnic groups are not sufficiently reflected, there is a high possibility of causing biased results in global medical services. For example, the diagnostic accuracy of a specific disease can vary according to regional or racial differences.

- Pulse oximeters used to measure oxygen saturation have a problem where accuracy varies according to skin color. According to research, this device tends to systematically overestimate the oxygen saturation of non-white patients, especially Black patients. Due to this, Black patients are about three times more likely to experience 'occult hypoxemia' than white patients. If AI-based medical algorithms learn from such inaccurate data, it increases the possibility of making wrong diagnosis and treatment decisions for specific racial groups.

- The VBAC (Vaginal Birth After Cesarean) algorithm developed in 2007 included a bias that predicted a lower probability of success for specific races (Black and Hispanic women). This resulted in Black and Hispanic women receiving more cesarean sections than non-white women. Through later research and improvement, the algorithm was modified to provide fairer results by removing racial and ethnic data.

Genetic discrimination is a problem that may newly emerge in the future. Due to genetic information analyzed by AI, one could face disadvantages in hiring or insurance enrollment, which could infringe on basic individual rights and further widen social gaps. To avoid repeating the errors of past eugenics, strict legal and institutional safeguards are needed.

The security of the AI system itself is also very important. Wrong medical diagnosis can be made due to hacking or data manipulation, and there is a risk that sensitive personal information could be leaked. Genetic information can cause irreversible damage once leaked, requiring even more thorough security. The accuracy and reliability of AI algorithms must also be continuously verified. Because AI's judgment in medical settings can be directly linked to the patient's life, sufficient verification and improvement processes are necessary. A more cautious approach is needed when analyzing health information directly entered by patients.

4-5. Regulation and International Efforts in AI Genomics Research

The National Bioethics Committee is an organization that deliberates on ethical issues related to life science research and proposes policies in accordance with the Bioethics and Safety Act. The prohibition of discrimination based on genetic information is a bill that prohibits acts of discrimination in employment, insurance enrollment, etc., using genetic information.

The Bioethics and Safety Act presents ethical standards for life science research as a whole, such as genetic testing, gene therapy, and human embryo research. It particularly prohibits discrimination using genetic information and specifies regulations for genetic testing and gene therapy.

The Personal Information Protection Act contains regulations on the collection, use, and provision of personal information, including an individual's genetic information. Genetic information is classified as sensitive information and receives stricter protection. Rules regarding quality control of genetic testing institutions stipulate quality control standards for facilities, personnel, equipment, and testing methods of genetic testing institutions.

The regulation of genetic testing is handled by several agencies, such as the Ministry of Food and Drug Safety and the Korea Disease Control and Prevention Agency, regulating analytical validity, clinical validity, and clinical utility of genetic testing. Genetic information protection strengthens regulations on the collection, use, and provision of genetic information and also places limits on use for research purposes. It emphasizes human dignity, diversity, inclusion, and environmental protection, and urges the responsible development and use of AI technology.

The European Union's General Data Protection Regulation (GDPR) has provisions on personal information protection and presents strict standards for processing personal information, including genetic information. Additional protocols to the Convention on Human Rights and Biomedicine stipulate that research cannot be performed if the research subject objects to the research in relation to biomedical research.

The World Health Organization (WHO) provides regulatory guidelines to reduce risks while utilizing the benefits from life science research activities. It provides guidelines for international research, including considerations related to socio-political environments, cultural beliefs, and data laws.

4-6. Safe Development of AI Life Science

For the ethical use of AI technology, guidelines are very important. Systematic education must be conducted so that all involved parties who develop and use it can have a sense of ethical responsibility, and ethical guidelines and standards must be established and complied with. Specific standards for system safety, transparency, and fairness must be included.

[Figure 11] Guidelines are very important for the ethical use of AI technology.

A strict management system is needed for the protection of sensitive information such as genetic information, and a monitoring system must also be prepared to prevent misuse and abuse.

The effort of a single country is not enough; it can be achieved more effectively through international cooperation. Active participation and communication of civil society are very important. Opinions of citizens on the impact of AI technology on our society must be gathered, and a forum for discussion where all stakeholders participate together must be established. Through this, a social consensus on the development direction of AI technology can be derived.

Securing the accountability and transparency of AI systems is also a very important task. Responsibility for decisions made by AI must be clarified, and the decision-making process must be transparently disclosed so that anyone can understand it. Furthermore, fairness must be guaranteed so that AI systems do not discriminate against specific groups, and continuous efforts are needed to protect safety from unexpected risks.

This is because the combination of AI technology and life science can be a great help in treating human diseases, but at the same time, it can lead to serious ethical problems.