ChatGPT is making headlines as the craze continues. The super-large AI model is an AI that can perform large-scale learning and fast calculations like humans, and to implement it, it requires dedicated semiconductors and computing infrastructure that can perform it. Accordingly, investment in core infrastructure of the AI market, such as domestic and international AI semiconductors and data centers, is being promoted.

In the AI era, a key point is that high-performance and high-capacity memory is essential to drive high-performance processors. It is predicted that demand for high-bandwidth memory (HBM) that directly provides data to GPUs or AI accelerators, as well as high-capacity server DRAMs of 128 GB or more, will increase.

In particular, based on the memory sector where Korea is gaining ground to take the lead in the market, the combination of high-performance processors capable of mass calculations and high-capacity memory to support them is in the spotlight. There are growing predictions that in the long term, AI semiconductors specialized in memory will determine the performance of generative AI.

■ Samsung Electronics - Tightly Holding on to 'PIM', SK Hynix - Steady Memory Innovation

Samsung Electronics is continuing to develop 'HBM-PIM', which combines memory semiconductors and AI processors in 2021.

Samsung Electronics said in its 2022 earnings conference call in January that “natural language-based conversational AI services will impact the demand for next-generation memory.”

'PIM (Processing-in-memory)' is a memory that adds AI processor functions that perform computational work inside the memory. It reduces the movement of large amounts of data between memories, thereby increasing the performance and efficiency of the AI accelerator system.

The HBM-PIM from here has emerged as an AI-tailored solution that maximizes the performance of AI accelerators. HBM has been responsible for improving the high-speed processing of high-performance computers (HPC). HBM is expected to see increased demand as a next-generation memory that can complement the limitations of DRAM. AMD has collaborated with Samsung Electronics to install Samsung Electronics' HBM-PIM memory in its AI processor 'MI-100'.

At the PIM AI Semiconductor Symposium, Samsung Electronics Master Kyo-Min Son said, “If HBM-PIM is installed in an AI system, the performance will increase by about twice or more compared to the existing HBM2-based system, and the system energy will be reduced by more than 70%.”

Meanwhile, Samsung Electronics joined hands with Naver in December last year. The two companies agreed to cooperate to resolve AI system bottlenecks and maximize power efficiency. They will develop a next-generation AI-only semiconductor solution by combining Samsung Electronics’ semiconductor design and manufacturing technology with Naver’s AI algorithm development and verification and AI service experience.

Naver Cloud CTO Kwak Yong-jae explained, “Together with Samsung Electronics, we are creating a lightweight AI semiconductor solution that has all the functions necessary for LLM (large-scale language model) calculation, learning, and inference, while having a model size that is 1/10 the size of existing GPUs and more than four times the power efficiency.”
SK Hynix has long-standing capabilities in the HBM field, including developing the world's first HBM product together with AMD. In October of last year, it successfully developed the next-generation stacked memory 'HBM3' optimized for AI inference and learning performance.

HBM3 meets the requirements of higher-density system levels with a combination of 16Gb core die density and TSV (Through Silicon Via) technology. When combined with NVIDIA H100, data processing speeds are shown to be 78% faster than the existing HBM2E.

In January, the company continued to demonstrate its technological leadership by developing the mobile DRAM 'LPDDR5T'. The product boasts an ultra-high speed of 9.6 gigabits per second, which is 13% faster than LPDDR5X, and operates at a voltage of 1.01 to 1.12 V, implementing ultra-low power characteristics. It is based on the 10nm-class 4th generation (1a) fine process and will enter mass production in the second half of the year.

SK Hynix said, “We expect the scope of application of this LPDDR5T to expand beyond smartphones to AI, machine learning, and AR/VR.”

In addition to memory semiconductors, domestic AI semiconductor companies such as Sapion, Rebellion, and Furiosa AI, which have joined hands with the three major telecommunications companies, are also throwing their hats into the ring. Meta, Naver, and Kakao are also not leaving out the competition by announcing plans to release large-scale language models.

■ Expanding investment in data centers, essential infrastructure for AI implementation

For large-scale computation of AI models with billions of parameters, the role of data centers has become important not only for AI semiconductors but also for improving AI performance and stably supporting data processing.

Data centers provide cloud computing services that can collect, store, and analyze big data, and are suitable for training AI models. In Korea, investment in data centers to secure stable infrastructure is accelerating along with the development of AI semiconductors.
Naver Cloud is moving forward with establishing 'Gak Sejong' based on its Chuncheon data center operation capabilities. 'Gak Sejong' aims to complete construction within the second quarter of this year and begin actual operation in the third quarter. 'Each Sejong' secures stable power capacity and is equipped with cooling technology to increase the efficiency of AI infrastructure.

Naver Cloud IT Service Headquarters Director Jeong Su-hwan said, “Each Sejong will become the foundation for Naver’s large-scale AI HyperClova, which has been attracting attention recently, to grow and expand across the world.”

Kwak Yong-jae, CTO of Naver Cloud, said, “Considering that large-scale AI will become more advanced in the future, an integrated approach to software-hardware-operating environment is necessary,” adding, “If the AI semiconductor solution developed in cooperation with Samsung Electronics is operated in each Sejong location, more efficient and stable operation will be possible.”

SKT has begun improving the operational efficiency of its data center business in line with the growth of its AI service, ‘Adot.’

Last February, SKT expanded the number of NVIDIA A100 GPUs in its supercomputer ‘Titan’, the brain of its giant AI model ‘Adot’, to 1,040, doubling the number from the previous number. As a result, SKT’s supercomputer now supports a performance of over 17.1 petaflops, enabling Adot to perform more sophisticated learning than before.

■ DPU growth to be noted along with the rise of AI semiconductors

In this flow, DPU (Data Processing Unit) is a notable keyword. DPU is a processor that handles data processing such as CPU and GPU installed in high-performance servers, and supports faster and more expandable data centers.

According to Market and Market, the global DPU market is expected to grow by an average of about 35% per year, reaching a size of about KRW 100 trillion by 2027.

Quoting Jensen Huang, founder and CEO of NVIDIA, “Following CPU and GPU, DPU will be the other core axis of data-centric accelerated computing,” and “If CPU is for general computing and GPU is for accelerated computing, DPU, which moves data in the data center, is in charge of data processing.”
On February 28, Minister Lee Jong-ho of the Ministry of Science and ICT visited DPU startup Mango Boost Co., Ltd. to share the development status and achievements.

MangoBoost is a startup company founded this year by Professor Jang-Woo Kim of the Department of Electrical and Computer Engineering at Seoul National University and his students, specializing in the development of semiconductors and software related to DPUs. Last year, it attracted 13 billion won in investment. MangoBoost claims that “DPU significantly improves application performance and reduces CPU and server costs for infrastructure processing.”

The Ministry of Science and ICT said, “As competition to develop ultra-large AI such as ChatGPT intensifies, the DPU used in data centers that form the basis of ultra-large AI is also expected to see rapid growth in its market,” adding, “In addition to AI semiconductors that are currently receiving intensive support, the development of semiconductor technologies related to data processing is becoming increasingly important.”