服务器kvm管理，服务器KVM控制台卡顿的全面解决方案，从故障诊断到性能优化

服务器kvm控制台卡顿问题可通过系统化诊断与优化解决，故障诊断需优先检查网络连接稳定性（包括VNC/KVM通道带宽及防火墙设置），使用top/htop监控CPU/内存使用率，排查资源过载；验证虚拟化配置（如QEMU-KVM参数、vga模式）及宿主机内核参数（如numa配置），性能优化方面，建议降低视频分辨率、禁用不必要的图形特效，启用VNC的压缩协议（如zlib/h264编码）；对网络通道可尝试TCP Keepalive、调整MTU值或改用UDP协议；宿主机可通过调整nohz_full内核参数减少进程调度延迟，并升级虚拟化驱动至最新版本，对于长期卡顿，需评估硬件性能（CPU核心数、GPU资源）或迁移至高性能计算节点，同时建议部署监控工具（如Zabbix）实现实时性能预警。

KVM控制台卡顿问题的本质分析

1 虚拟化环境的技术特性

KVM作为开源虚拟化平台，其控制台卡顿现象具有特殊性，不同于物理机直接操作，KVM通过QEMU模拟器与硬件交互，形成"硬件抽象层-虚拟机实例-用户界面"三层架构，这种架构在提升灵活性的同时,也引入了性能损耗的潜在风险。

图片来源于网络，如有侵权联系删除

2 卡顿现象的表现特征

• 界面响应延迟：鼠标操作延迟超过500ms
• 画面刷新异常：帧率波动超过15%±
• 资源占用突增：CPU使用率持续>90%
• 数据传输卡顿：文件传输速率骤降

3 典型场景分布

根据2023年Linux虚拟化监测报告,KVM控制台卡顿多发生在以下场景：

图片来源于网络，如有侵权联系删除

1. 虚拟桌面环境（X11/Wayland）
2. GPU加速应用（CUDA/OpenCL）
3. 大型数据库日志传输
4. 高并发网络服务（Nginx/Apache）

多维诊断方法论

1 硬件层诊断

1.1 CPU性能瓶颈检测

• 频率监控：使用lscpu检查CPU实际频率是否低于标称值（如Intel P-系列锁定在2.0GHz）
• 热功耗分析：通过sensors监测CPU TDP是否达到阈值（>95%）
• 多核利用率：使用mpstat 1 5分析逻辑核心负载均衡

1.2 内存子系统检测

• 内存碎片分析：执行sudo slabtop查看slab缓存使用情况
• 页面交换监控：使用vmstat 1 60观察si/sd指标
• ECC错误检测：检查/var/log/dmesg | grep error日志

1.3 存储性能评估

• IOPS压力测试：运行fio -io random write -direct=1 -size=1G -numjobs=32 -runtime=60模拟写入负载
• 队列深度分析：通过iostat -x 1监测磁盘队列长度
• 缓存策略验证：检查/etc/lvm/lvm.conf中的cache=write-through配置

2 软件层诊断

2.1 内核参数优化

• 实时内核测试：临时修改/etc/sysctl.conf添加kernel实时进程调度参数
• 内存页回收策略：调整vm页回收阈值（默认设置：/sysctl vm.trim_mighty=1）
• 文件系统日志：检查ext4日志级别（dmesg | grep ext4）

2.2 QEMU/KVM配置审计

• QEMU参数分析：检查/etc/kvm/qemu-system-x86_64.conf中的以下关键参数：

  [virtio-gpu]
  vblankmode=on
  heads=2
  [video]
  vga=on

• 驱动兼容性验证：对比/proc/scsi/scsi输出与lsmod | grep virtio

2.3 用户态工具诊断

• Xorg性能分析：运行xorgtrace记录显示流水线事件
• Wayland协议追踪：使用wayland-client --trace生成事件日志
• 图形渲染压力测试：执行glmark2 --test gl2 --num=100

分层优化策略

1 硬件性能调优

1.1 CPU配置优化

• 超线程管理：通过nohpet禁用Intel PT技术（需物理机支持）
• 内存通道绑定：使用lscpu -G确定通道拓扑，执行numactl -i绑定进程
• 频率调节策略：配置/sys/devices/system/cpu/cpu0/cpufreq/scaling_gov为powersave

1.2 存储性能提升

• RAID策略重构：将RAID1改为RAID10提升并发性能
• SSD缓存策略：在/etc/fstab中添加noatime,discard选项
• 多核I/O调度：设置nofile=65535并绑定到物理核心

2 软件配置优化

2.1 KVM/QEMU参数调整

• 图形性能优化：

  [video]
  vga=on
  vram=16M
  [virtio-gpu]
  vblankmode=on
  heads=4

• 网络性能增强：

  [network]
  model=e1000
  virtio-net-pci=on

2.2 内核参数优化

• 实时进程调度：

  echo "kernel实时进程调度=1" >> /etc/sysctl.conf
  sysctl -p

• 内存页回收：

  echo "vm.trim_mighty=1" >> /etc/sysctl.conf

2.3 用户界面优化

• Xorg配置调整：

  Section "ServerFlags"
    Option "AutoAddGPU" "on"
  EndSection

• Wayland性能设置：

  [Wayland]
  enabled=true
  default-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer-pointer