Не стабильная работа GSM

Добрый день!

Контроллер отвалился, появился только после перезагрузки по питанию. Можете проверить?

Смущают строки в логе:
port ttyUSB1 timed out 10 consecutive times, marking modem as invalid

приложен диагностический архив, доступен только сотрудникам поддержки
(313,9 КБ)

Добрый день.
Да, в dmesg

[ 1149.175995] usb 2-1.3: USB disconnect, device number 6
[ 1149.221843] rndis_host 2-1.3:1.0 usb0: unregister 'rndis_host' usb-ci_hdrc.1-1.3, RNDIS device
[ 1149.329405] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
[ 1149.464280] option 2-1.3:1.2: device disconnected
[ 1149.545722] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
[ 1149.677040] option 2-1.3:1.4: device disconnected
[ 1149.756321] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2
[ 1149.855748] option 2-1.3:1.5: device disconnected
[ 1170.722492] usb 2-1.3: new high-speed USB device number 7 using ci_hdrc
[ 1170.897735] usb 2-1.3: New USB device found, idVendor=1286, idProduct=812a, bcdDevice= 0.00
[ 1170.906554] usb 2-1.3: New USB device strings: Mfr=3, Product=2, SerialNumber=0
[ 1170.914324] usb 2-1.3: Product: WUKONG
[ 1170.918391] usb 2-1.3: Manufacturer: MARVELL
[ 1171.191781] usb 2-1.3: USB disconnect, device number 7
[ 1179.423611] usb 2-1.3: new high-speed USB device number 8 using ci_hdrc
[ 1179.573174] usb 2-1.3: config 1 has an invalid interface number: 5 but max is 4
[ 1179.580616] usb 2-1.3: config 1 has no interface number 3
[ 1179.592465] usb 2-1.3: New USB device found, idVendor=1e0e, idProduct=9011, bcdDevice= 1.00
[ 1179.600902] usb 2-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 1179.608287] usb 2-1.3: Product: Mobile Composite Device Bus
[ 1179.613896] usb 2-1.3: Manufacturer: Marvell

При этом как до перезапуска так и после в логах MM

Jul 01 00:07:56 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:07:56 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator code: No network service
Jul 01 00:07:56 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator name: No network service
Jul 01 00:07:56 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:08:19 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:08:19 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:08:37 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:08:37 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:08:37 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator code: No network service
Jul 01 00:08:38 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator name: No network service
Jul 01 00:08:38 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:08:59 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:08:59 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:09:59 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:09:59 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:09:59 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator code: No network service
Jul 01 00:09:59 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator name: No network service
Jul 01 00:09:59 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:10:22 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:10:22 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (searching -> idle)
Jul 01 00:11:00 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (idle -> registering)
Jul 01 00:11:00 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:15:33 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:15:33 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:15:33 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator code: No network service
Jul 01 00:15:33 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator name: No network service
Jul 01 00:15:33 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:15:54 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:15:54 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:16:12 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:16:12 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:16:12 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator code: No network service
Jul 01 00:16:12 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator name: No network service
Jul 01 00:16:12 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:16:36 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:16:36 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (searching -> idle)
Jul 01 00:17:13 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (idle -> registering)
Jul 01 00:17:13 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:25:52 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:25:52 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:25:52 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator code: No network service
Jul 01 00:25:52 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator name: No network service
Jul 01 00:25:52 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:26:15 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:26:15 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (searching -> idle)
Jul 01 00:26:50 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (idle -> registering)
Jul 01 00:26:50 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:29:51 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:29:51 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:29:51 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator code: No network service
Jul 01 00:29:51 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator name: No network service
Jul 01 00:29:51 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:30:10 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:30:10 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (registering -> home)
Jul 01 00:31:40 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> registering)
Jul 01 00:31:40 IQ458000 ModemManager[413]: <info>  [modem3] 3GPP registration state changed (home -> searching)
Jul 01 00:31:40 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator code: No network service
Jul 01 00:31:40 IQ458000 ModemManager[413]: <warn>  [modem3] couldn't load operator name: No network service

То есть нет связи с оператором, нет регистрации в сети.

И явно отказано в регистрации тут:

Jul 02 07:20:34 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (idle -> denied)
Jul 02 07:20:34 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (denied -> idle)
Jul 02 07:20:34 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (idle -> denied)
Jul 02 07:21:07 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (denied -> idle)
Jul 02 07:22:01 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (idle -> denied)
Jul 02 07:22:30 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (denied -> idle)
Jul 02 07:24:27 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (idle -> denied)
Jul 02 07:24:27 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (denied -> idle)
Jul 02 07:24:27 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (idle -> denied)
Jul 02 07:26:21 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (denied -> idle)
Jul 02 07:33:17 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (idle -> registering)
Jul 02 07:33:19 IQ458000 ModemManager[413]: <info>  [modem1] 3GPP registration state changed (registering -> home)
Jul 02 07:33:19 IQ458000 ModemManager[413]: <info>  [modem1] state changed (enabled -> registered)
Jul 02 09:34:17 IQ458000 ModemManager[413]: <info>  [modem1] simple connect started...
Jul 02 09:34:17 IQ458000 ModemManager[413]: <info>  [modem1] simple connect state (6/10): register
Jul 02 09:34:17 IQ458000 ModemManager[413]: <info>  [modem1] simple connect state (7/10): wait to get packet service state attached
Jul 02 09:34:17 IQ458000 ModemManager[413]: <info>  [modem1] simple connect state (8/10): bearer
Jul 02 09:34:17 IQ458000 ModemManager[413]: <info>  [modem1] simple connect state (9/10): connect
Jul 02 09:34:17 IQ458000 ModemManager[413]: <info>  [modem1] state changed (registered -> connecting)
Jul 02 09:34:17 IQ458000 ModemManager[413]: <info>  [modem1] state changed (connecting -> connected)
Jul 02 09:34:17 IQ458000 ModemManager[413]: <info>  [modem1] simple connect state (10/10): all done

То есть несколько минут просто отказ - а потом сразу подключение…

Ну, то есть причина - нет регистрации, MM и так и этак пробовал и модем перезапускал по питанию несколько раз.

То есть симки он переключал, но всё равно не мог зарегистрироваться в сети?

Это совпало с обновлением ОС с 2401 до 2404. Это сильно смущает.

Так, это интересно То есть было обновление но контроллер продолжал работать?

Заменили 2 симки на новые. Контроллер работает на WiFi.

приложен диагностический архив, доступен только сотрудникам поддержки
(312,1 КБ)

Не совсем могу сказать, когда именно отвалился, но контроллер не новый, работал на 2401, было нормально. Тут обновляли большую партию на 2404 + наш софт и начались проблемы на этом контроллере. Я не уверен, что проблема именно с ОС, но я помню багу с отваливающимся модемом (когда сделали вочдог для перезапуска модема) - хотелось бы это исключить.

Во время работы на WiFi как продиагностировать работу sim? На контроллере сижу удаленно, случайно обрубить канал и ехать за контроллером очень не хочется.

Последние логи вот такие:

root@IQ458000:/var/log# journalctl -u ModemManager --since "1 hour ago" | tail -n 50
-- Journal begins at Tue 2024-07-02 08:19:37 UTC, ends at Wed 2024-07-03 14:59:55 UTC. --
Jul 03 14:21:46 IQ458000 systemd[1]: Starting Modem Manager...
Jul 03 14:21:47 IQ458000 ModemManager[412]: <info>  ModemManager (version 1.20.0) starting in system bus...
Jul 03 14:21:48 IQ458000 systemd[1]: Started Modem Manager.
Jul 03 14:21:52 IQ458000 ModemManager[412]: <info>  [base-manager] couldn't check support for device '/sys/devices/platform/soc/2000000.bus/20b4000.ethernet': not supported by any plugin
Jul 03 14:21:52 IQ458000 ModemManager[412]: <info>  [base-manager] couldn't check support for device '/sys/devices/platform/soc/2100000.bus/2184200.usb/ci_hdrc.1/usb2/2-1/2-1.1': not supported by any plugin
Jul 03 14:21:52 IQ458000 ModemManager[412]: <info>  [base-manager] couldn't check support for device '/sys/devices/platform/soc/2100000.bus/2188000.ethernet': not supported by any plugin
Jul 03 14:25:27 IQ458000 ModemManager[412]: <info>  [device wbc] creating modem with plugin 'simtech' and '4' ports
Jul 03 14:25:27 IQ458000 ModemManager[412]: <info>  [base-manager] modem for device 'wbc' successfully created
Jul 03 14:25:28 IQ458000 ModemManager[412]: <warn>  [modem0] couldn't load supported modes: Couldn't retrieve supported modes
Jul 03 14:25:31 IQ458000 ModemManager[412]: <warn>  [modem0/sim0] couldn't load list of emergency numbers: Failed to parse CRSM query result '+CRSM: 105,129'
Jul 03 14:25:32 IQ458000 ModemManager[412]: <warn>  [modem0/sim0] couldn't load list of emergency numbers: Failed to parse CRSM query result '+CRSM: 105,129'
Jul 03 14:25:33 IQ458000 ModemManager[412]: <info>  [modem0] state changed (unknown -> disabled)
Jul 03 14:25:33 IQ458000 ModemManager[412]: <info>  [modem0] state changed (disabled -> enabling)
Jul 03 14:25:34 IQ458000 ModemManager[412]: <info>  [modem0] power state updated: on
Jul 03 14:25:54 IQ458000 ModemManager[412]: <info>  [modem0] mobile equipment forced PS detach: all contexts have been deactivated
Jul 03 14:25:54 IQ458000 ModemManager[412]: <info>  [modem0] network request to deactivate context (cid 1)
Jul 03 14:25:54 IQ458000 ModemManager[412]: <info>  [modem0] 3GPP registration state changed (unknown -> idle)
Jul 03 14:26:02 IQ458000 ModemManager[412]: <info>  [modem0] state changed (enabling -> enabled)
Jul 03 14:26:02 IQ458000 ModemManager[412]: <info>  [modem0] state changed (enabled -> disabling)
Jul 03 14:26:08 IQ458000 ModemManager[412]: <info>  [modem0] 3GPP registration state changed (idle -> unknown)
Jul 03 14:26:10 IQ458000 ModemManager[412]: <info>  [device wbc] creating modem with plugin 'simtech' and '4' ports
Jul 03 14:26:21 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 2 consecutive times
Jul 03 14:26:27 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 3 consecutive times
Jul 03 14:26:32 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 4 consecutive times
Jul 03 14:26:34 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 5 consecutive times
Jul 03 14:26:39 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 6 consecutive times
Jul 03 14:26:45 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 7 consecutive times
Jul 03 14:26:45 IQ458000 ModemManager[412]: <warn>  [modem1] modem couldn't be initialized: Failed to load current capabilities: Failed to determine modem capabilities.
Jul 03 14:26:45 IQ458000 ModemManager[412]: <info>  [modem1] state changed (unknown -> failed)
Jul 03 14:26:53 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 8 consecutive times
Jul 03 14:26:53 IQ458000 ModemManager[412]: <warn>  [modem1] error initializing: Modem in failed state: unknown-capabilities

Все таки нужно продолжать дергать симки даже если есть более приоритетный WiFi.

Точно ли проблема в SIM, а не в модеме?

Через 2,5 часа GSM заработал
journalctl -u ModemManager -b 0
-- Journal begins at Tue 2024-07-02 08:19:37 UTC, ends at Wed 2024-07-03 18:22:04 UTC. --
Jul 03 14:21:46 IQ458000 systemd[1]: Starting Modem Manager...
Jul 03 14:21:47 IQ458000 ModemManager[412]: <info>  ModemManager (version 1.20.0) starting in system bus...
Jul 03 14:21:48 IQ458000 systemd[1]: Started Modem Manager.
Jul 03 14:21:52 IQ458000 ModemManager[412]: <info>  [base-manager] couldn't check support for device '/sys/devices/platform/soc/2000000.bus/20b4000.ethernet': not suppo>
Jul 03 14:21:52 IQ458000 ModemManager[412]: <info>  [base-manager] couldn't check support for device '/sys/devices/platform/soc/2100000.bus/2184200.usb/ci_hdrc.1/usb2/2>
Jul 03 14:21:52 IQ458000 ModemManager[412]: <info>  [base-manager] couldn't check support for device '/sys/devices/platform/soc/2100000.bus/2188000.ethernet': not suppo>
Jul 03 14:25:27 IQ458000 ModemManager[412]: <info>  [device wbc] creating modem with plugin 'simtech' and '4' ports
Jul 03 14:25:27 IQ458000 ModemManager[412]: <info>  [base-manager] modem for device 'wbc' successfully created
Jul 03 14:25:28 IQ458000 ModemManager[412]: <warn>  [modem0] couldn't load supported modes: Couldn't retrieve supported modes
Jul 03 14:25:31 IQ458000 ModemManager[412]: <warn>  [modem0/sim0] couldn't load list of emergency numbers: Failed to parse CRSM query result '+CRSM: 105,129'
Jul 03 14:25:32 IQ458000 ModemManager[412]: <warn>  [modem0/sim0] couldn't load list of emergency numbers: Failed to parse CRSM query result '+CRSM: 105,129'
Jul 03 14:25:33 IQ458000 ModemManager[412]: <info>  [modem0] state changed (unknown -> disabled)
Jul 03 14:25:33 IQ458000 ModemManager[412]: <info>  [modem0] state changed (disabled -> enabling)
Jul 03 14:25:34 IQ458000 ModemManager[412]: <info>  [modem0] power state updated: on
Jul 03 14:25:54 IQ458000 ModemManager[412]: <info>  [modem0] mobile equipment forced PS detach: all contexts have been deactivated
Jul 03 14:25:54 IQ458000 ModemManager[412]: <info>  [modem0] network request to deactivate context (cid 1)
Jul 03 14:25:54 IQ458000 ModemManager[412]: <info>  [modem0] 3GPP registration state changed (unknown -> idle)
Jul 03 14:26:02 IQ458000 ModemManager[412]: <info>  [modem0] state changed (enabling -> enabled)
Jul 03 14:26:02 IQ458000 ModemManager[412]: <info>  [modem0] state changed (enabled -> disabling)
Jul 03 14:26:08 IQ458000 ModemManager[412]: <info>  [modem0] 3GPP registration state changed (idle -> unknown)
Jul 03 14:26:10 IQ458000 ModemManager[412]: <info>  [device wbc] creating modem with plugin 'simtech' and '4' ports
Jul 03 14:26:21 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 2 consecutive times
Jul 03 14:26:27 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 3 consecutive times
Jul 03 14:26:32 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 4 consecutive times
Jul 03 14:26:34 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 5 consecutive times
Jul 03 14:26:39 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 6 consecutive times
Jul 03 14:26:45 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 7 consecutive times
Jul 03 14:26:45 IQ458000 ModemManager[412]: <warn>  [modem1] modem couldn't be initialized: Failed to load current capabilities: Failed to determine modem capabilities.
Jul 03 14:26:45 IQ458000 ModemManager[412]: <info>  [modem1] state changed (unknown -> failed)
Jul 03 14:26:53 IQ458000 ModemManager[412]: <warn>  [modem1] port ttyUSB1 timed out 8 consecutive times
Jul 03 14:26:53 IQ458000 ModemManager[412]: <warn>  [modem1] error initializing: Modem in failed state: unknown-capabilities
Jul 03 17:07:32 IQ458000 ModemManager[412]: <info>  [base-manager] port ttyUSB0 released by device 'wbc'
Jul 03 17:07:32 IQ458000 ModemManager[412]: <info>  [base-manager] port ttyUSB1 released by device 'wbc'
Jul 03 17:07:32 IQ458000 ModemManager[412]: <info>  [base-manager] port usb0 released by device 'wbc'
Jul 03 17:07:32 IQ458000 ModemManager[412]: <info>  [base-manager] port ttyUSB2 released by device 'wbc'
Jul 03 17:07:44 IQ458000 ModemManager[412]: <info>  [device wbc] creating modem with plugin 'simtech' and '4' ports
Jul 03 17:07:44 IQ458000 ModemManager[412]: <info>  [base-manager] modem for device 'wbc' successfully created
Jul 03 17:07:47 IQ458000 ModemManager[412]: <warn>  [modem2] couldn't load supported modes: Couldn't retrieve supported modes
Jul 03 17:07:49 IQ458000 ModemManager[412]: <warn>  [modem2/sim1] couldn't load list of emergency numbers: Failed to parse CRSM query result '+CRSM: 105,129'
Jul 03 17:07:49 IQ458000 ModemManager[412]: <warn>  [modem2/sim1] couldn't load GID1: Memory failure
Jul 03 17:07:49 IQ458000 ModemManager[412]: <warn>  [modem2/sim1] couldn't load GID2: Memory failure
Jul 03 17:07:50 IQ458000 ModemManager[412]: <warn>  [modem2/sim1] couldn't load list of emergency numbers: Failed to parse CRSM query result '+CRSM: 105,129'
Jul 03 17:07:50 IQ458000 ModemManager[412]: <warn>  [modem2/sim1] couldn't load GID1: Memory failure
Jul 03 17:07:50 IQ458000 ModemManager[412]: <warn>  [modem2/sim1] couldn't load GID2: Memory failure
Jul 03 17:07:52 IQ458000 ModemManager[412]: <info>  [modem2] state changed (unknown -> disabled)
Jul 03 17:07:52 IQ458000 ModemManager[412]: <info>  [modem2] state changed (disabled -> enabling)
Jul 03 17:07:53 IQ458000 ModemManager[412]: <info>  [modem2] power state updated: on
Jul 03 17:08:17 IQ458000 ModemManager[412]: <info>  [modem2] state changed (enabling -> enabled)
Jul 03 17:08:17 IQ458000 ModemManager[412]: <info>  [modem2] 3GPP registration state changed (unknown -> registering)
Jul 03 17:08:17 IQ458000 ModemManager[412]: <info>  [modem2] 3GPP registration state changed (registering -> home)
Jul 03 17:08:17 IQ458000 ModemManager[412]: <info>  [modem2] state changed (enabled -> registered)
Jul 03 17:08:19 IQ458000 ModemManager[412]: <info>  [modem2] simple connect started...
Jul 03 17:08:19 IQ458000 ModemManager[412]: <info>  [modem2] simple connect state (6/10): register
Jul 03 17:08:19 IQ458000 ModemManager[412]: <info>  [modem2] simple connect state (7/10): wait to get packet service state attached
Jul 03 17:08:19 IQ458000 ModemManager[412]: <info>  [modem2] simple connect state (8/10): bearer
Jul 03 17:08:19 IQ458000 ModemManager[412]: <info>  [modem2] simple connect state (9/10): connect
Jul 03 17:08:19 IQ458000 ModemManager[412]: <info>  [modem2] state changed (registered -> connecting)
Jul 03 17:08:20 IQ458000 ModemManager[412]: <info>  [modem2] state changed (connecting -> connected)
Jul 03 17:08:20 IQ458000 ModemManager[412]: <info>  [modem2] simple connect state (10/10): all done
Jul 03 17:08:22 IQ458000 ModemManager[412]: <info>  [modem2] mobile equipment request to activate context (cid 3)

Кажется проблема не в GSM.

Сейчас поймал вот такой лог:

Jul 03 20:45:03 IQ458000 systemd[1]: wb-rules.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 03 20:45:03 IQ458000 systemd[1]: wb-rules.service: Failed with result 'exit-code'.
Jul 03 20:45:04 IQ458000 systemd[1]: wb-rules.service: Scheduled restart job, restart counter is at 2.
Jul 03 20:45:04 IQ458000 systemd[1]: Stopped MQTT Rule engine for Wiren Board.
Jul 03 20:45:04 IQ458000 systemd[1]: Started MQTT Rule engine for Wiren Board.
Jul 03 20:45:04 IQ458000 wb-rules[12367]: INFO: broker URL is default and mosquitto socket detected, trying to connect via it
Jul 03 20:45:04 IQ458000 wb-rules[12367]: INFO: driver is created
Jul 03 20:45:04 IQ458000 wb-rules[12367]: panic: network Error : dial unix /var/run/mosquitto/mosquitto.sock: connect: resource temporarily unavailable
Jul 03 20:45:04 IQ458000 wb-rules[12367]: goroutine 1 [running]:
Jul 03 20:45:04 IQ458000 wb-rules[12367]: github.com/wirenboard/wbgo-private.(*PahoMQTTClient).Start(0x18e4200)
Jul 03 20:45:04 IQ458000 wb-rules[12367]:         github.com/wirenboard/wbgo-private/mqtt.go:207 +0x194
Jul 03 20:45:04 IQ458000 wb-rules[12367]: github.com/wirenboard/wbgo-private.(*DriverMQTTBackend).Start(0x18a8cc0, 0x945271e8, 0x18719e0)
Jul 03 20:45:04 IQ458000 wb-rules[12367]:         github.com/wirenboard/wbgo-private/driver_backend.go:1611 +0xa4
Jul 03 20:45:04 IQ458000 wb-rules[12367]: github.com/wirenboard/wbgo-private.(*driverBase).StartLoop(0x18719e0, 0x18cbf4c, 0x1)
Jul 03 20:45:04 IQ458000 wb-rules[12367]:         github.com/wirenboard/wbgo-private/driver_frontend.go:737 +0xa8
Jul 03 20:45:04 IQ458000 wb-rules[12367]: main.main()
Jul 03 20:45:04 IQ458000 wb-rules[12367]:         github.com/wirenboard/wb-rules/main.go:147 +0x84c
Jul 03 20:45:04 IQ458000 systemd[1]: wb-rules.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 03 20:45:04 IQ458000 systemd[1]: wb-rules.service: Failed with result 'exit-code'.
Jul 03 20:45:06 IQ458000 systemd[1]: wb-rules.service: Scheduled restart job, restart counter is at 3.
Jul 03 20:45:06 IQ458000 systemd[1]: Stopped MQTT Rule engine for Wiren Board.
Jul 03 20:45:06 IQ458000 systemd[1]: Started MQTT Rule engine for Wiren Board.
Jul 03 20:45:06 IQ458000 wb-rules[12372]: INFO: broker URL is default and mosquitto socket detected, trying to connect via it
Jul 03 20:45:06 IQ458000 wb-rules[12372]: INFO: driver is created
Jul 03 20:45:06 IQ458000 wb-rules[12372]: panic: network Error : dial unix /var/run/mosquitto/mosquitto.sock: connect: resource temporarily unavailable
Jul 03 20:45:06 IQ458000 wb-rules[12372]: goroutine 1 [running]:
Jul 03 20:45:06 IQ458000 wb-rules[12372]: github.com/wirenboard/wbgo-private.(*PahoMQTTClient).Start(0x28e4200)
Jul 03 20:45:06 IQ458000 wb-rules[12372]:         github.com/wirenboard/wbgo-private/mqtt.go:207 +0x194
Jul 03 20:45:06 IQ458000 wb-rules[12372]: github.com/wirenboard/wbgo-private.(*DriverMQTTBackend).Start(0x28a8cc0, 0x946271e8, 0x2871830)
Jul 03 20:45:06 IQ458000 wb-rules[12372]:         github.com/wirenboard/wbgo-private/driver_backend.go:1611 +0xa4
Jul 03 20:45:06 IQ458000 wb-rules[12372]: github.com/wirenboard/wbgo-private.(*driverBase).StartLoop(0x2871830, 0x28cbf4c, 0x1)
Jul 03 20:45:06 IQ458000 wb-rules[12372]:         github.com/wirenboard/wbgo-private/driver_frontend.go:737 +0xa8
Jul 03 20:45:06 IQ458000 wb-rules[12372]: main.main()
Jul 03 20:45:06 IQ458000 wb-rules[12372]:         github.com/wirenboard/wb-rules/main.go:147 +0x84c
Jul 03 20:45:06 IQ458000 systemd[1]: wb-rules.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 03 20:45:06 IQ458000 systemd[1]: wb-rules.service: Failed with result 'exit-code'.

Падает mosquitto?

приложен диагностический архив, доступен только сотрудникам поддержки
(335,0 КБ)

Еще нашел сервис wb-mqtt-mbgate, который периодически сильно грузил проц:

journalctl -u wb-mqtt-mbgate -f -n 200
-- Journal begins at Tue 2024-07-02 10:10:33 UTC. --
Jul 03 21:20:25 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Control process exited, code=exited, status=1/FAILURE
Jul 03 21:20:25 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Failed with result 'exit-code'.
Jul 03 21:20:25 IQ458000 systemd[1]: Failed to start Wiren Board MQTT to Modbus TCP gateway.
Jul 03 21:20:25 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Consumed 1.133s CPU time.
Jul 03 21:20:35 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Scheduled restart job, restart counter is at 1758.
Jul 03 21:20:35 IQ458000 systemd[1]: Stopped Wiren Board MQTT to Modbus TCP gateway.
Jul 03 21:20:35 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Consumed 1.133s CPU time.
Jul 03 21:20:35 IQ458000 systemd[1]: Starting Wiren Board MQTT to Modbus TCP gateway...
Jul 03 21:20:36 IQ458000 wb-mqtt-mbgate-confgen[22555]: Failed to open config
Jul 03 21:20:37 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Control process exited, code=exited, status=1/FAILURE
Jul 03 21:20:37 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Failed with result 'exit-code'.
Jul 03 21:20:37 IQ458000 systemd[1]: Failed to start Wiren Board MQTT to Modbus TCP gateway.
Jul 03 21:20:37 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Consumed 1.126s CPU time.
Jul 03 21:20:47 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Scheduled restart job, restart counter is at 1759.
Jul 03 21:20:47 IQ458000 systemd[1]: Stopped Wiren Board MQTT to Modbus TCP gateway.
Jul 03 21:20:47 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Consumed 1.126s CPU time.
Jul 03 21:20:47 IQ458000 systemd[1]: Starting Wiren Board MQTT to Modbus TCP gateway...
Jul 03 21:20:49 IQ458000 wb-mqtt-mbgate-confgen[22630]: Failed to open config
Jul 03 21:20:49 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Control process exited, code=exited, status=1/FAILURE
Jul 03 21:20:49 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Failed with result 'exit-code'.
Jul 03 21:20:49 IQ458000 systemd[1]: Failed to start Wiren Board MQTT to Modbus TCP gateway.
Jul 03 21:20:49 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Consumed 1.143s CPU time.
Jul 03 21:20:59 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Scheduled restart job, restart counter is at 1760.
Jul 03 21:20:59 IQ458000 systemd[1]: Stopped Wiren Board MQTT to Modbus TCP gateway.
Jul 03 21:20:59 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Consumed 1.143s CPU time.
Jul 03 21:20:59 IQ458000 systemd[1]: Starting Wiren Board MQTT to Modbus TCP gateway...
Jul 03 21:21:01 IQ458000 wb-mqtt-mbgate-confgen[22660]: Failed to open config
Jul 03 21:21:01 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Control process exited, code=exited, status=1/FAILURE
Jul 03 21:21:01 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Failed with result 'exit-code'.
Jul 03 21:21:01 IQ458000 systemd[1]: Failed to start Wiren Board MQTT to Modbus TCP gateway.
Jul 03 21:21:01 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Consumed 1.130s CPU time.
Jul 03 21:21:11 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Scheduled restart job, restart counter is at 1761.
Jul 03 21:21:11 IQ458000 systemd[1]: Stopped Wiren Board MQTT to Modbus TCP gateway.
Jul 03 21:21:11 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Consumed 1.130s CPU time.
Jul 03 21:21:11 IQ458000 systemd[1]: Starting Wiren Board MQTT to Modbus TCP gateway...
Jul 03 21:21:13 IQ458000 wb-mqtt-mbgate-confgen[22679]: Failed to open config
Jul 03 21:21:13 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Control process exited, code=exited, status=1/FAILURE
Jul 03 21:21:13 IQ458000 systemd[1]: wb-mqtt-mbgate.service: Failed with result 'exit-code'.
Jul 03 21:21:13 IQ458000 systemd[1]: Failed to start Wiren Board MQTT to Modbus TCP gateway.

Отключил:

service wb-mqtt-mbgate stop
systemctl disable wb-mqtt-mbgate

Причем конфиг был пустой:

ls -lh /mnt/data/etc/wb-mqtt-mbgate.conf
-rw-r--r-- 1 root root 0 Jun 30 06:37 /mnt/data/etc/wb-mqtt-mbgate.conf

Сейчас контроллер работает заметно шустрее, буду наблюдать.

обновление 30 числа в 9 часов было?
А сам конфиг он пуст именно после обновления или до него тоже был пустой?

30е число похоже на правду, но не уверен.

Мы не используем этот сервис, я не слежу за этим конфигом и никак его не редактирую.