Logs

Logs are a vital resource for troubleshooting issues and optimizing the performance of your EMQX deployment. EMQX deployment logs capture information about access, operations, and network events. Through the EMQX Platform Console, you can view logs in real-time or download them for offline analysis.

View Deployment Logs

To access deployment logs, navigate to your deployment in the EMQX Platform Console and select Logs from the menu to open the Logs page.

Filter by Time

By default, the Logs page displays logs from the past 3 days. You can filter logs by the following predefined time ranges:

• Last 1 hour
• Last 12 hours
• Last 1 day

You can also use the calendar picker to select a custom time range. However:

• A maximum window of 3 days can be selected at once.
• You can view logs from up to 14 days in the past.

Filter by Log Level

The deployment logs are categorized into five severity levels:

• Warning
• Error
• Critical
• Alert
• Emergency

You can filter the logs by selecting a severity level from the drop-down box and clicking the "search" button.

Filter by Error Type

You can filter logs by specific error categories:

• Data Integration: Data integration-related errors, such as MySQL being offline, authorization failures, or schema issues.
• Clients: Client-related errors, including incorrect authentication information, incorrect access control information, and other reasons for not being able to connect.
• Messages: Message-related errors, such as encoding problems, message loss, or dropped messages.
• Authentication: Authentication errors, such as extended authentication errors due to inability to connect to the corresponding backend service.
• Authorization: Authorization errors, such as failures in authorization rule enforcement.

Filter by Client ID, Client IP, and More

You can filter logs using the following fields:

• Client ID
• Client IP

For additional filtering options, open the More Filters drop-down, which includes:

• Username
• Topic
• Connector Name
• Action Name
• Rule ID

Reset and Download Logs

• Click Reset to restore the default log view.
• Click Download to export up to 1,000 log records from the search results.

Common Log Analysis

This section summarizes common warning and error logs in EMQX, including log examples, causes of issues, and recommended solutions to assist with operations and troubleshooting.

Client Connection and Authorization Logs

[Warning] Client Authentication Failed

Log Example:

clientid: xx, peername: xxx:xx, username: 1, pid: <0.99601.0>, reason: not_authorized, tag: AUTHN, msg: authentication_failure

This log indicates that no corresponding MQTT username authentication data was found.

clientid: xx, peername: xxx:xx, username: test, pid: <0.15269.0>, reason: bad_username_or_password, tag: AUTHN, msg: authentication_failure

This log indicates that the MQTT username was found, but the password validation failed.

Solution: Verify that the username and password configured on the client are correct.

[Warning] Client ACL Authorization Failed

Log Example:

clientid: xx, peername: xxx:xx, username: test, topic: test, pid: <0.34387.0>, action: SUBSCRIBE(Q0), source: built_in_database, tag: AUTHZ, msg: authorization_permission_denied

Issue: The client passed authentication but attempted to subscribe/publish to a topic without permission, triggering an authorization denial.

Solution:

• Check whether the access control (ACL) configuration is correct.
• Confirm whether the current user has permission to access the specified topic.

[Warning] Non-standard MQTT Protocol Connection Failure

Log Example:

[MQTT] Parse failed for function_clause[{emqx_frame,parse_packet,[{mqtt_packet_header,4,false,3,true}

Issue: The client used a non-standard MQTT protocol or incorrect packet format during connection.

Solution: Ensure the client is using a standard MQTT protocol version (e.g., 3.1.1 or 5.0).

Message Delivery and Flow Control Logs

[Warning] Message Dropped (Queue Full)

Log Example:

clientid: xx, peername: xxx:xx, username: test, topic: test, pid: <0.198894.0>, payload: xxxxxxxxxx, payload_encode: hex, queue: {"store_qos0":true,"max_len":1000,"len":1000,"dropped":56942}, msg: dropped_msg_due_to_mqueue_is_full

Issue: Messages were dropped because the client was offline or slow to consume messages, causing the server-side queue to fill up.

Solution:

1. If the client is frequently offline:
   • Set clean session = true.
   • Enable automatic reconnection to avoid session loss after disconnection.
2. If the client is slow to consume messages:
   • Optimize the client SDK to improve consumption rate.
   • Use shared subscriptions to distribute message load.

[Warning] TPS Limit Exceeded

Log Example:

clientid: xx, peername: xxx:xx, username: test, topic: test, pid: <0.194662.0>, reason: {failed_to_consume_from_limiter,{{listener,'tcp:default'},messages}}, tag: QUOTA, msg: cannot_publish_to_topic_due_to_quota_exceeded

Issue: The client’s publish rate exceeded the deployment-defined Transactions Per Second (TPS) limit.

Solution:

• Reduce the client’s message publish rate to stay below the configured TPS limit.
• Alternatively, adjust the TPS quota in the deployment according to business needs.

Deployment Data Integration Logs

[Error] MySQL Connector Startup Failure

Log Example:

[warning] tag: CONNECTOR/MYSQL, msg: start_resource_failed, reason: {start_pool_failed,<<"connector:mysql:test">>,{shutdown,#{cause => ehostunreach,port => 3306,user => <<"root">>,host => "xxx",database => <<"mqtt">>}}}, resource_id: <<"connector:mysql:test">>

Issue: The MySQL data connector failed to initialize, likely due to unreachable database service or incorrect connection configuration.

Solution:

• Check the MySQL server status and network connectivity.
• Verify the connector configuration (host, port, username, database name, etc.).

[Error] HTTP Connector Startup Failure

Log Example:

pid: <0.28885448.0>, line: 810, reason: timeout, resource_id: connector:http:c-xxX, tag: CONNECTOR/WEBHOOK, msg: start_resource_failed

Issue: The Webhook or HTTP connector failed to start, usually due to a timeout when connecting to the HTTP service.

Solution:

• Ensure the target HTTP service is accessible.
• Check whether the URL, port, and authentication settings in the HTTP connector configuration are correct.

[Error] RocketMQ Connector Startup Failure

Log Example:

[error] crasher: initial call: rocketmq_client:init/1, pid: <0.5161.0>, exit: {fail_to_connect_rocketmq_server,...}
[warning] tag: CONNECTOR/ROCKETMQ, msg: start_resource_failed, reason: {fail_to_connect_rocketmq_server,...}, resource_id: <<"connector:rocketmq:test">>

Issue: The RocketMQ client failed to connect to the target server, causing the connector to fail during initialization.

Solution:

• Check whether the RocketMQ service is running correctly.
• Verify the connector configuration (address, port, credentials, etc.).

[Error] Kafka Resource Exception

Log Example:

topic: xxx, pid: <0.25097.0>, group: ..., reason: failed_to_fetch_metadata, msg: failed_to_refresh_partition_count_will_retry
topic: xxx, pid: <0.25048.0>, errors: [{"reason":"connection_timed_out","host":"xxx:xx"}], msg: failed_to_fetch_metadata

Issue: The Kafka connector failed to fetch topic metadata, possibly due to connection timeouts or misconfigured Kafka cluster settings.

Solution:

• Check the availability and connectivity of the Kafka service.
• Ensure the Kafka topic configured in the action has already been created in the Kafka cluster.

[Error] MQTT Resource Exception

Log Example:

pid: <0.5959.0>, line: 59, reason: timeout, resource_id: connector:mqtt:c-xxx, tag: RESOURCE, msg: start_ecpool_error
pid: <0.5959.0>, line: 895, reason: timeout, resource_id: connector:mqtt:c-xxx, tag: CONNECTOR/MQTT, msg: start_resource_failed

Issue: The MQTT connector initialization timed out, possibly due to unreachable target MQTT service or misconfiguration.

Recommendations:

• Check the target MQTT broker status and network connectivity.
• Verify that the MQTT connector configuration (host, port, authentication, etc.) is correct.