Bug #53624: cephadm agent: set_store mon returned -27: error: entry size limited to 65536 bytes - Orchestrator - Ceph

Actions

Copy link

Bug #53624

closed

cephadm agent: set_store mon returned -27: error: entry size limited to 65536 bytes

Added by Sebastian Wagner over 2 years ago. Updated almost 2 years ago.

Status:

Resolved

Priority:

High

Assignee:

Adam King

Category:

cephadm

Target version:

% Done:

Source:

Tags:

Backport:

quincy

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

46376

Crash signature (v1):

Crash signature (v2):

Description

+p08UJQl6kAB3NtutBnSLaukwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTIxMTIxNjAyMjAzOFoXDTMx\nMTIxNzAyMjAzOFowFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAzqwANUb3
zFx4981X8YjxlvEPjLITotzyfgu0\neX6FdQTEfEiJjpv8Ikgq7QSKj0mf43J0uy/Y/+OENuHIdPSvtE3/ICQ7qX8HeOAS\noYS41vpAb8la9AODkohMlZa7OogPeoUpnU2Da2E4gUkxMcduMw3CsbF6Url8tiMY\nAqJyNUjQyZN9ah2NUsU4Bqhd2bXBSObDuKqA8M1M/nGwouOyQ
eVZmn1A4yQ3AOS8\nYb+fA/EWQtHZsdyNNBIU1bI68BaSSEtx00Fc1TFPiYf9JTZo5hT2sO3mBUKI/6uP\nwlIr3w23R2vP80WhbpLIZSSbZuQ0uyKc9XWqGvC4rX6VgWgs8xXW2FQGPfhzSWoL\nK4t9tafT8mLlokP5uelA4Da/JyrMf+U+ntEHbKjt/Jns8cR3dEEQQgzKh1197I
NZ\nq3lMHKOwVrkko2rwXaR4BTiv/9YDX1G/10Wc38U7V4Nyuc1MgShaBoy2GyjKl3Cy\nmiz3Jg0pehvqOjFTAfhqsbbocESyxWqZTZRK688NH7+EDOZT6gCEcCmhYJoNFhi1\nc2MJ2Zb4WgAanRcOto6LEYZ0kQQqZhce94RXKkJ265VZOFKcFqymGKAoxBhuBIAK\nJrmYvd1oE
GYROpHVL16tOzlyXxiNR+8zEMUWcRz9JmyE1ZwUhGqB4I2y9xGSsVlC\nK3ckXsECAwEAAaMkMCIwDwYDVR0RBAgwBocErBUCZjAPBgNVHRMBAf8EBTADAQH/\nMA0GCSqGSIb3DQEBCwUAA4ICAQAee7nTsfD5oQMYiZQA0qjeiAYDpG1bPFtyHxZL\n/TY96maIvBxqEx+haOmKvV
RTrV080NIo9VsftiLoFYj5M19bufkMlM764viIKOPE\n0fIHqCDvIresYuOIwB7WGqwiIzp4vQv/7QPqZdXRRW5WUN2fh2EhsSPlxBWoOwMf\nBNqsRwrYOxBzcXjeLJGGO4wmRa5AsFBwZC8cho+bc1W7FJUpXkZVnXmUPXx6ZzA8\nxvXBZfJgQ/2ya0uDBvXXwkxX1/7FHK4IsI7
gCGWHv95b5N2nM8A0Y8p5YvDDHnX2\nhnBmr97LdIfqgzScjblnNeQfNQWJj2kcuY0hjtHxb1Pg/t24N8JsJS1Jp5vshcbS\n8jWWK8wpUIkihDQCezY/5hUfhvQz4RozrAXgQKxAvGaXqWw2RXkwf7cbcXxIlS6A\nrt4L2Vg3xWS/2DIhAFwk8W53F0du3MfFOLIgVuBTOgbghapo
wqppo0qYtvH3nb6Y\nXhK12rDLzQvueQyxZZkjQ6UtYk1++eL+PojYMWFYYFesSayTs3xGi/6wvouVQb6v\njW8GeO2cf1mDJFCNDxAmtDoPCOJfT/j8dy8gCvkT0ypzdm69HqjO+sf8+Ik2VlrR\n3rJbNDd1WwP0o2CYLSIe2KOcQUvX505ojDFznTiLeUJ728Yd2qsnHTIOOK7NJ0x9\nd8hCvw==\n-----END CERTIFICATE-----\n", "172.21.2.102", "7150", "False"], "last_config": "2021-12-16T02:21:06.723847Z"}, "osd.1024": {"deps": [], "last_config": "2021-12-12T22:53:04.829290Z"}, "osd.1025": {"deps": [], "last_config": "2021-12-12T22:54:43.028992Z"}, "osd.1026": {"deps": [], "last_config": "2021-12-12T22:55:48.651847Z"}, "osd.1027": {"deps": [], "last_config": "2021-12-12T22:57:10.880016Z"}, "osd.1028": {"deps": [], "last_config": "2021-12-12T23:00:35.405950Z"}}, "last_daemon_update": "2021-12-16T02:27:35.437721Z", "last_device_update": "2021-12-12T22:52:34.049019Z", "last_network_update": "2021-12-16T02:27:15.196650Z", "last_device_change": "2021-12-12T22:44:01.089290Z", "networks_and_interfaces": {"172.21.0.0/20": {"enp1s0f0": ["172.21.2.145"]}, "fe80::/64": {"enp1s0f0": ["fe80::3eec:efff:fe3d:91b4"]}}, "last_host_check": "2021-12-16T02:22:44.926479Z", "last_client_files": {}, "scheduled_daemon_actions": {}, "agent_counter": 1, "agent_keys": "[client.agent.gibba045]\n\tkey = AQDuerZhDQN2GhAAMFdgQncZtlLlmjT/7IgaYA==\n"$` failed: (27) File too large
Dec 16 02:27:35 gibba002 ceph-mgr[421345]: mgr set_store mon returned -27: error: entry size limited to 65536 bytes. Use 'mon config key max entry size' to manually adjust

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Sebastian Wagner over 2 years ago

Category set to cephadm
Assignee set to Adam King

Actions

Copy link

Updated by Paul Cuzner over 2 years ago

Some more info.

The issue was encountered on a storage dense node. In this case the server was reporting 100+ devices (albeit not real devices!), but there are expansion chassis out there that support over 60 drives. The Dell MD1280 [1] was the largest I found @ 84 drives, and 45drives [2] provide a 60 drive chassis option for their ceph product,

[1] https://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell_Storage_MD1280_Spec_Sheet.pdf
[2] https://www.45drives.com/products/network-attached-storage/

Actions

Copy link

Updated by Sebastian Wagner over 2 years ago

Priority changed from Normal to High

But in any case we have to fix this before porting things to pacific.

Actions

Copy link

Updated by Adam King over 2 years ago

Status changed from New to In Progress
Pull request ID set to 44517

Actions

Copy link

Updated by Sebastian Wagner about 2 years ago

Status changed from In Progress to Fix Under Review

Actions

Copy link

Updated by Adam King about 2 years ago

Related to Bug #54251: ceph orch upgrade get stuck - mgr set_store mon returned -27: error: entry size limited to 65536 bytes added

Actions

Copy link

Updated by Adam King about 2 years ago

Status changed from Fix Under Review to New
Pull request ID deleted (~~44517~~)

linked PR didn't actually fix the issue. It's not an agent bug, this is happening on pacific which does not currently have the agent. Typically the issue is caused by too much device info on dense nodes.

Actions

Copy link